Low-level Vision Processing Algorithms Speaker: Ito, Dang Supporter: Ishii, Toyama and Y. Murakami

Transcription

1 Low-level Vision Processing Algorithms Speaker: Ito, Dang Supporter: Ishii, Toyama and Y. Murakami Adaptive Systems Lab The University of Aizu

2 Overview Introduction What is Vision Processing? Basic Knowledge Image formation Transformation Low level Algorithm Filtering Edge/Border Detection Conner Detection Conclusion 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 2

3 Overview Introduction What is Vision Processing? Basic Knowledge Image formation Transformation Low level Algorithm Filtering Edge/Border Detection Conner Detection Conclusion 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 3

4 What is Vision Processing? Make computers understand images and video: Images to Models What kind of scene? Where are the cars? How far is the building? Slide credit James Hays 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 4

5 Application of Vision Processing Optical Character Recognition Face detection Object recognition Shape/motion capture Medical Imaging 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 5

6 Why we need? Human limitations measurement accuracy, darkness, etc 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 6/?

7 Overview Introduction What is Vision Processing? Basic Knowledge Image formation Transformation Low level Algorithms Filtering Border Detection Edge Detection Conclusion 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 7

8 Basic Knowledge Image and Color. Camera, Lens. Color representation. Fourier Transform. 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 8

9 Image Formation Digital Camera Film The Eye

10 Digital camera A digital camera replaces film with a sensor array Each cell in the array is light-sensitive diode that converts photons to electrons Two common types Charge Coupled Device (CCD) CMOS

11 Sensor Array CMOS sensor

12 Electromagnetic Spectrum Human Luminance Sensitivity Function

13 The Physics of Light Any patch of light can be completely described physically by its spectrum: the number of photons (per time unit) at each wavelength nm. # Photons (per ms.) Wavelength (nm.) Stephen E. Palmer, 2002

14 % Photons Reflected The Physics of Light Some examples of the reflectance spectra of surfaces Red Yellow Blue Purple Wavelength (nm) Stephen E. Palmer, 2002

15 Color Image R G B

16 Color spaces How can we represent color?

17 Color spaces: RGB Default color space 0,,0 R (G=0,B=0),0,0 G (R=0,B=0) 0,0, Some drawbacks Strongly correlated channels Non-perceptual B (R=0,G=0) Image from:

18 Color spaces: HSV Intuitive color space H (S=,V=) S (H=,V=) V (H=,S=0)

19 Color spaces: YCbCr Fast to compute, good for compression, used by TV Y=0 Y=0.5 Y (Cb=0.5,Cr=0.5) Cr Cb Y= Cb (Y=0.5,Cr=0.5) Cr (Y=0.5,Cb=05)

20 Fourier Transform Any univariate function can be rewritten as a weighted sum of sines and cosines of different frequencies. 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 20/?

21 A sum of sines Our building block: Asin( x Add enough of them to get any signal g(x) you want!

22 Frequency Spectra example : g(t) = sin(2πf t) + (/3)sin(2π(3f) t) + = Slides: Efros

23 Fourier analysis in images Intensity Image Fourier Image

24 Signals can be composed + = More:

25 Overview Introduction What is Vision Processing? Basic Knowledge Image formation Transformation Low level Algorithms Filtering Border Detection Edge Detection Conclusion 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 25

26 Low Level Algorithms Filtering Simple Block matching. Border Detection Edge Detection 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 26

27 NEXT COSCO? Slide credit Fei Fei Li 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 27

28 Image Filtering Compute function of local neighborhood at each position. Why: Enhance images: Denoise, resize, increase contrast, so on. Extract information from images Texture, edges, distinctive points, so on. Detect patterns: Template matching 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 28

29 Image Filtering (2) g[, ] f [.,.] h[.,.] h[ m, n] g[ k, l] k, l f [ m k, n l] Credit: S. Seitz 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 29

30 Image Filtering (3) g[ h[.,.], ] f [.,.] h[ m, n] g[ k, l] k, l f [ m k, n l] Credit: S. Seitz 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 30

31 Image Filtering (4) g[, ] f [.,.] h[.,.] h[ m, n] g[ k, l] k, l f [ m k, n l] Credit: S. Seitz 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 3

32 Image Filtering (5) g[, ] f [.,.] h[.,.] h[ m, n] g[ k, l] k, l f [ m k, n l] Credit: S. Seitz 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 32

33 Example of Filters Original Shifted left By pixel Source: D. Lowe 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 33

34 Original Sharpening filter - Accentuates differences with local average Source: D. Lowe 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 34

35 Sharpening Source: D. Lowe 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 35

36 Sobel Filter: Vertical Vertical Edge (absolute value) 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 36

37 Sobel Filter: Horizontal Horizontal Edge (absolute value) 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 37

38 Gaussian Filter Weight contributions of neighboring pixels by nearness. Gaussian Filter is low pass filter x 5, = 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 38

39 Gaussian Filter 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 39

40 Box Filter 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 40

41 Filtering in spatial domain * = 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 4

42 Filtering in frequency domain FFT FFT = Inverse FFT Slide: Hoiem 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 42

43 Template matching Goal: find in image Main challenge: What is a good similarity or distance measure between two patches? Correlation Zero-mean correlation Sum Square Difference Normalized Cross Correlation 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 43

44 Matching with filters Goal: find in image Method : SSD h[ m, n] ( g[ k, l] f [ m k, n l]) k, l 2 True detections Input - sqrt(ssd) Thresholded Image

45 Matching with filters Goal: find in image Method 3: Normalized cross-correlation True detections Input Normalized X-Correlation Thresholded Image

46 Matching with filters Goal: find in image Method : Normalized cross-correlation 0.5, 2,, 2,, ) ], [ ( ) ], [ ( ) ], [ )( ], [ ( ], [ l k m n l k m n l k f l n k m f g l k g f l n k m f g l k g m n h Matlab: normxcorr2(template, im) mean image patch mean template

47 Compare SSD: faster, sensitive to overall intensity Normalized cross-correlation: slower, invariant to local average intensity and contrast Matching smaller / larger eyes? Image pyramid: down-sampling and matching 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 47

48 Down-sampling Image Gaussian Filter Low-Pass Filtered Image Sample Low-Res Image Source: Forsyth 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 48

49 Matching with Image pyramid Input: Image, Template. Match template at current scale 2. Downsample image 3. Repeat -2 until image is very small 4. Take responses above some threshold, perhaps with non-maxima suppression 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 49

50 Coarse-to-fine Image Registration. Compute Gaussian pyramid 2. Align with coarse pyramid 3. Successively align with finer pyramids Search smaller range Why is this faster? Are we guaranteed to get the same result?

51 Edge/Border Detection Goal: Identify sudden changes (discontinuities) in an image Intuitively, most semantic and shape information from the image can be encoded in the edges More compact than pixels Ideal: artist s line drawing (but artist is also using object-level knowledge) 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 5

52 surface normal discontinuity depth discontinuity surface color discontinuity illumination discontinuity Edges are caused by a variety of factors Source: Steve Seitz 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 52

53 Characterizing edges An edge is a place of rapid change in the image intensity function image intensity function (along horizontal scanline) first derivative edges correspond to extrema of derivative

54 Intensity profile Source: D. Hoiem

55 With a little Gaussian noise Gradient Source: D. Hoiem

56 Effects of noise Consider a single row or column of the image Plotting intensity as a function of position gives a signal Source: S. Seitz

57 Solution f g f * g d dx ( f g) To find edges, look for peaks in ( f g) d dx 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 57

58 Derivative theorem of convolution Differentiation is convolution, and convolution is associative: d d ( f g) f g dx dx This saves us one operation: f d dx g f d dx g Source: S. Seitz

59 Canny edge detector This is probably the most widely used edge detector in computer vision Theoretical model: step-edges corrupted by additive Gaussian noise Canny has shown that the first derivative of the Gaussian closely approximates the operator that optimizes the product of signalto-noise ratio and localization J. Canny, A Computational Approach To Edge Detection, IEEE Trans. Pattern Analysis and Machine Intelligence, 8:679-74, 986. Source: L. Fei-Fei

60 Derivation of Gaussian * [ -] = 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 60

61 Derivative of Gaussian filter x-direction y-direction

62 Example original image (Lena)

63 Compute Gradients (DoG) X-Derivative of Gaussian Y-Derivative of Gaussian Gradient Magnitude

64 Hysteresis thresholding Threshold at low/high levels to get weak/strong edge pixels Do connected components, starting from strong edge pixels

65 Hysteresis thresholding Check that maximum value of gradient value is sufficiently large drop-outs? use hysteresis use a high threshold to start edge curves and a low threshold to continue them. Source: S. Seitz

66 Final Canny Edges

67 pb boundary detector Martin, Fowlkes, Malik 2004: Learning to Detect Natural Boundaries S/vision/grouping/papers/mfm-pami-boundary.pdf Figure from Fowlkes

68 pb Boundary Detector Figure from Fowlkes

69 Brightness Color Texture Combined Human

70 45 years of boundary detection 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 70

71 Conner Detection For matching, interesting points are most concerned. Most of interesting points are conner: 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 7

72 Corner Detection: Basic Idea We should easily recognize the point by looking through a small window Shifting a window in any direction should give a large change in intensity Source: A. Efros flat region: no change in all directions edge : no change along the edge direction corner : significant change in all directions

73 Corner Detection: Mathematics Change in appearance of window w(x,y) for the shift [u,v]: xy, 2 E( u, v) w( x, y) I( x u, y v) I( x, y) I(x, y) E(u, v) E(3,2) w(x, y)

74 Interpreting the second moment matrix Consider a horizontal slice of E(u, v): [ u v] M u v This is the equation of an ellipse. Diagonalization of M: M R 0 R 0 2 The axis lengths of the ellipse are determined by the eigenvalues and the orientation is determined by R const direction of the fastest change direction of the slowest change ( max ) -/2 ( min ) -/2

75 Interpreting the eigenvalues Classification of image points using eigenvalues of M: 2 Edge 2 >> Corner and 2 are large, ~ 2 ; E increases in all directions and 2 are small; E is almost constant in all directions Flat region Edge >> 2

76 Corner response function R det( M ) trace( M ) 2 ( ) α: constant (0.04 to 0.06) Edge R < 0 Corner R > 0 Flat region R small Edge R < 0

77 Harris corner detector ) Compute M matrix for each image window to get their cornerness scores. 2) Find points whose surrounding window gave large corner response (f> threshold) 3) Take the points of local maxima, i.e., perform non-maximum suppression C.Harris and M.Stephens. A Combined Corner and Edge Detector. Proceedings of the 4th Alvey Vision Conference: pages 47 5, 988.

78 Harris Detector [Harris88] Second moment matrix 2 I x ( D ) I xi y ( D ) ( I, D) g( ) 2. Image I I xi y ( D ) I y ( D ) derivatives (optionally, blur first) I x I y det M trace M Square of derivatives 3. Gaussian filter g( I ) I x 2 I y 2 I x I y g(i x2 ) g(i y2 ) g(i x I y ) 4. Cornerness function both eigenvalues are strong 2 har det[ (, )] [trace( (, )) ] g I D ( I x ) g( I y ) [ g( I xi y)] [ g( I x ) g( I y I D )] 2 5. Non-maxima suppression har 78

79 Harris Detector: Steps

80 Harris Detector: Steps Compute corner response R

81 Harris Detector: Steps Find points with large corner response: R>threshold

82 Harris Detector: Steps Take only the points of local maxima of R

83 Harris Detector: Steps

84 Conclusion Vision is the act of knowing what is where by looking. Aristotle The motivation of vision processing is let computer understand the image/video. To do it, the image is firstly analyzed to distract the feature points. In this presentation: Basic knowledge for vision processing. Filtering/Matching. Border/Edge detection. Conner detection. For more advanced processing, we need further algorithm and analysis. 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 84

85 References CS 43 Introduction to Computer Vision - James Hays ; Computer Vision: Algorithms and Applications - Richard Szeliski 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 85

86 Thank you for your attention! Let s do Question and Answer!!! 6 Oct. 205 COSCO V Low-level Vision Processing Algorithms 86