Image Processing and Image Analysis VU

Size: px

Start display at page:

Download "Image Processing and Image Analysis VU"

Oswald Randall
6 years ago
Views:

1 Image Processing and Image Analysis VU Torsten Möller + Hrvoje Bogunovic + Georg Langs + Yll Haxhimusa torsten.moeller@univie.ac.at / hrvoje.bogunovic@meduniwien.ac.at / georg.langs@meduniwien.ac.at / yll.haxhimusa@medunwien.ac.at vda.univie.ac.at/teaching//17w/

2 Today Administration Content of the course Short summary of the SIP 2

3 Meeting place PC Unterrichtsraum 1, Währinger Straße 29 1.UG Meeting time Tue + Thu 9:45 11:15 Pre requisites In order to take this class you need to have basic signal processing knowledge. Hence, Signal and Image Processing (SIP, ) is a pre requisite. 3

4 Syllabus See web page: vda.univie.ac.at/teaching//17w/ 4

5 Content Edge detection and thresholding Region based segmentation (region growing, graph cuts) Active contours: Snakes and level sets Feature extraction Image pattern classification Registration Shape classification 5

6 Definition of Image Analysis Also known as Machine Vision or Computer Vision. Definition from Wikipedia: a subfield of artificial intelligence. The purpose of computer vision is to program a computer to understand a scene or features in an image. Holy grail : Make machine see! 6

7 Makes use of image processing, pattern recognition, machine learning, (medical) expert knowledge, Graphics Image processing Artificial intelligence Computer vision Algorithms Machine learning Cognitive science 7

8 Movies, news, sports Surveillance and security Medical and scientific images 8

9 Why vision? As image sources multiply, so do applications Relieve humans of boring, easy tasks Enhance human abilities: human computer interaction, visualization Perception for robotics / autonomous agents Organize and give access to visual content 9

10 Photosynth Based on Photo Tourism technology developed by Noah Snavely, Steve Seitz, and Rick Szeliski 10

recognition, AT&T labs http://www.research.att.

11 Optical character recognition(ocr) Technology to convert scanned docs to text If you have a scanner, it probably came with OCR software Digit recognition, AT&T labs License plate readers 11

12 Face detection Many new digital cameras now detect faces Canon, Sony, Fuji, your smart phone 12

13 Face recognition Who is she? 13

14 Vision based biometrics How the Afghan Girl was Identified by Her Iris Patterns Read the 14

15 Login without a password Fingerprint scanners on many new laptops, other devices Face recognition systems now beginning to appear more widely 15

16 Special effects: shape capture The Matrix movies, ESC Entertainment, XYZRGB, NRC 16

17 Special effects: motion capture Pirates of the Carribean, Industrial Light and Magic 17

18 Sports Sportvision first down line Nice explanation on 18

19 Smart cars Slide content courtesy of Amnon Shashua Mobileye Vision systems currently in high end BMW, GM, Volvo More and more into the low end cars 19

Vision in space NASA'S Mars Exploration Rover Spirit captured this westward view from atop a low plateau where Spirit spent the closing months of 2007.

20 Vision in space NASA'S Mars Exploration Rover Spirit captured this westward view from atop a low plateau where Spirit spent the closing months of Vision systems (JPL) used for several tasks Panorama stitching 3D terrain modeling Obstacle detection, position tracking For more, read Computer Vision on Mars by Matthies et. al. 20

21 Robotics NASA s Mars Spirit Rover

22 Medical imaging 3D imaging MRI, CT Image guided surgery Grimson et al., MIT 22

23 Current state of the art This is a very active research area, and rapidly changing Many new apps in the years to come To learn more about vision applications and companies David Lowe maintains an excellent overview of vision companies 23

24 More: Readings & Resources Main text for this class: Gonzalez and Woods Digital Image Processing, 4th edition Pearson/Prentice Hall Richard Szeliski Computer Vision: Algorithms and Applications Springer Sonka, V. Hlavac, and R. Boyle Image Processing, Analysis, and Machine Vision 4th Edition Cengage Learning,

25 Short Summary of SIP 25

26 Image Formation 26 Slide credit: Derek Hoiem

27 Eye Sampling 27

28 Digital camera A digital camera replaces film with a sensor array Each cell in the array is light sensitive diode that converts photons to electrons camera.htm 28 Slide by Steve Seitz

29 Digital images Sample the 2D space on a regular grid Quantize each sample (round to nearest integer) Image thus represented as a matrix of integer values. f(x,y) y x 2D 1D 29 Adapted from S. Seitz

30 Sampling: Number and density of pixel measurements Quantization: Number of levels permitted in pixel values. 30

31 Image digitisation A CCD camera samples a scene as a set of points (pixels). 31

32 Rather than a strict point measurement, each pixel value usually represents the mean value of the sensed signal averaged over a sampling window. Note. Very low resolution sampling of this image! 32

33 Digital images 33 Slide credit: Derek Hoiem

D f Greyscale image formal definition: A greyscale image f is a mapping of a subset D f of Z n,

34 Types of Digital Images Binary Greyscale Binary image formal definition: A binary image f is a mapping of a subset D f of Ζ n, called the definition domain of f, into the couple {0, 1}: f : n 0,1 D f Greyscale image formal definition: A greyscale image f is a mapping of a subset D f of Z n, called the definition domain of f, into a bounded set of nonnegative n integers N 0 : f : D Z 0,1,, t f max 34

35 Types of Digital Images t max is the maximum value of the data type used for storing the image (e.g. 2 n 1 for pixel values coded on n bits). A Digital Elevation Model (DEM) is a digital image in which each pixel greyscale represents the height of the land. For certain operations, it is useful to consider a greyscale image as a DEM. DEM of a small mountain range in Alaska 35

36 DEM 36

37 Types of Digital Images A multichannel image (also called multiband image) consists of an array of monochannel images (either binary or greyscale) defined over a common definition domain. It follows that a vector of scalar values is associated with each pixel of a multichannel image. Denoting by f a multichannel image with m channels, the values of each pixel p of the definition domain of f define an m dimensional vector: f(p) = ( f 1 (p), f 2 (p),, f m (p)) 37

Images optically acquired in more than one

38 Images optically acquired in more than one spectral or wavelength interval are called multispectral. Example: SPOT satellite image of the Rhone valley over the city of Arles. 1 st channel for wavelengths m 2 nd channel for wavelengths m 3 rd channel for wavelengths m 38

39 Types of Digital Images Colour images are multispectral images containing three channels, one for each primary colour in the red green blue (RGB) colour model. Colour image f (p) [ {0,, t max }, {0,, t max }, {0,, t max } ] 39

40 Digital color images Color images, RGB color space R G B 40

41 Types of Digital Images Hyperspectral images have a very high number of channels. Four images of a 31 channel hyperspectral image (each channel covers a wavelength range of 10nm). 41

42 Types of Digital Images All image types described so far can be extended to the class of multitemporal images (also called image sequences). They consist of a series of images defined over the same definition domain but collected at more then a single time. 42

43 Image to image transformations The transformed image has the same definition domain as the input image. These are divided into two groups: Point image transformations: the output value of a given pixel p is a function of the input value of this very pixel, i.e. without taking into account the values of other pixels. Neighbourhood image transformations (image filtering): the output value at a given pixel is a function of the values of the pixels falling within a neighbouring region centred on the considered pixel. 43

44 Histograms The greyvalue histogram H f (z) of an image provides the frequency of the greyvalue z in the image f : horizontal axis greyvalues z vertical axis number of pixels with greyvalue z 44

f Point image transformations Threshold operator: Sets all pixels lying in a given range of greyvalues to the value 1 and the remaining ones to the value 0, i.e. it maps greyscale images into binary image.

45 f Point image transformations Threshold operator: Sets all pixels lying in a given range of greyvalues to the value 1 and the remaining ones to the value 0, i.e. it maps greyscale images into binary image. In mathematical terms, the threshold operator T operates on all pixels x of the input image f as follows: T t i, t j f x 1 0 if t i f otherwise x t j T [130, 255] ( f ) 45

46 Greyscale transformation A transformation T of pixel brightness, from the original brightness p in the range [p 0, p k ] into brightness q in the range [q 0, q k ] is given by: q = T(p) A simple use is in inverting an image: T i (p) = p 0 + p k p 46

47 Contrast stretching Let the maximum possible greyvalue range of an image be given by [q 0, q k ] (for an 8 bit image, q 0 = 0, q k = 255). The greyvalue range of a certain image is narrower than the above range and is given by [p 0, p k ]. In practice, these are obtained by scanning all the pixels in the image. To stretch the range of greyvalues so that it fills the maximum possible range, we use the contrast stretching operator: T c q p q p k 0 p p p0 q0 k 0 47

48 Contrast stretching example: from HIPR: 48

49 Histogram equalisation Aim: Create an image with equally distributed brightness levels over the whole brightness scale. H(p) G(q) p q We have: Input histogram H(p) with range [p 0, p k ]. Output histogram G(q) with range [q 0, q k ]. This histogram must be uniform. We aim to find a monotonic pixel brightness transformation q = T e (p). The histogram can be treated as a discrete probability density function. 49

50 Histogram equalisation example: Histogram equalisation For comparison, histogram stretching from HIPR: 50

51 Neighbourhood image transformations (Spatial) Correlation (Spatial) Convolution Morphological Operators 51

52 Image filtering Compute a function of the local neighborhood at each pixel in the image Function specified by a filter or mask saying how to combine values from neighbors. Uses of filtering: Enhance an image (denoise, resize, etc) Extract information (texture, edges, etc) Detect patterns (template matching) 52 Adapted from Derek Hoiem

53 Motivation: noise reduction Even multiple images of the same static scene will not be identical. 53

Common types of noise Salt and pepper noise: random occurrences of black and white pixels Impulse noise: random occurrences of white pixels Gaussian

54 Common types of noise Salt and pepper noise: random occurrences of black and white pixels Impulse noise: random occurrences of white pixels Gaussian noise: variations in intensity drawn from a Gaussian normal distribution Original Salt and pepper noise Impulse noise Gaussian noise 54 Source: S. Seitz

55 Gaussian noise 55 Fig: M. Hebert

56 Motivation: noise reduction Even multiple images of the same static scene will not be identical. How could we reduce the noise, i.e., give an estimate of the true intensities? What if there s only one image? 56

57 First attempt at a solution Let s replace each pixel with an average of all the values in its neighborhood Assumptions: Expect pixels to be like their neighbors Expect noise processes to be independent from pixel to pixel 57

58 First attempt at a solution Let s replace each pixel with an average of all the values in its neighborhood Moving average in 1D: 58 Source: S. Marschner

59 Weighted Moving Average Can add weights to our moving average Weights [1, 1, 1, 1, 1] / 5 59 Source: S. Marschner

60 Weighted Moving Average Non uniform weights [1, 4, 6, 4, 1] / Source: S. Marschner

61 Moving Average In 2D Source: S. Seitz

62 Moving Average In 2D Source: S. Seitz

63 Moving Average In 2D Source: S. Seitz

64 Moving Average In 2D Source: S. Seitz

65 Moving Average In 2D Source: S. Seitz

66 Moving Average In 2D Source: S. Seitz

67 Correlation filtering Say the averaging window size is 2k+1 x 2k+1: Attribute uniform weight to each pixel Loop over all pixels in neighborhood around image pixel F[i,j] Now generalize to allow different weights depending on neighboring pixel s relative position: Non-uniform weights 67

68 Correlation filtering This is called cross-correlation, denoted Filtering an image: replace each pixel with a linear combination of its neighbors. The filter kernel or mask H[u,v] is the prescription for the weights in the linear combination. 68

69 Averaging filter What values belong in the kernel H for the moving average example? ? box filter

70 Smoothing by averaging depicts box filter: white = high value, black = low value original filtered What if the filter size was 5 x 5 instead of 3 x 3? 70

71 Smoothing by averaging original filtered 71

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0 90 90 90 90 90 0 0 0 0 0

72 Gaussian filter What if we want nearest neighboring pixels to have the most influence on the output? Removes high frequency components from the image ( low pass filter ) This kernel is an approximation of a 2d Gaussian function: 72 Source: S. Seitz

73 Smoothing with a Gaussian 73

74 Gaussian filters What parameters matter here? Size of kernel or mask Note, Gaussian function has infinite support, but discrete filters use finite kernels σ = 5 with 10 x 10 kernel σ = 5 with 30 x 30 kernel 74

75 Gaussian filters What parameters matter here? Variance of Gaussian: determines extent of smoothing σ = 2 with 30 x 30 kernel σ = 5 with 30 x 30 kernel 75

76 Ganglion cells have receptive fields with a Mexican hat shape (recall) Excitory input from the centre region Inhibitory input from the surrounding region Algebraic sum 3-dimensional model of the complete receptive field. 76

77 We have seen this! from 77

78 Matlab >> hsize = 10; >> sigma = 5; >> h = fspecial( gaussian hsize, sigma); >> mesh(h); >> imagesc(h); >> outim = imfilter(im, h); % correlation >> imshow(outim); outim 78

79 Smoothing with a Gaussian Parameter σ is the scale / width / spread of the Gaussian kernel, and controls the amount of smoothing. for sigma=1:3:11 h = fspecial('gaussian, fsize, sigma); out = imfilter(im, h); imshow(out); pause; end 79

80 Spatial Convolution We have: Input image F Second image H whose origin is usually located at the centre of its definition domain D H. This definition domain is usually much smaller than that of F, often a 3 3 window with its origin at the centre pixel. The output of the convolution at a given pixel x of F is then defined as: F H x F x b H b b D H Notation for convolution operator 80

81 Convolution Convolution: Flip the filter in both dimensions (bottom to top, right to left) Then apply cross correlation F Notation for convolution operator H 81

82 Convolution vs. correlation Convolution Cross correlation Note the difference! For a Gaussian or box filter, how will the outputs differ? If the input is an impulse signal, how will the outputs differ? 82

83 Filtering examples: sharpening 83

84 Fourier transform recall Convolution and correlation can be done efficiently in the Fourier Domain. The Fourier transform of an image g( x, y) is: The process takes a complex valued function of x and y (images are complex valued functions with zero imaginary component) and returns a complex valued function of u and v. 84

85 The exponential can be rewritten: These terms are sinusoids on the x, y plane, whose orientation and frequency are given by u, v. We are representing the image as a sum of sinusoids. 85

The real term is constant when ux + vy is

These images show the real parts of the Fourier

86 The real term is constant when ux + vy is constant. The magnitude of the vector (u, v) gives a frequency. Its direction gives an orientation. These images show the real parts of the Fourier basis elements. The brightest point has value 1 and the darkest has value 0. Increasing values of u and v 86

87 The Fourier transform of a real function is complex, which is difficult to plot and visualise. Instead, we usually consider the phase and magnitude of the complex function. Curious fact: All natural images have about the same magnitude transform. Hence phase seems to matter, magnitude largely doesn t. Demonstration: Take two pictures, swap the phase transforms and convert the inverse Fourier transform. 87

88 Original Images Magnitude Phase 88

89 Reconstruction with zebra phase and cheetah magnitude Reconstruction with cheetah phase and zebra magnitude 89

90 Correlation and Convolution In the two dimensional case, the cross correlation of F with G is the same as the convolution of F with the 180 rotation of G. More generally, if the image G is symmetric in its origin, i.e. G(x) = G( x), the spatial convolution and cross correlation are identical transformations. If F and G are the same images, the crosscorrelation is referred to as the autocorrelation. 90

91 Filtering application: Hybrid Images Aude Oliva & Antonio Torralba & Philippe G Schyns, SIGGRAPH

92 Application: Hybrid Images Gaussian Filter Laplacian Filter unit impulse Gaussian Laplacian of Gaussian Aude Oliva & Antonio Torralba & Philippe G Schyns, SIGGRAPH

93 93 Aude Oliva & Antonio Torralba & Philippe G Schyns, SIGGRAPH 2006

94 94 Aude Oliva & Antonio Torralba & Philippe G Schyns, SIGGRAPH 2006

95 What is a good representation for image analysis? Fourier transform domain tells you what (textural properties), but not where. In space, this representation is too spread out. Pixel domain representation tells you where (pixel location), but not what. In space, this representation is too localized Want an image representation that gives you a local description of image events what is happening where. That representation might be just right. wavelets, gabor jets, etc. From Torralba 95

96 Bogunovic/Möller 96

97 Wavelets recall Wavelets are functions defined over a finite interval and having an average value of zero. In general, a family of representations using: hierarchical (nested) basis functions finite ( compact ) support basis functions often orthogonal fast transforms, often linear time Bogunovic/Möller 97

98 Wavelets recall Linear combination of wavelet basis functions Father wavelet or scaling function Characterizes basic wavelet scale Mother wavelet or wavelet function Characterizes basic wavelet shape Each wavelet has a characteristic location and scale Bogunovic/Möller 98

99 Wavelets: Common types Bogunovic/Möller 99

100 Wavelets recall Wavelet transforms are based on small wavelets with limited duration. The translated version wavelets locate where we concern. Whereas the scaled version wavelets allow us to analyze the signal at different scales. As we dilate and translate the mother wavelet we can see very low freq. components at large scale while very high frequency components can be located at small scale. A balance between time domain and frequency domain domain due to Heisenberg uncertainty: we cannot locate both time and frequency Bogunovic/Möller 100

101 Bogunovic/Möller 101

102 DWT Bogunovic/Möller 102

103 2D Multiresolution Analysis Bogunovic/Möller 103

104 Bogunovic/Möller 104

105 Summary Content of the Course and some examples Recall of SIP Next: Edge detection 105

106 How old are you?! Have a look at this Microsoft web page and have fun old.net/ 106

107 Many thanks to R. Szeliski, K. Grauman, Ch. M. Bishop, A. Torralba, Gonzales etal. and other mentioned This slides are for your personal usage only and can be used only for educational purpose! The copyrights of all the authors are aknowledged. 107

CS4442/9542b Artificial Intelligence II prof. Olga Veksler

CS4442/9542b Artificial Intelligence II prof. Olga Veksler Lecture 2 Computer Vision Introduction, Filtering Some slides from: D. Jacobs, D. Lowe, S. Seitz, A.Efros, X. Li, R. Fergus, J. Hayes, S. Lazebnik,