COMP_4190 Artificial Intelligence Computer Vision. Computer Vision. Levels of Abstraction. Digital Images

Size: px

Start display at page:

Download "COMP_4190 Artificial Intelligence Computer Vision. Computer Vision. Levels of Abstraction. Digital Images"

Clare Blanche Hopkins
5 years ago
Views:

1 COMP_49 Artificial Intelligence Computer Vision Jacky Baltes Department of Computer Science University of Manitoba Winnipeg, Manitoba Canada, RT N jacky@cs.umanitoba.ca Introduction Digital images Edge detection Segmentation Watershed algorithm Snakes Hough transform detect lines general Hough transform Computer Vision Scene interpretation Waltz Algorithm Digital Images Levels of Abstraction Computer graphics: model -> image Computer vision: image -> model Two dimensional array of picture elements (pixels) Captured through a sensor camera, scanner structured lighting Lowest level: Pre-processing, noise removal, edge detection Middle level: Segmentation into regions High level: Object recognition, motion analysis World model

2 Optical Illusions Optical Illusions PPM File Format We will use the PPM file format in this course (easy,lossless) Colour format with three channels (red, green, blue) ASCII Header P Magic number width height # of colours per channel (depth) Followed by width * height * log(depth) * bytes P PPM File Format <><D9><D6><D7><D4><D4><D6><D7><DA ><DA><E><DB><E6><D8><E6><D6><E5>< D6>

3 Edge Detection Colours are very susceptible to lighting An important pre-processing step is edge detection How do we find an edge? Sharp contrast in the image Derivative of the image function I(i,j) Approximate derivative with I(i+,j) - I(i-,j) Edge Detection and many other image preprocessing steps can be implemented as a convolution A convolution mask is a matrix that is applied to each pixel in the image Specifies weights of the neighbors Convolution Derivative - + Convolution What is the output of [, 55,,,... ] -55? Use divisor and offset to normalize result of convolution to.. 55 How to deal with colour images? Convert to grey scale Handle each channel seperately Sobel Edge Detection To reduce noise, average over several rows Weigh rows differently Divisor = 8, Offset =

Sobel Edge Detection Use seperate convolution matrices for horizontal and vertial - + - + - + Horizontal - - - + + + Vertical Convolution Many other filters can be implemented efficiently as

4 Sobel Edge Detection Use seperate convolution matrices for horizontal and vertial Horizontal Vertical Convolution Many other filters can be implemented efficiently as convolution One problem: what to do at the borders What does the following filter do? Blurring (Simple) Blurring is used to reduce noise in the image Template Matching Convolution Find Specific features in the image Divisor = 4

5 Image Segmentation Segmentation Full segmentation: Individual objects are separated from the background and given individual ID numbers (labels). Partial segmentation: The amount of data is reduced (usually by separating objects from background) to speed up the further processing. Segmentation is often the most difficult problem to solve in the process; there is no universal solution! The problem can be made much easier if solved in cooperation with the constructor of the imaging system (choice of sensors, illumination, background etc). Three Types of Segmentation Classification Based on some similarity measure between pixel values. The simplest form is thresholding. Edge-based Search for edges in the image. They are then used as borders between regions Region-based Region growing, merge & split Common idea: search for discontinuities or/and similitudes in the image Thresholding (Global and Local) Global: based on some kind of histogram: grey-level, edge, feature etc. Lighting conditions are extremely important, and it will only work under very controlled circumstances. Fixed thresholds: the same value is used in the whole image Local (or dynamic thresholding): depends on the position in the image. The image is divided into overlapping sections which are thresholded one by one.

6 Classical Automatic Thresholding Algorithm. Select an initial estimate for T. Segment the image using T. This produces groups: G, pixels with value >T and G, with value <T Optimal Thresholding Based on the shape of the current image histogram. Search for valleys, Gaussian distributions etc.. Compute µ and µ, average pixel value of G and G 4. New threshold: T=/(µ +µ ) 5. Repeat steps to 4 until T stabilizes. Very easy + very fast Assumptions: normal dist. + low noise Histograms Thresholding and illumination Solutions: Calibration of the imaging system Percentile filter with very large mask Morphological operators

MR non-uniformity More thresholding Can also be used on other kinds of histogram: grey-level,

Multivariate data ( see next lectures) Problems: Only considers the graylevel pixel value, so it

Solution: post-processing with morphological operators Requires strong assumptions to be efficient

procedure Based on finding discontinuities (local variations of image intensity).

7 MR non-uniformity More thresholding Can also be used on other kinds of histogram: grey-level, edge, feature etc. Multivariate data ( see next lectures) Problems: Only considers the graylevel pixel value, so it can leave holes in segmented objects. Solution: post-processing with morphological operators Requires strong assumptions to be efficient Local thresholding is better see region growing techniques Edge-based Segmentation Gradient based procedure Based on finding discontinuities (local variations of image intensity). Apply an edge detector ex gradient operator (Sobel) second derivative (Laplace) 4. Threshold the edge image to get a binary image 5. Depending on the type of edge detector: Link edges together to close shapes (using edge direction for example) Remove spurious edges

8 Zero-crossing based procedure Laplacian of Gaussian Edge-based Segmentation: examples Region based segmentation Work by extending some region based on local similarities between pixels region growing (bottom-up method) region splitting and merging (top-down method) Bottom-up: from data to representation Top-down: from model to data

Include neighbouring pixels with similar

9 Region growing: Bottom Up Method. Find starting points. Include neighbouring pixels with similar features (greylevel, texture, color). 5. Continue untill all pixels have been included with one of the starting points. Region Growing: Watershed Algorithm Think of the grey-level image as a landscape. Let water rise from the bottom of each valley (the water from each valley is given its own label). As soon as the water from two valleys meet, build a dam, or watershed. These watersheds will then define the borders between different regions. Problems: Not trivial to find good starting points, difficult to automate Need good criteria for similarity.

Watershed: Problems and Solutions Oversegmentation Watershed from markers Computation new algorithm for fast watershed Graylevel might not be the optimal choice as the local similarity measure bigger

10 Watershed: Problems and Solutions Oversegmentation Watershed from markers Computation new algorithm for fast watershed Graylevel might not be the optimal choice as the local similarity measure bigger neighborhood when growing other local features (statistical, edge enhanced image, distance transformed image ) Region Growing: Split & Merge. ) Set up some criteria for what is a uniform area (ex mean, variance, bimodality of histogram, texture, etc ).) Start with the full image and split it in to 4 sub-images.) Check each sub-image. If not uniform, divide into 4 new sub-images 4.) After each iteration, compare touching regions with neighboring regions end merge if uniform The method is also called "quadtree" decomposition/division (and is also used for compression) Split & Merge The Hough transform A method for finding global relationships between pixels. Example: We want to find straight lines in an image. Apply edge enhancing filter (ex Laplace). Set a threshold for what filter response is considered a true edge pixel. Extract the pixels that are on a straight line using the Hough transform

Finding straight lines: The Hough transform The Hough transform. consider a pixel in position (xi,yi). equation of a straight line yi=axi+b. set b=-axi+ yi and draw this (single) line in ab-space 4.

11 Finding straight lines: The Hough transform The Hough transform. consider a pixel in position (xi,yi). equation of a straight line yi=axi+b. set b=-axi+ yi and draw this (single) line in ab-space 4. consider the next pixel with position (xj,yj) and draw the line b=axj+ yj ab-space (also called parameter space). The points (a,b ) where the two lines intersect represent the line y=a x+b in xy-space which will go through both (xi,yi) and (xj,yj). 5. draw the line in ab-space corresponding to each pixel in xy-space. 6. divide ab-space into accumulator cells and find most common (a, b ) which will give the line connecting the largest number of pixels The Hough transform In reality we have a problem with y=ax+b because a reaches infinity for vertical lines. Use instead. The Hough transform It is common to use filters for finding the intersection: butterfly filters Different variations of the Hough transform can also be used for finding other shapes of the form g(v,c)=, v is a vector of coordinates, c is a vector of coefficients. Possible to find any kind of simple shape ex. circle: (D parameter space) xc yc c

Conclusions The segmentation procedure Pre-processing Segmentation Snakes Example: segmentation of the brain in MRI

important User interaction Snake after initialization Snake at equilibrium 46 Snakes (active contours) Internal Energy A snake

s ranges from to Each position is associated to an energy: E snake E int v sds E ext v sds The final position corresponds to

12 Conclusions The segmentation procedure Pre-processing Segmentation Snakes Example: segmentation of the brain in MRI Post-processing Like any IP procedure There exists NO universal segmentation method Evaluation of segmentation performance is important User interaction Snake after initialization Snake at equilibrium 46 Snakes (active contours) Internal Energy A snake is an active contour parametrically represented by its position v(s)=(x(s), y(s)). s ranges from to Each position is associated to an energy: E snake E int v sds E ext v sds The final position corresponds to the minimum of the energy The internal energy of the snake is due to bending and it is associated with a priori constraints: (s) controls the tension of the contour (s) controls its rigidity

External Energy Applications The external energy depends on the image and accounts for a posteriori

commonly used to attract snakes towards edges is: Applications Considerations The number of nodes is an

If we want a closed contour, we set the first and the last point equal.

It may be necessary to allow a snake contour to divide into two contours, or two contours to merge into

13 External Energy Applications The external energy depends on the image and accounts for a posteriori information Several energy forms have been proposed based on features of interest in the image An energy commonly used to attract snakes towards edges is: Applications Considerations The number of nodes is an important factor for the behavior of the snake. Ability to resample the contour may be necessary. If we want a closed contour, we set the first and the last point equal. Anchor points are necessary to keep the snake in position if the image forces are not enough. It may be necessary to allow a snake contour to divide into two contours, or two contours to merge into one contour. Different applications may need different potential functions and different settings of the control parameters (damping, tension and rigidity). 5

Applications More segmentation Tracking of a moving object An initial estimate for the contour (e.g. interactively defined) is used in the first frame.

Reconstruction from serial sections The user draws an approximate contour in the first slice. The contour at equilibrium is used as the starting contour in the next slice.

14 Applications More segmentation Tracking of a moving object An initial estimate for the contour (e.g. interactively defined) is used in the first frame. The contour at equilibrium is used as the starting contour for the next frame. The snake locks on to the object. Reconstruction from serial sections The user draws an approximate contour in the first slice. The contour at equilibrium is used as the starting contour in the next slice. The D object is reconstructed from the contours using triangulation... 5 Important in Image Processing in general: If you can use expert knowledge (user interaction, modelling, ) at relatively low cost (development, computational, ) JUST DO IT!! Based on contours: chain codes poligonal approximation signature Fourier descriptors, Based on regions: area, PA, topology moments, Shape descriptions Representation schemes based on boundary or region? reconstruction? stable under scaling, rotation, and translation? recognition by incomplete representation? Goal: description of im age inform ation suitable to use for classification, recognition, interpretation

Chain coding Example of chain coding boundary as a sequence of straight lines 4- or 8-connectedness 4-connectedness 8-connectedness 4-code: 8-code: 7 7 7 6 5 4 4 5 4 Problems with chain coding More

15 Chain coding Example of chain coding boundary as a sequence of straight lines 4- or 8-connectedness 4-connectedness 8-connectedness 4-code: 8-code: Problems with chain coding More stable Long code Dependent on Scale Starting point Rotation Small change in the boundary (noise or bad segmentation) can generate a chain code that does not reflect the shape of the object Rescale Starting point treat as a circular sequence define as starting point, the point giving the least magnitude Rotation use difference code

Codes from different starting points A B B x B o B 4 5 7 7 5 5 5 7 A x 4 5 7 7 5 5 5 7 A o 4 5 7 7 5 5 5 7 A Shape number Object descriptor by chain coded edge

..... but dependent on orientation of grid Shape numbers of order 8 order 4 shape no.

16 Codes from different starting points A B B x B o B A x A o A Shape number Object descriptor by chain coded edge Difference code Starting point giving least magnitude Number of digits gives the order rotation independent & unique but dependent on orientation of grid Shape numbers of order 8 order 4 shape no. difference chain code order 6 order 8 Basic rectangle diameter for a boundary B is defined as diam(b)=max(d(p i,p j )), p i,p j B length orientation diameter defines major axis minor axis perpendicular to major axis basic rectangle minimal enclosing rectangle using minor & major

Shape number using basic rectangle Shape order n Choose rectangle of order n best approximating the basic rectangle Rectangle gives gridsize n=: 4,, 5 a unique shape number Polygonal approximation

17 Shape number using basic rectangle Shape order n Choose rectangle of order n best approximating the basic rectangle Rectangle gives gridsize n=: 4,, 5 a unique shape number Polygonal approximation Minimum perimeter polygon represent the boundary with a polygon reduction of discretization effects idea: find a simple polygon that reflects the shape of the boundary fit straight lines to boundary often not trivial and time consuming original minimum perimeter

Polygon: merge, split Signature D functional representation of the boundary simplest approach: distance from centroid as function of angle D boundary D function problems with rotation and scaling

18 Polygon: merge, split Signature D functional representation of the boundary simplest approach: distance from centroid as function of angle D boundary D function problems with rotation and scaling select centroid select start point, e.g., as furthest for centroid rescale function, e.g., so values[,] Signatures for two boundaries Boundary Segments decomposition of boundary into segments reduce complexity simplify description often of interest in case of concavities use decomposition based on convex hull

Curvature Using Curvature Information rate of change of slope Bending energy for boundary of length L s(t)/ t difficult in the discrete case (= images) due to uneven boundary use boundary segments n

19 Curvature Using Curvature Information rate of change of slope Bending energy for boundary of length L s(t)/ t difficult in the discrete case (= images) due to uneven boundary use boundary segments n n energy necessary to bend a rod to desired shape... sum of squared curvature c(k) L BE c k L k fit straight line define as change of slope for n Decomposition based on points of extreme curvature Fourier descriptors represent boundary (of K pixels) by a sequence of coordinates s(k)=(x(k),y(k)), k=,,,...,k- complex number s(k)=x(k)+iy(k) D D discrete Fourier Transform (DFT) Fourier descriptors reconstruction reconstruction of boundary using inverse DFT approximation of boundary using only P first Fourier coefficients gives the same number of points in boundary remember: high-frequency for fine details low-frequency for global shape a(u) Fourier descriptors of the boundary

Properties of Fourier descriptors Measures length of boundary perimeter transformation identity boundary s(k) fourier descriptor a(u) count pixels ROUGH approximation chain code simple representation

20 Properties of Fourier descriptors Measures length of boundary perimeter transformation identity boundary s(k) fourier descriptor a(u) count pixels ROUGH approximation chain code simple representation a x # edge steps + b x # point steps b a rotation translation scaling starting point s r (k)=s(k)e j s t (k)=s(k)+ xy s s (k)=s(u) s p (k)=s(k-k ) a r (u)=a(u)e j a t (u)=a(u)+ xy (u) a s (k)=a(u) a p (u)=a(u)e -jk u/k diameter: largest distance between two pixels on the boundary length direction xy =x+ jy Area Compactness Num ber of pixels P /A: perimeter x perimeter / area dimensionless minimal for disc rotation invariant blue = green = 4 compact not compact

P /A Examples of P /A measurements perim eter x

perim eter = 798 P /A= 6.4, P /A norm = 4.

8 Eccentricity longest chord/max perpendicular

high/width of the minimal bounding rectangle

21 P /A Examples of P /A measurements perim eter x perim eter / area norm alization: area = 58, perim eter = 798 P /A= 6.4, P /A norm = 4.8 area = 59, perim eter = P /A=.6, P /A norm =.8 Eccentricity longest chord/max perpendicular chord Elongatedness. high/width of the minimal bounding rectangle there are different definitions! 7. area/(d ) d is maximum width 8. longest possible path OK not OK

area of region/ area of bounding rectangle

of connected components H number of holes Invariant

Projections convex region R for any x,x R, straight

region R smallest convex set containing R convex

22 area of region/ area of bounding rectangle Rectangularity Euler number Topology E=C-H C number of connected components H number of holes Invariant under rubber sheet transformation Convex hull Projections convex region R for any x,x R, straight line between x and x is in R convex hull H of a region R smallest convex set containing R convex deficiency D p h i j f i, j R=green H=green +brown D=brown p v j i f i, j

23 Principle component analysis (PCA) points as D vectors (a,b) a x-coordinate b y-coordinate mean vector and covariance matrix find eigenvectors e and e rotate Moments Given a D discrete function f(x,y) moment of order p+ q central moment of order p+ q centre of mass Hotelling transform Measurements with moments Example: Cell Classification µ horizontal centralness µ vertical centralness µ diagonality indicates with respect to centroid where R has more mass µ horizontal divergence indicates the relative extent of the left of R compared to the right µ vertical divergence indicates the relative extent of the right of R compared to the left µ horizontal imbalance location of center of gravity with respect to half horizontal extent µ vertical imbalance location of center of gravity with respect to half vertical extent n,,n6 = normal Shapes differing in size, orientation, border irregularity d,,d6 = diabetic

24 Waltz Algorithm One of the earliest examples of a constraint satisfaction problem Interpret line drawings from solid polyhedra Look at all intersections Waltz's Algorithm What type of intersection is this? Concave intersection of three planes Externel convex intersection Adjacent intersections impose constraints on each other CSP to find a unique set of labels First step in image understanding Waltz Algorithm Waltz Algorithm Assumptions No shadows, no cracks Three-faced vertices General position: no junctions that move with small movement of the eye Then each line in the images is Boundary line of the object (right hand = solid, left hand = outside) Convex edge (+) Concave edge (-) Labeling of the edges

25 8 legal types of junctions Waltz Algorithm Types of Junctions Label each junction as one of those 8 Both ends of a line must be consistent (same label) Reformulate as a CSP. Constraint propagation always works perfectly Waltz Examples References Les Kitchen (Lecture Slides) Lucia Ballerini (Lecture slides) Andrew Moore's lecture slides (CMU) pdf

Digital Image Processing

Digital Image Processing Part 9: Representation and Description AASS Learning Systems Lab, Dep. Teknik Room T1209 (Fr, 11-12 o'clock) achim.lilienthal@oru.se Course Book Chapter 11 2011-05-17 Contents