Image Processing and Image Analysis 052617 VU Yll Haxhimusa yll.haxhimusa@medunwien.ac.at vda.univie.ac.at/teaching/ipa/17w/
Outline What are grouping problems in vision? Inspiration from human perception Gestalt properties Bottom up segmentation Algorithms: Merging and splitting Mode finding and mean shift: k means, mean shift Graph based: normalized cuts Features: color, texture, IPA 2
Grouping in vision The whole is equal to the sum of its parts The whole is greater the sum of its parts Goals: Euclid Max Wertheimer Gather features that belong together Obtain an intermediate representation that compactly describes key image or video parts IPA 3
A simple segmentation problem 4 IPA 4
It can get a lot harder Brady, M. J., & Kersten, D. (2003). Bootstrapped learning of novel objects. J Vis, 3(6), 413-422 IPA 5
Examples of grouping in vision [Figure by J. Shi] Determine image regions [http://poseidon.csd.auth.gr/lab_research/latest/imgs/s peakdepvidindex_img2.jpg] Group video frames into shots Fg / Bg [Figure by Wang & Suter] Figure-ground [Figure by Grauman & Darrell] Object-level grouping IPA 6
Grouping in vision Goals: Gather features that belong together Obtain an intermediate representation that compactly describes key image (video) parts Top down vs. bottom up segmentation Top down: pixels belong together because they are from the same object Bottom up: pixels belong together because they look similar Hard to measure success What is interesting depends on the app. IPA 7
IPA 8
Muller Lyer illusion IPA 9
What things should be grouped? What cues indicate groups? IPA 10
Perceptual organization the processes by which the bits and pieces of visual information that are available in the retinal image are structured into the larger units of perceived objects and their interrelations Stephen E. Palmer, Vision Science, 1999 IPA 11
Gestalt Gestalt: whole or group Whole is greater than sum of its parts Relationships among parts can yield new properties/features Psychologists identified series of factors that predispose set of elements to be grouped (by human visual system) Gestalt principles There are hundreds of different grouping laws IPA 12
IPA 13
Familiar configuration IPA 14
Similarity IPA 15 http://chicagoist.com/attachments/chicagoist_alicia/geese.jpg, http://wwwdelivery.superstock.com/wi/223/1532/previewcomp/superstock_1532r-0831.jpg
Symmetry http://seedmagazine.com/news/2006/10/beauty_is_in_the_processingtim.php IPA 16
Common fate Image credit: Arthus-Bertrand (via F. Durand) IPA 17
Proximity http://www.capital.edu/resources/images/outside6_035.jpg IPA 18
Illusory/subjective contours Interesting tendency to explain by occlusion In Vision, D. Marr, 1982 IPA 19
IPA 20
Continuity, explanation by occlusion IPA 21
Grouping is a global process IPA 22
Grouping is a global process Occlusion is an important cue in grouping. IPA 23
but not too global IPA 24
Magritte, 1957 IPA 25
Figure ground IPA 26
Groupings by Invisible Completions IPA 27
IPA 28 In Vision, D. Marr, 1982; from J. L. Marroquin, Human visual perception of structure, 1976.
Familiarity IPA 29
Familiarity IPA 30
Influences of grouping Grouping influences other perceptual mechanisms such as lightness perception IPAhttp://web.mit.edu/persci/people/adelson/publications/gazzan.dir/koffka.html 31
Gestalt Gestalt: whole or group Whole is greater than sum of its parts Relationships among parts can yield new properties/features Psychologists identified series of factors that predispose set of elements to be grouped (by human visual system) Inspiring observations/explanations; challenge remains how to best map to algorithms. IPA 32
Image segmentation Has the goal of dividing an image into parts (regions) that have a strong correlation with objects or areas of the real world contained in the image (Sonka et al., 1999) For narrow domain applications, automatic image segmentation can produce very good results: IPA 33 Automatic segmentation of silver grains on a photographic plate (Beucher)
The goals of segmentation Separate image into coherent objects image human segmentation IPA 34 Source: Lana Lazebnik
The goals of segmentation Separate image into coherent objects Group together similar looking pixels for efficiency of further processing superpixels X. Ren and J. Malik. Learning a classification IPA model for segmentation. ICCV 2003. 35 Source: Lana Lazebnik
Partitions X 1 X 2 X 3 The usual aim of segmentation is to produce a partition of the image. X X 6 4 X 5 A partition is defined as a division of a set into nonempty disjoint sets which completely cover the set. Mathematically, a collection of sets X 1,, X n defines a partition of a set X if an only if: 1.for all X i (X), X i (no subset is empty); 2.for all X i, X j (X), i j X i X j = (subsets are disjoint); 3. i {X i X i (X) } = X (the subsets cover X). IPA 36 Each subset X i is referred to as a region of X. X Euclid
Examples of partitions IPA 37
For wider domain applications, such as general image understanding, automatic segmentation is not capable of separating out all image objects. This is because: Segmentation uses only criteria based on homogeneity or heterogeneity of image properties (colour, texture, ) No knowledge of objects is present. IPA 38
There is no correct segmentation, as can be seen by these segmentations, produced by a number of humans. From the Berkeley segmentation benchmark IPA 39
When asked to segment an image, humans automatically make use of contextual information. In this case, it is: Shadows are not objects, even if they have a strong boundary. Information about the shape of a horse there is no visible boundary between the foot and the shadow. IPA 40 From the Berkeley segmentation benchmark
Depending on how the homogeneity criteria are set up, the results of segmentation can be: As required Over segmented Under segmented IPA 41
Image segmentation: toy example 1 2 3 pixel count black pixels gray pixels white pixels input image intensity These intensities define the three groups. We could label every pixel in the image according to which of these primary intensities it is. i.e., segment the image based on the intensity feature. What if the image isn t quite so simple? Kristen Grauman IPA 42
input image pixel count intensity pixel count Kristen Grauman input image intensity IPA 43
pixel count input image intensity Now how to determine the three main intensities that define our groups? Kristen Grauman IPA 44
Segmentation by Thresholding The simplest segmentation method. Suitable if: objects do not touch each other; their greylevels are clearly distinct from the background greylevels. Many methods exist for choosing the threshold automatically based on histogram analysis. Smooth and then search for minima. Approximation of the histogram by normal distributions. etc. IPA 45
Example: Greylevel histogram based segmentation Noise free Low noise High noise From Spann IPA 46
Greylevel histogram based segmentation How do we characterise low noise and high noise? We can consider the histograms of our images For the noise free image, its simply two spikes at i=100, i=150 For the low noise image, there are two clear peaks centred on i=100, i=150 For the high noise image, there is a single peak two greylevel populations corresponding to object and background have merged IPA 47
Greylevel histogram based segmentation 2500.00 h(i) 2000.00 1500.00 1000.00 Noise free Low noise High noise 500.00 0.00 0.00 50.00 100.00 150.00 200.00 250.00 i IPA 48
Greylevel histogram based segmentation We can define the input image signal to noise ratio in terms of the mean greylevel value of the object pixels and background pixels and the noise standard deviation S / N b o IPA 49
Greylevel histogram based segmentation For our test images : S/N (noise free) = S/N (low noise) = 5 S/N (high noise) = 2 IPA 50
Greylevel thresholding We can easily understand segmentation based on thresholding by looking at the histogram of the low noise object/background image There is a clear valley between to two peaks IPA 51
Greylevel thresholding 2500.00 h(i) 2000.00 Background 1500.00 1000.00 Object 500.00 0.00 0.00 50.00 100.00 150.00 200.00 250.00 i T IPA 52
Greylevel thresholding We can define the greylevel thresholding algorithm as follows: If the greylevel of pixel p <=T then pixel p is an object pixel else Pixel p is a background pixel IPA 53
Greylevel thresholding This simple threshold test begs the obvious question how do we determine the threshold? Many approaches possible Interactive threshold Adaptive threshold Minimisation method IPA 54
Greylevel thresholding We will consider in detail a minimisation method for determining the threshold Minimisation of the within group variance Robot Vision, Haralick & Shapiro, volume 1, page 20 IPA 55
Greylevel thresholding Idealized object/background image histogram 2500.00 h(i) 2000.00 1500.00 1000.00 500.00 0.00 0.00 50.00 100.00 150.00 200.00 250.00 T i IPA 56
Greylevel thresholding Any threshold separates the histogram into 2 groups with each group having its own statistics (mean, variance) The homogeneity of each group is measured by the within group variance The optimum threshold is that threshold which minimizes the within group variance thus maximizing the homogeneity of each group Otsu s Thresholding method (1979) IPA 57
Greylevel thresholding Let group o (object) be those pixels with greylevel <=T Let group b (background) be those pixels with greylevel >T The prior probability of group o is p o (T) The prior probability of group b is p b (T) IPA 58
Greylevel thresholding The following expressions can easily be derived for prior probabilities of object and background p ( T) P( i) o i 0 255 p ( T) P( i) b where h(i) is the histogram of an N pixel image T i T 1 P(i ) h( i )/ N IPA 59
Greylevel thresholding The mean and variance of each group are as follows : o T ( T) ip(i)/ p ( T) i 0 255 b i T 1 ( T) ip( i)/ p ( T) T 2 2 o o i 0 ( T) i ( T) P(i)/ p ( T) 255 2 2 b b i T 1 ( T) i ( T) P( i)/ p ( T) o b o b IPA 60
Greylevel thresholding The within group variance is defined as : 2 ( T) 2 ( T) p ( T) 2 ( T) p ( T) W o o b b We determine the optimum T by minimizing this expression with respect to T Only requires 256 comparisons for and 8 bit greylevel image IPA 61
Greylevel thresholding h(i) T opt 2500.00 2000.00 1500.00 Histogram Within group variance 1000.00 500.00 0.00 0.00 50.00 100.00 150.00 200.00 250.00 i IPA 62
Greylevel thresholding We can examine the performance of this algorithm on our low and high noise image For the low noise case, it gives an optimum threshold of T=124 Almost exactly halfway between the object and background peaks We can apply this optimum threshold to both the low and high noise images IPA 63
Greylevel thresholding Low noise image Thresholded at T=124 IPA 64
Greylevel thresholding Low noise image Thresholded at T=124 IPA 65
Greylevel thresholding High level of pixel miss classification noticeable This is typical performance for thresholding The extent of pixel miss classification is determined by the overlap between object and background histograms. IPA 66
Greylevel thresholding p(x) 0.02 Background 0.01 Object 0.00 x o b T IPA 67
Greylevel thresholding p(x) 0.02 Background 0.01 Object 0.00 x o b T IPA 68
Greylevel thresholding Easy to see that, in both cases, for any value of the threshold, object pixels will be miss classified as background and vice versa For greater histogram overlap, the pixel miss classification is obviously greater We could even quantify the probability of error in terms of the mean and standard deviations of the object and background histograms IPA 69
Greylevel clustering Consider an idealized object/background histogram Background Object IPA 70 c 1 c 2
Edge based segmentation We look for borders separating regions in the image. Rely on edges found by edge detecting operators these edges mark discontinuities in greylevel, colour, texture, etc. These edges cannot be directly used as a segmentation Post processing steps combine edges into edge chains which correspond better with borders in the image. Common problems: caused by noise or unsuitable information in an image. Edge presence in locations where there is no border. no edge presence where a real border exists. IPA 71
Edge image thresholding Small edge values in gradient images correspond to non significant greylevel changes. Thresholding of an edge image can remove these small values. from Sonka et al. Original image Edge image (weak edges enhanced) IPA Edge image thresholded at 30 Edge image thresholded 72 at 10
Comments on edge based segmentation Some edge refinement methods are available, such as edge relaxation: The edge information within a neighbourhood is taken into account to increase or decrease the probability that edges are borders. use of border local information: if one has an assumption as to where a border should be, then edges close to this assumed border in the correct direction are part of it. edge linking etc. Regions can be constructed from detected borders that are complete (i.e. form closed boundaries). However, detected borders are almost always incomplete IPA 73
Region based segmentation These methods construct regions directly. The results of edge based and region based methods are usually not the same. Homogeneity is an important property of regions it can be based on greylevel, colour, texture, shape, etc. IPA 74
Definitions Based on sets. Each image R is a set of regions R i. R 7 R 6 Every pixel belongs to one region. One pixel can only belong to a single region. R 2 R 1 R 3 R 5 R 4 R S i 1 R i R i R j IPA 75
R 7 R 6 R 1 R 5 R 2 R 3 R 4 IPA 76
Basic Formulation Let R represent the entire image region. Segmentation partitions R into n subregions, R 1, R 2,..., R n, such that: a) b) c) d) e) n i 1 R i R i R i R is a connected region, i R j for all i and 1, j, i P( R i ) TRUE for i 1,2,..., n. P( R R ) FALSE for i j. i j j 2,..., n. a) Every pixel must be in a region b) Points in a region must be connected. c) Regions must be disjoint. d) All pixels in a region satisfy specific properties. e) Different regions have different properties. IPA 77
How do we form regions? Region Growing Region Merging Region Splitting Split and Merge Watershed... What a computer sees IPA 78
Region growing Groups pixels into larger regions. Starts with a seed region. Grows region by merging neighboring pixels. Iterative process How to start? How to iterate? When to stop? Finish Initial Regions Iterations Stop Condition IPA 79
IPA 80
Region merging Algorithm Divide image into an initial set of regions. One region per pixel. Define a similarity criteria for merging regions. Merge similar regions. Repeat previous step until no more merge operations are possible. IPA 81
Similarity Criteria Homogeneity of regions is used as the main segmentation criterion in region growing. gray level color, texture shape model etc. Choice of criteria affects segmentation results dramatically! IPA 82
Region splitting Algorithm One initial set that includes the whole image. Similarity criteria. Iteratively split regions into sub regions. Stop when no more splittings are possible. R 1 R 1 R 2 R 1 R 2 R 3 R 4 R 5 R 6 R 7 R 3 R 4 R 1 R 2 R 3 R 4 R 6 R 5 R 7 IPA 83
IPA 84 [Machine Vision; David Vernon]
Split and Merge Combination of both algorithms. Can handle a larger variety of shapes. Simply apply previous algorithms consecutively. IPA 85
Watershed transformation So far we have seen some: Region based methods: Regions are grown based on homogeneity criteria. Edge based methods: Discontinuities between regions are looked for. You can combine region and edge based methods watersheds: It groups pixels around the regional minima of the image. Boundaries of adjacent groupings are precisely located along the crest lines of the gradient image. Advantages: Usually produces more stable segmentation results. Always produces continuous region boundaries. Provides a simple way of including knowledge based constraints. IPA 86
Watershed transformation basis (1) We consider a greyscale image as a topographic representation. Reminder: IPA 87
Watershed transformation basis (2) If a drop of water falls on such a topographic surface, it will obey the law of gravitation and flow along the path of steepest slope until it reaches a minimum. The whole set of points of the surface whose paths of steepest slope reach a given minimum constitutes the catchment basin associated with this minimum. The watersheds are the zones dividing adjacent catchment basins. Watershed Lines Catchment Basins surface of the function Minima IPA 88
Watershed construction by flooding (1) Suppose that holes are made in each local minimum and that the surface is flooded from these holes. Progressively, the water level will increase. In order to prevent the merging of water coming from two different holes, a dam is progressively built at each contact point. At the end, the Water level union of all complete dams constitute the watersheds. Dams under construction IPA 89 Minima
Watershed construction by flooding (2) Example: from http://cmm.ensmp.fr/~beucher/wtshed.html IPA 90
Segmentation as clustering Cluster together (pixels, tokens, etc.) that belong together Agglomerative clustering/merging/growing merge cluster that are closer to each other repeat Divisive clustering/splitting split cluster along best boundary repeat Dendrograms yield a picture of output as clustering process continues IPA 91
Summary Segmentation to find object boundaries or mid level regions. Bottom up segmentation via merging Next segmentation as clustering K means Mean shift Graph cut, normalized cuts IPA 92
Many thanks to R. Szeliski, K. Grauman, Ch. M. Bishop, A. Torralba, Gonzales etal. and other mentioned This slides are for your personal usage only and can be used only for educational purpose! The copyrights of all the authors are aknowledged.