Image Processing and Image Analysis VU

Transcription

1 Image Processing and Image Analysis VU Yll Haxhimusa vda.univie.ac.at/teaching/ipa/17w/

2 Outline What are grouping problems in vision? Inspiration from human perception Gestalt properties Bottom up segmentation Algorithms: Merging and splitting Mode finding and mean shift: k means, mean shift Graph based: normalized cuts Features: color, texture, IPA 2

3 Grouping in vision The whole is equal to the sum of its parts The whole is greater the sum of its parts Goals: Euclid Max Wertheimer Gather features that belong together Obtain an intermediate representation that compactly describes key image or video parts IPA 3

4 A simple segmentation problem 4 IPA 4

5 It can get a lot harder Brady, M. J., & Kersten, D. (2003). Bootstrapped learning of novel objects. J Vis, 3(6), IPA 5

6 Examples of grouping in vision [Figure by J. Shi] Determine image regions [ peakdepvidindex_img2.jpg] Group video frames into shots Fg / Bg [Figure by Wang & Suter] Figure-ground [Figure by Grauman & Darrell] Object-level grouping IPA 6

7 Grouping in vision Goals: Gather features that belong together Obtain an intermediate representation that compactly describes key image (video) parts Top down vs. bottom up segmentation Top down: pixels belong together because they are from the same object Bottom up: pixels belong together because they look similar Hard to measure success What is interesting depends on the app. IPA 7

8 IPA 8

9 Muller Lyer illusion IPA 9

10 What things should be grouped? What cues indicate groups? IPA 10

11 Perceptual organization the processes by which the bits and pieces of visual information that are available in the retinal image are structured into the larger units of perceived objects and their interrelations Stephen E. Palmer, Vision Science, 1999 IPA 11

12 Gestalt Gestalt: whole or group Whole is greater than sum of its parts Relationships among parts can yield new properties/features Psychologists identified series of factors that predispose set of elements to be grouped (by human visual system) Gestalt principles There are hundreds of different grouping laws IPA 12

13 IPA 13

14 Familiar configuration IPA 14

15 Similarity IPA

16 Symmetry IPA 16

17 Common fate Image credit: Arthus-Bertrand (via F. Durand) IPA 17

18 Proximity IPA 18

19 Illusory/subjective contours Interesting tendency to explain by occlusion In Vision, D. Marr, 1982 IPA 19

20 IPA 20

21 Continuity, explanation by occlusion IPA 21

22 Grouping is a global process IPA 22

23 Grouping is a global process Occlusion is an important cue in grouping. IPA 23

24 but not too global IPA 24

25 Magritte, 1957 IPA 25

26 Figure ground IPA 26

27 Groupings by Invisible Completions IPA 27

28 IPA 28 In Vision, D. Marr, 1982; from J. L. Marroquin, Human visual perception of structure, 1976.

29 Familiarity IPA 29

30 Familiarity IPA 30

31 Influences of grouping Grouping influences other perceptual mechanisms such as lightness perception IPAhttp://web.mit.edu/persci/people/adelson/publications/gazzan.dir/koffka.html 31

32 Gestalt Gestalt: whole or group Whole is greater than sum of its parts Relationships among parts can yield new properties/features Psychologists identified series of factors that predispose set of elements to be grouped (by human visual system) Inspiring observations/explanations; challenge remains how to best map to algorithms. IPA 32

33 Image segmentation Has the goal of dividing an image into parts (regions) that have a strong correlation with objects or areas of the real world contained in the image (Sonka et al., 1999) For narrow domain applications, automatic image segmentation can produce very good results: IPA 33 Automatic segmentation of silver grains on a photographic plate (Beucher)

34 The goals of segmentation Separate image into coherent objects image human segmentation IPA 34 Source: Lana Lazebnik

35 The goals of segmentation Separate image into coherent objects Group together similar looking pixels for efficiency of further processing superpixels X. Ren and J. Malik. Learning a classification IPA model for segmentation. ICCV Source: Lana Lazebnik

36 Partitions X 1 X 2 X 3 The usual aim of segmentation is to produce a partition of the image. X X 6 4 X 5 A partition is defined as a division of a set into nonempty disjoint sets which completely cover the set. Mathematically, a collection of sets X 1,, X n defines a partition of a set X if an only if: 1.for all X i (X), X i (no subset is empty); 2.for all X i, X j (X), i j X i X j = (subsets are disjoint); 3. i {X i X i (X) } = X (the subsets cover X). IPA 36 Each subset X i is referred to as a region of X. X Euclid

37 Examples of partitions IPA 37

38 For wider domain applications, such as general image understanding, automatic segmentation is not capable of separating out all image objects. This is because: Segmentation uses only criteria based on homogeneity or heterogeneity of image properties (colour, texture, ) No knowledge of objects is present. IPA 38

39 There is no correct segmentation, as can be seen by these segmentations, produced by a number of humans. From the Berkeley segmentation benchmark IPA 39

40 When asked to segment an image, humans automatically make use of contextual information. In this case, it is: Shadows are not objects, even if they have a strong boundary. Information about the shape of a horse there is no visible boundary between the foot and the shadow. IPA 40 From the Berkeley segmentation benchmark

41 Depending on how the homogeneity criteria are set up, the results of segmentation can be: As required Over segmented Under segmented IPA 41

42 Image segmentation: toy example pixel count black pixels gray pixels white pixels input image intensity These intensities define the three groups. We could label every pixel in the image according to which of these primary intensities it is. i.e., segment the image based on the intensity feature. What if the image isn t quite so simple? Kristen Grauman IPA 42

43 input image pixel count intensity pixel count Kristen Grauman input image intensity IPA 43

44 pixel count input image intensity Now how to determine the three main intensities that define our groups? Kristen Grauman IPA 44

45 Segmentation by Thresholding The simplest segmentation method. Suitable if: objects do not touch each other; their greylevels are clearly distinct from the background greylevels. Many methods exist for choosing the threshold automatically based on histogram analysis. Smooth and then search for minima. Approximation of the histogram by normal distributions. etc. IPA 45

46 Example: Greylevel histogram based segmentation Noise free Low noise High noise From Spann IPA 46

47 Greylevel histogram based segmentation How do we characterise low noise and high noise? We can consider the histograms of our images For the noise free image, its simply two spikes at i=100, i=150 For the low noise image, there are two clear peaks centred on i=100, i=150 For the high noise image, there is a single peak two greylevel populations corresponding to object and background have merged IPA 47

48 Greylevel histogram based segmentation h(i) Noise free Low noise High noise i IPA 48

49 Greylevel histogram based segmentation We can define the input image signal to noise ratio in terms of the mean greylevel value of the object pixels and background pixels and the noise standard deviation S / N b o IPA 49

50 Greylevel histogram based segmentation For our test images : S/N (noise free) = S/N (low noise) = 5 S/N (high noise) = 2 IPA 50

51 Greylevel thresholding We can easily understand segmentation based on thresholding by looking at the histogram of the low noise object/background image There is a clear valley between to two peaks IPA 51

52 Greylevel thresholding h(i) Background Object i T IPA 52

53 Greylevel thresholding We can define the greylevel thresholding algorithm as follows: If the greylevel of pixel p <=T then pixel p is an object pixel else Pixel p is a background pixel IPA 53

54 Greylevel thresholding This simple threshold test begs the obvious question how do we determine the threshold? Many approaches possible Interactive threshold Adaptive threshold Minimisation method IPA 54

55 Greylevel thresholding We will consider in detail a minimisation method for determining the threshold Minimisation of the within group variance Robot Vision, Haralick & Shapiro, volume 1, page 20 IPA 55

56 Greylevel thresholding Idealized object/background image histogram h(i) T i IPA 56

57 Greylevel thresholding Any threshold separates the histogram into 2 groups with each group having its own statistics (mean, variance) The homogeneity of each group is measured by the within group variance The optimum threshold is that threshold which minimizes the within group variance thus maximizing the homogeneity of each group Otsu s Thresholding method (1979) IPA 57

58 Greylevel thresholding Let group o (object) be those pixels with greylevel <=T Let group b (background) be those pixels with greylevel >T The prior probability of group o is p o (T) The prior probability of group b is p b (T) IPA 58

59 Greylevel thresholding The following expressions can easily be derived for prior probabilities of object and background p ( T) P( i) o i p ( T) P( i) b where h(i) is the histogram of an N pixel image T i T 1 P(i ) h( i )/ N IPA 59

60 Greylevel thresholding The mean and variance of each group are as follows : o T ( T) ip(i)/ p ( T) i b i T 1 ( T) ip( i)/ p ( T) T 2 2 o o i 0 ( T) i ( T) P(i)/ p ( T) b b i T 1 ( T) i ( T) P( i)/ p ( T) o b o b IPA 60

61 Greylevel thresholding The within group variance is defined as : 2 ( T) 2 ( T) p ( T) 2 ( T) p ( T) W o o b b We determine the optimum T by minimizing this expression with respect to T Only requires 256 comparisons for and 8 bit greylevel image IPA 61

62 Greylevel thresholding h(i) T opt Histogram Within group variance i IPA 62

63 Greylevel thresholding We can examine the performance of this algorithm on our low and high noise image For the low noise case, it gives an optimum threshold of T=124 Almost exactly halfway between the object and background peaks We can apply this optimum threshold to both the low and high noise images IPA 63

64 Greylevel thresholding Low noise image Thresholded at T=124 IPA 64

65 Greylevel thresholding Low noise image Thresholded at T=124 IPA 65

66 Greylevel thresholding High level of pixel miss classification noticeable This is typical performance for thresholding The extent of pixel miss classification is determined by the overlap between object and background histograms. IPA 66

67 Greylevel thresholding p(x) 0.02 Background 0.01 Object 0.00 x o b T IPA 67

68 Greylevel thresholding p(x) 0.02 Background 0.01 Object 0.00 x o b T IPA 68

69 Greylevel thresholding Easy to see that, in both cases, for any value of the threshold, object pixels will be miss classified as background and vice versa For greater histogram overlap, the pixel miss classification is obviously greater We could even quantify the probability of error in terms of the mean and standard deviations of the object and background histograms IPA 69

70 Greylevel clustering Consider an idealized object/background histogram Background Object IPA 70 c 1 c 2

71 Edge based segmentation We look for borders separating regions in the image. Rely on edges found by edge detecting operators these edges mark discontinuities in greylevel, colour, texture, etc. These edges cannot be directly used as a segmentation Post processing steps combine edges into edge chains which correspond better with borders in the image. Common problems: caused by noise or unsuitable information in an image. Edge presence in locations where there is no border. no edge presence where a real border exists. IPA 71

72 Edge image thresholding Small edge values in gradient images correspond to non significant greylevel changes. Thresholding of an edge image can remove these small values. from Sonka et al. Original image Edge image (weak edges enhanced) IPA Edge image thresholded at 30 Edge image thresholded 72 at 10

73 Comments on edge based segmentation Some edge refinement methods are available, such as edge relaxation: The edge information within a neighbourhood is taken into account to increase or decrease the probability that edges are borders. use of border local information: if one has an assumption as to where a border should be, then edges close to this assumed border in the correct direction are part of it. edge linking etc. Regions can be constructed from detected borders that are complete (i.e. form closed boundaries). However, detected borders are almost always incomplete IPA 73

74 Region based segmentation These methods construct regions directly. The results of edge based and region based methods are usually not the same. Homogeneity is an important property of regions it can be based on greylevel, colour, texture, shape, etc. IPA 74

75 Definitions Based on sets. Each image R is a set of regions R i. R 7 R 6 Every pixel belongs to one region. One pixel can only belong to a single region. R 2 R 1 R 3 R 5 R 4 R S i 1 R i R i R j IPA 75

76 R 7 R 6 R 1 R 5 R 2 R 3 R 4 IPA 76

77 Basic Formulation Let R represent the entire image region. Segmentation partitions R into n subregions, R 1, R 2,..., R n, such that: a) b) c) d) e) n i 1 R i R i R i R is a connected region, i R j for all i and 1, j, i P( R i ) TRUE for i 1,2,..., n. P( R R ) FALSE for i j. i j j 2,..., n. a) Every pixel must be in a region b) Points in a region must be connected. c) Regions must be disjoint. d) All pixels in a region satisfy specific properties. e) Different regions have different properties. IPA 77

78 How do we form regions? Region Growing Region Merging Region Splitting Split and Merge Watershed... What a computer sees IPA 78

79 Region growing Groups pixels into larger regions. Starts with a seed region. Grows region by merging neighboring pixels. Iterative process How to start? How to iterate? When to stop? Finish Initial Regions Iterations Stop Condition IPA 79

80 IPA 80

81 Region merging Algorithm Divide image into an initial set of regions. One region per pixel. Define a similarity criteria for merging regions. Merge similar regions. Repeat previous step until no more merge operations are possible. IPA 81

82 Similarity Criteria Homogeneity of regions is used as the main segmentation criterion in region growing. gray level color, texture shape model etc. Choice of criteria affects segmentation results dramatically! IPA 82

83 Region splitting Algorithm One initial set that includes the whole image. Similarity criteria. Iteratively split regions into sub regions. Stop when no more splittings are possible. R 1 R 1 R 2 R 1 R 2 R 3 R 4 R 5 R 6 R 7 R 3 R 4 R 1 R 2 R 3 R 4 R 6 R 5 R 7 IPA 83

84 IPA 84 [Machine Vision; David Vernon]

85 Split and Merge Combination of both algorithms. Can handle a larger variety of shapes. Simply apply previous algorithms consecutively. IPA 85

86 Watershed transformation So far we have seen some: Region based methods: Regions are grown based on homogeneity criteria. Edge based methods: Discontinuities between regions are looked for. You can combine region and edge based methods watersheds: It groups pixels around the regional minima of the image. Boundaries of adjacent groupings are precisely located along the crest lines of the gradient image. Advantages: Usually produces more stable segmentation results. Always produces continuous region boundaries. Provides a simple way of including knowledge based constraints. IPA 86

87 Watershed transformation basis (1) We consider a greyscale image as a topographic representation. Reminder: IPA 87

88 Watershed transformation basis (2) If a drop of water falls on such a topographic surface, it will obey the law of gravitation and flow along the path of steepest slope until it reaches a minimum. The whole set of points of the surface whose paths of steepest slope reach a given minimum constitutes the catchment basin associated with this minimum. The watersheds are the zones dividing adjacent catchment basins. Watershed Lines Catchment Basins surface of the function Minima IPA 88

89 Watershed construction by flooding (1) Suppose that holes are made in each local minimum and that the surface is flooded from these holes. Progressively, the water level will increase. In order to prevent the merging of water coming from two different holes, a dam is progressively built at each contact point. At the end, the Water level union of all complete dams constitute the watersheds. Dams under construction IPA 89 Minima

90 Watershed construction by flooding (2) Example: from IPA 90

91 Segmentation as clustering Cluster together (pixels, tokens, etc.) that belong together Agglomerative clustering/merging/growing merge cluster that are closer to each other repeat Divisive clustering/splitting split cluster along best boundary repeat Dendrograms yield a picture of output as clustering process continues IPA 91

92 Summary Segmentation to find object boundaries or mid level regions. Bottom up segmentation via merging Next segmentation as clustering K means Mean shift Graph cut, normalized cuts IPA 92

93 Many thanks to R. Szeliski, K. Grauman, Ch. M. Bishop, A. Torralba, Gonzales etal. and other mentioned This slides are for your personal usage only and can be used only for educational purpose! The copyrights of all the authors are aknowledged.