Knowledge-driven morphological approaches for image segmentation and object detection

Transcription

1 Knowledge-driven morphological approaches for image segmentation and object detection Image Sciences, Computer Sciences and Remote Sensing Laboratory (LSIIT) Models, Image and Vision Team (MIV) Discrete Geometry and Mathematical Morphology Group (GDMM) University Louis Pasteur - CNRS, Strasbourg / France lefevre@lsiit.u-strasbg.fr LMB Freiburg April 21st and 28th, 2008

2 Mathematical morphology, a widely used toolbox in image processing However, defining an adequate MM-based solution either : requires to be an expert in the field of MM, or to limit oneself to some basic morphological tools and domain knowledge is hardly taken into account in MM. Knowledge-driven morphological approaches We will investigate some approaches for : image segmentation with the watershed transform, object detection with the hit-or-miss transform. Knowledge-driven morphological approaches for image segmentation and object detection 2

3 Mathematical morphology, a widely used toolbox in image processing However, defining an adequate MM-based solution either : requires to be an expert in the field of MM, or to limit oneself to some basic morphological tools and domain knowledge is hardly taken into account in MM. Knowledge-driven morphological approaches We will investigate some approaches for : image segmentation with the watershed transform, object detection with the hit-or-miss transform. Knowledge-driven morphological approaches for image segmentation and object detection 2

4 Outline 1 Knowledge & MM 2 Watershed 3 Hit-or-Miss 4 Remote Sensing 5 Other results 6 Conclusion and perspectives within LMB Knowledge-driven morphological approaches for image segmentation and object detection 3

5 Outline 1 Knowledge & MM 2 Watershed 3 Hit-or-Miss 4 Remote Sensing 5 Other results 6 Conclusion and perspectives within LMB Knowledge-driven morphological approaches for image segmentation and object detection 3

6 Outline 1 Knowledge & MM 2 Watershed 3 Hit-or-Miss 4 Remote Sensing 5 Other results 6 Conclusion and perspectives within LMB Knowledge-driven morphological approaches for image segmentation and object detection 3

7 Outline 1 Knowledge & MM 2 Watershed 3 Hit-or-Miss 4 Remote Sensing 5 Other results 6 Conclusion and perspectives within LMB Knowledge-driven morphological approaches for image segmentation and object detection 3

8 Outline 1 Knowledge & MM 2 Watershed 3 Hit-or-Miss 4 Remote Sensing 5 Other results 6 Conclusion and perspectives within LMB Knowledge-driven morphological approaches for image segmentation and object detection 3

9 Outline 1 Knowledge & MM 2 Watershed 3 Hit-or-Miss 4 Remote Sensing 5 Other results 6 Conclusion and perspectives within LMB Knowledge-driven morphological approaches for image segmentation and object detection 3

10 Basics of Mathematical morphology Definition A framework for analysis of spatial structures in images. A non-linear alternative to existing image processing toolboxes. A set of operators applied on an image f with a pattern (SE) B. Two fundamentals operators (erosion ε and dilation δ) from which are built more complex ones. Theoretical framework For binary images, MM may be formalized with set theory... but not for more complex images (grayscale, multispectral). Thus MM is rather defined using complete lattice theory. Knowledge-driven morphological approaches for image segmentation and object detection 4

11 Basics of Mathematical morphology Definition A framework for analysis of spatial structures in images. A non-linear alternative to existing image processing toolboxes. A set of operators applied on an image f with a pattern (SE) B. Two fundamentals operators (erosion ε and dilation δ) from which are built more complex ones. Theoretical framework For binary images, MM may be formalized with set theory... but not for more complex images (grayscale, multispectral). Thus MM is rather defined using complete lattice theory. Knowledge-driven morphological approaches for image segmentation and object detection 4

12 Basics of Mathematical morphology Complete lattice theory A complete lattice is defined from : a partially ordered set (L, ) (e.g. the natural order of scalars) an infimum or greatest lower bound (e.g. minimum) a supremum or least upper bound (e.g. maximum) Main operators Erosion : ε B (f )(x, y) = (r,s) B Dilation : δ B (f )(x, y) = Opening : Closing : (r,s) B γ B (f )(p) = δ B(ε B (f )(p)) φ B B(f )(p) = ε B (δ B (f )(p)) f (x + r, y + s), (x, y) E f (x + r, y + s), (x, y) E Knowledge-driven morphological approaches for image segmentation and object detection 5

13 Basics of Mathematical morphology Complete lattice theory A complete lattice is defined from : a partially ordered set (L, ) (e.g. the natural order of scalars) an infimum or greatest lower bound (e.g. minimum) a supremum or least upper bound (e.g. maximum) Main operators Erosion : ε B (f )(x, y) = (r,s) B Dilation : δ B (f )(x, y) = Opening : Closing : (r,s) B γ B (f )(p) = δ B(ε B (f )(p)) φ B B(f )(p) = ε B (δ B (f )(p)) f (x + r, y + s), (x, y) E f (x + r, y + s), (x, y) E Knowledge-driven morphological approaches for image segmentation and object detection 5

14 Domain knowledge and MM Existing methods for knowledge integration Definition of structuring elements It is possible to define the shape and size of the SE depending on the problem under consideration. But definition of the SE may be irrelevant for some objects! Choice of morphological operators To solve a problem, a morphological expert will select the appropriate operators or algorithms (including pre/post-processing). But the expert is not always here! Parameters tuning Many algorithms require some parameters or thresholds to be set (e.g. oversegmentation reduction). But this task is most often very awkward! Knowledge-driven morphological approaches for image segmentation and object detection 6

15 Domain knowledge and MM Various knowledge may be available for the object(s) or image(s) shape(s) size(s) scale(s) spectral composition(s) textural description(s) number and models of object classes noise level of the signal... Knowledge-driven morphological approaches for image segmentation and object detection 7

16 Two kinds of knowledge Expert knowledge A database, a set of rules or parameters, or an ontology contains knowledge given and formalized by the expert. Learning samples A set of visual examples which are semantically meaningful for the user, thus he can give and label some learning data for a subsequent classification approach. Knowledge-driven morphological approaches for image segmentation and object detection 8

17 Two kinds of knowledge Expert knowledge A database, a set of rules or parameters, or an ontology contains knowledge given and formalized by the expert. Learning samples A set of visual examples which are semantically meaningful for the user, thus he can give and label some learning data for a subsequent classification approach. Knowledge-driven morphological approaches for image segmentation and object detection 8

18 Two kinds of knowledge Knowledge-driven morphological approaches for image segmentation and object detection 9

19 Using expert knowledge From objects to structuring elements An object is characterized by : spatial properties spectral properties combination of both thus it can lead to a predefined SE. Definition of a SE Various kind of SE can be used : flat SE (only shape/size information) functional SE (+ intensity profile) vectorial SE (+ spectral profile) composite SE which consists in a set of single SEs Knowledge-driven morphological approaches for image segmentation and object detection 10

20 Using expert knowledge From objects to structuring elements An object is characterized by : spatial properties spectral properties combination of both thus it can lead to a predefined SE. Definition of a SE Various kind of SE can be used : flat SE (only shape/size information) functional SE (+ intensity profile) vectorial SE (+ spectral profile) composite SE which consists in a set of single SEs Knowledge-driven morphological approaches for image segmentation and object detection 10

21 Using expert knowledge Incomplete definition of the object or SE The expert knowledge may be less precise : only the number of classes of interest is known only an approximate size and shape are known In both cases, several single results are computed : for each cluster or class for each object size for each object shape... There is finally a need to merge the results (e.g. with union or supremum). Knowledge-driven morphological approaches for image segmentation and object detection 11

22 Using learning samples Learning dataset and supervised classification process Spectral/textural information is given by the learning dataset. This dataset can be used by a hard or fuzzy supervised classification process. A knowledge-based representation space A pixel is associated with membership values to each class. Vectors contain class membership values instead of spectral values. But both can also be combined (e.g. average, product). Knowledge-driven morphological approaches for image segmentation and object detection 12

23 Using learning samples Learning dataset and supervised classification process Spectral/textural information is given by the learning dataset. This dataset can be used by a hard or fuzzy supervised classification process. A knowledge-based representation space A pixel is associated with membership values to each class. Vectors contain class membership values instead of spectral values. But both can also be combined (e.g. average, product). Knowledge-driven morphological approaches for image segmentation and object detection 12

24 Using learning samples And spatial information? Remark : the spatial location of learning samples may also be relevant and could be used. Towards an iterative process If posterior evaluation is possible, optimal parameters for the morphological process can be estimated : shape size spectral band other parameters (e.g. for oversegmentation reduction) Knowledge-driven morphological approaches for image segmentation and object detection 13

25 Using learning samples And spatial information? Remark : the spatial location of learning samples may also be relevant and could be used. Towards an iterative process If posterior evaluation is possible, optimal parameters for the morphological process can be estimated : shape size spectral band other parameters (e.g. for oversegmentation reduction) Knowledge-driven morphological approaches for image segmentation and object detection 13

26 Watershed Transform Principle of the immersion paradigm 1 the input image is considered as a topographic surface 2 this surface is flooded from its local minima (=basins) 3 dams are built to prevent waters coming from two different sources to merge 4 when all the surface is flooded, dams are the watershed lines Comments the algorithm is often applied on an image gradient this method is prone to oversegmentation, solutions are : using some markers (initial basins), thus limiting the number of created regions involving some oversegmentation reduction techniques how to guide this algorithm with some knowledge? Knowledge-driven morphological approaches for image segmentation and object detection 14

27 Watershed Transform Principle of the immersion paradigm 1 the input image is considered as a topographic surface 2 this surface is flooded from its local minima (=basins) 3 dams are built to prevent waters coming from two different sources to merge 4 when all the surface is flooded, dams are the watershed lines Comments the algorithm is often applied on an image gradient this method is prone to oversegmentation, solutions are : using some markers (initial basins), thus limiting the number of created regions involving some oversegmentation reduction techniques how to guide this algorithm with some knowledge? Knowledge-driven morphological approaches for image segmentation and object detection 14

28 Watershed transform Knowledge-driven morphological approaches for image segmentation and object detection 15

29 Learning-based segmentation How to make use of the memberships? Several solutions : before segmentation : input image transformation during segmentation : watershed algorithm modification after segmentation : segmentation parameters optimization Notations f the input image to be segmented, p a pixel w i (p) is the membership of p to class C i M i (p) is the learning set for class C i Knowledge-driven morphological approaches for image segmentation and object detection 16

30 Learning-based segmentation How to make use of the memberships? Several solutions : before segmentation : input image transformation during segmentation : watershed algorithm modification after segmentation : segmentation parameters optimization Notations f the input image to be segmented, p a pixel w i (p) is the membership of p to class C i M i (p) is the learning set for class C i Knowledge-driven morphological approaches for image segmentation and object detection 16

31 1. Input image transformation Synopsis 1 Space change from f to w 2 Topographic surface creation by gradient computation on w 3 Application of the watershed transform Knowledge-driven morphological approaches for image segmentation and object detection 17

32 2. Watershed algorithm modification Motivations : definition of watershed lines (i.e. borders) f (p) is low p is not a line f (p) is high but i : w i (p) is high p should not be a line otherwise p is and should be a watershed line = we define f i = (1 w i ) f Synopsis 1 manually define markers as learning sets M 2 build a topographic surface for each class, f i = (1 w i ) f 3 flood from each marker M i considering the f i surface 4 p is set to M i if it is firstly reached by the M i flooding Knowledge-driven morphological approaches for image segmentation and object detection 18

33 2. Watershed algorithm modification Motivations : definition of watershed lines (i.e. borders) f (p) is low p is not a line f (p) is high but i : w i (p) is high p should not be a line otherwise p is and should be a watershed line = we define f i = (1 w i ) f Synopsis 1 manually define markers as learning sets M 2 build a topographic surface for each class, f i = (1 w i ) f 3 flood from each marker M i considering the f i surface 4 p is set to M i if it is firstly reached by the M i flooding Knowledge-driven morphological approaches for image segmentation and object detection 18

34 2. Watershed algorithm modification Illustrative example Knowledge-driven morphological approaches for image segmentation and object detection 19

35 2. Watershed algorithm modification Illustrative example Knowledge-driven morphological approaches for image segmentation and object detection 19

36 2. Watershed algorithm modification Illustrative example Knowledge-driven morphological approaches for image segmentation and object detection 19

37 2. Watershed algorithm modification Illustrative example Knowledge-driven morphological approaches for image segmentation and object detection 19

38 3. Segmentation parameters optimization Oversegmentation reduction Threshold local minima lower than hmin Do not flood basins with low depth < d Merge connected regions with similar average values Knowledge-driven morphological approaches for image segmentation and object detection 20

39 3. Segmentation parameters optimization Select optimal parameters with a genetic algorithm individual (phenotype) = set of normalized parameters fitness function = accuracy of region-based supervised classification learning zones used in cross-validation for each iteration, n segmentations are performed! Knowledge-driven morphological approaches for image segmentation and object detection 21

40 Binary Hit-or-Miss Definition Consider simultaneously two SE A and B. A must fit the foreground f and B the background f c. η A,B (f )(p) = ε A (f )(p) ε B (f c )(p) Knowledge-driven morphological approaches for image segmentation and object detection 22

41 Grayscale Hit-or-Miss Several definitions! g,h (f )(p) = max{ε g (f )(p) δ h (f )(p), 0} { ηg,h Ronse ε g (f )(p) if ε g (f )(p) δ h (f )(p) (f )(p) = 0 otherwise η Soille Knowledge-driven morphological approaches for image segmentation and object detection 23

42 Shortcomings of existing HMT Shortcomings The previous HMT cannot deal with : objects neither dark or bright objects which do not fit within exactly the couple of SE multispectral objects Solutions Clustering-based HMT Fuzzy HMT Multivariate HMT Knowledge-driven morphological approaches for image segmentation and object detection 24

43 Shortcomings of existing HMT Shortcomings The previous HMT cannot deal with : objects neither dark or bright objects which do not fit within exactly the couple of SE multispectral objects Solutions Clustering-based HMT Fuzzy HMT Multivariate HMT Knowledge-driven morphological approaches for image segmentation and object detection 24

44 Clustering-based HMT From a single uncertain SE pair to multiple SE η α,k,l A,B (f )(p) = ε Aαk,αl (f )(p) ε Bk,l (f c )(p) k K,l L α is the uncertainty coefficient, and A k,l is a SE of size k l. From a grayscale image to binary images Since the object may be neither brighter nor darker than all the other objects, split the graylevels through a clustering. Knowledge-driven morphological approaches for image segmentation and object detection 25

45 Clustering-based HMT More binary images The object may have a regular shape, not a regular content. The object may be rather composed of homogeneous parts. Thus it can spread over several clusters! All previous binary images are combined to build new ones. Synopsis 1 Build n binary images from a n-class clustering of the input 2 Merge each pair (or n-tuple) of images to create new ones 3 On each image, apply HMT with several uncertain shapes/sizes 4 Fusion the results (e.g. by taking the supremum) Knowledge-driven morphological approaches for image segmentation and object detection 26

46 Clustering-based HMT More binary images The object may have a regular shape, not a regular content. The object may be rather composed of homogeneous parts. Thus it can spread over several clusters! All previous binary images are combined to build new ones. Synopsis 1 Build n binary images from a n-class clustering of the input 2 Merge each pair (or n-tuple) of images to create new ones 3 On each image, apply HMT with several uncertain shapes/sizes 4 Fusion the results (e.g. by taking the supremum) Knowledge-driven morphological approaches for image segmentation and object detection 26

47 Fuzzy HMT Principles g and h are considered as soft lower and upper bounds. we count the pixel values comprised between these bounds : ζ g,h (f )(x, y) = card (g(q) f (q) h(q), q = x + r, y + s) (r,s) S we compute the best possible fit : η g,h (f )(p) = max (ζ g+t,h+t(f )(p)/ card(s)) t T Knowledge-driven morphological approaches for image segmentation and object detection 27

48 Multivariate HMT Extended SE Shape Spectral band Class (foreground, background), i.e. operator to be used Threshold (if flat SE) or profile (if functional SE) Knowledge-based use of multivariate HMT 1 define each SE as a spatial-spectral template 2 keep a pixel if it is fitted by all SE 3 valuate the pixel by averaging the single valuations Knowledge-driven morphological approaches for image segmentation and object detection 28

49 Multivariate HMT Extended SE Shape Spectral band Class (foreground, background), i.e. operator to be used Threshold (if flat SE) or profile (if functional SE) Knowledge-based use of multivariate HMT 1 define each SE as a spatial-spectral template 2 keep a pixel if it is fitted by all SE 3 valuate the pixel by averaging the single valuations Knowledge-driven morphological approaches for image segmentation and object detection 28

50 Multivariate HMT HMT fitting Fitting s (f )(p) = { ε sshape (f sb )(p) s t δ sshape (f sb )(p) s t if s type = ε if s type = δ HMT Valuation Valuation s (f )(p) = ε sshape (f sb )(p) s t f s + b s t s t γ sshape (f sb )(p) s t f s b if s type = ε if s type = δ where [f k, f + k ] is the range of pixel values for the band f k. Knowledge-driven morphological approaches for image segmentation and object detection 29

51 Multivariate HMT HMT fitting Fitting s (f )(p) = { ε sshape (f sb )(p) s t δ sshape (f sb )(p) s t if s type = ε if s type = δ HMT Valuation Valuation s (f )(p) = ε sshape (f sb )(p) s t f s + b s t s t γ sshape (f sb )(p) s t f s b if s type = ε if s type = δ where [f k, f + k ] is the range of pixel values for the band f k. Knowledge-driven morphological approaches for image segmentation and object detection 29

52 Applications in remote sensing Project FODOMUST Multi-strategy data mining for extraction and analysis of urban vegetation from image databases. Project ECOSGIL Spatial knowledge extraction for integrated management of coastal areas. Knowledge-driven morphological approaches for image segmentation and object detection 30

53 Urban object detection : flowchart Knowledge-driven morphological approaches for image segmentation and object detection 31

54 Urban object detection : uncertainty in the HMT How to deal with defaults in the shape? input image binary object Hit-Or-Miss Knowledge-driven morphological approaches for image segmentation and object detection 32

55 Urban object detection : results Knowledge-driven morphological approaches for image segmentation and object detection 33

56 Coastal object detection : definition of the SEs {sshape 1 = , s1 band = NDVI, s1 thr = 0.5, s1 type = G} {sshape 2 = , s2 band = NDVI, s2 thr = 0.5, s2 type = L} {sshape 3 = , s3 band = R, s3 thr = 0.05, s3 type = L} Knowledge-driven morphological approaches for image segmentation and object detection 34

57 Knowledge & MM Watershed Hit-or-Miss Remote Sensing Other results Conclusion and perspectives within LMB Coastal object detection : visual result Synopsis 1 Apply the MHMT in all orientations (and take the max) 2 Connect the detected points through a double threshold procedure Se bastien Lefe vre Knowledge-driven morphological approaches for image segmentation and object detection 35

58 Coastal object detection : comparison with other results Knowledge-driven morphological approaches for image segmentation and object detection 36

59 Markerless segmentation of urban areas Results classical WT Genetic optimization Space change space change and optimization Knowledge-driven morphological approaches for image segmentation and object detection 37

60 Marker-based segmentation of urban areas marker-based watershed using spectral markers with one marker for each building and a single marker (bottom right) for the background Knowledge-driven morphological approaches for image segmentation and object detection 38

61 Marker-based segmentation of coastal areas Context, coastline extraction In such a case, a segmentation into two objects may be useful. input image result with markers Knowledge-driven morphological approaches for image segmentation and object detection 39

62 Marker-based segmentation of coastal areas marker-based watershed using spectral markers Knowledge-driven morphological approaches for image segmentation and object detection 40

63 Marker-based segmentation and Berkeley dataset Marker-based vs. markerless watershed segmentation Input image Markerless segmentation Aptoula & Lefèvre ACCV 2007 Input markers Marker-based segmentation Lefèvre CAIP 2007 Knowledge-driven morphological approaches for image segmentation and object detection 41

64 Marker-based segmentation and Berkeley dataset Supervised vs. classical marker-based watershed segmentation markers classical WT supervised WT Knowledge-driven morphological approaches for image segmentation and object detection 42

65 Marker-based segmentation and Berkeley dataset 2 markers markers and classification 3 markers markers and classification standard and supervised WT standard and supervised WT Knowledge-driven morphological approaches for image segmentation and object detection 43

66 Astronomical imaging Detection of LSB galaxies Low Surface Brightness galaxies are very faint objects A challenging problem No multispectral information Analysis was performed on single grayscale images Noisy images Very low SNR Poisson + impulse noise Difference between LSB and background = 0.7σ Knowledge-driven morphological approaches for image segmentation and object detection 44

67 Astronomical imaging LSB vs. HSB galaxies HSB galaxy LSB galaxy Knowledge-driven morphological approaches for image segmentation and object detection 45

68 Astronomical imaging A very low signal-to-noise ratio Noise-related knowledge Noise level σ defines the gap between SE, i.e. h = g + 2σ. Knowledge-driven morphological approaches for image segmentation and object detection 46

69 Astronomical imaging Flowchart Examples of structuring elements various orientations and elongations Some results input, FHMT and binary map Knowledge-driven morphological approaches for image segmentation and object detection 47

70 Conclusions Issues Knowledge integration for MM-based image analysis Focus on : Image segmentation with the watershed transform Object detection with the hit-or-miss transform Contributions Supervised (learning-based) markerless and marker-based watershed methods Supervised (expert knowledge-based) extended hit-or-miss transforms Illustrative examples, mainly in remote sensing Knowledge-driven morphological approaches for image segmentation and object detection 48

71 Conclusions Issues Knowledge integration for MM-based image analysis Focus on : Image segmentation with the watershed transform Object detection with the hit-or-miss transform Contributions Supervised (learning-based) markerless and marker-based watershed methods Supervised (expert knowledge-based) extended hit-or-miss transforms Illustrative examples, mainly in remote sensing Knowledge-driven morphological approaches for image segmentation and object detection 48

72 Some perspectives Image segmentation Make the parameter optimization process more efficient Avoid the Euclidean norm of the membership gradients Define automatically or semi-automatically the markers Extend to segmentation of 3D images and 2D+t sequences Object detection Consider fuzzy or soft HMT to prevent missing detections Build the SE from learning samples Perform some optimizations to limit CPU time Make SE invariants to various transforms and check theoretical properties of the proposed solutions... Knowledge-driven morphological approaches for image segmentation and object detection 49

73 Some perspectives Image segmentation Make the parameter optimization process more efficient Avoid the Euclidean norm of the membership gradients Define automatically or semi-automatically the markers Extend to segmentation of 3D images and 2D+t sequences Object detection Consider fuzzy or soft HMT to prevent missing detections Build the SE from learning samples Perform some optimizations to limit CPU time Make SE invariants to various transforms and check theoretical properties of the proposed solutions... Knowledge-driven morphological approaches for image segmentation and object detection 49

74 Some perspectives Image segmentation Make the parameter optimization process more efficient Avoid the Euclidean norm of the membership gradients Define automatically or semi-automatically the markers Extend to segmentation of 3D images and 2D+t sequences Object detection Consider fuzzy or soft HMT to prevent missing detections Build the SE from learning samples Perform some optimizations to limit CPU time Make SE invariants to various transforms and check theoretical properties of the proposed solutions... Knowledge-driven morphological approaches for image segmentation and object detection 49

75 Some further readings... Knowledge-driven watershed transform for image segmentation Derivaux et al (2006) Watershed Segmentation of Remotely Sensed Images Based on a Supervised Fuzzy Pixel Classification, IEEE IGARSS Derivaux et al (2007) On machine learning in watershed segmentation, IEEE International Workshop on Machine Learning in Signal Processing Lefèvre (2007), Knowledge from markers in watershed segmentation, IAPR CAIP Knowledge-driven hit-or-miss transform for object detection Lefèvre et al (2007), Automatic building extraction in VHR images using advanced morphological operators, IEEE/ISPRS URBAN Weber and Lefèvre (2008), A multivariate Hit-or-Miss Transform for conjoint spatial and spectral template matching, IEEE International Conference on Image and Signal Processing Perret et al (2008), A robust Hit-or-Miss Transform for template matching in very noisy astronomical images, European Signal Processing Conference Knowledge-driven morphological approaches for image segmentation and object detection 50