Edge Detection and Active Contours

Transcription

1 Edge Detection and Active Contours Elsa Angelini, Florence Tupin Department TSI, Telecom ParisTech 2011

2 Outline Introduction Edge Detection Active Contours

3 Introduction The segmentation Problem Edges Regions Textures Segmentation Measures Shape Recognition Structural Scene Analysis

4 What is an edge? Introduction

5 What is an edge? Introduction

6 What is an edge? Introduction

7 Introduction Contours profiles Staircase Ramp Roof

8 Introduction The segmentation problem: Partition an image into objects: 2 approaches: Region-based Contour-based Edge Detection: Detection of «abrupt changes» in image gradient. Analysis of first and second derivatives of image gradients.

9 Outline Introduction Edge Detection Active Contours

10 Ideal 1D edge Edge detection

11 Ideal 1D edge Edge detection

12 Ideal 1D edge Edge detection

13 Case of a 2D image Edge detection

14 Edge Detection Gradient-based Image G I( x, y) I( x, y) with a continuous representation: I x, y x I x, y y G I( x, y) 2 2 I x, y I x, y x y

15 Gradient-based

16 Edge detection Detection of gradient modulus maxima, in gradient direction

17 Edge detection Detection of gradient modulus maxima, in gradient direction

18 Edge detection Gradient-based Filters Gradient Roberts Prewitt Sobel Oriented edges.

19 Edge detection Gradient-based Dedicated Gradient Filters 1. Pre-processing: filtering (Gaussian, Median). 2. Segmentation via thresholding or local maxima detection. 3. Post-processing: contour closing, curve fitting, smoothing.

20 Edge Detection Gradient-based example: Boundary Tracking Edge map Boundary tracking is very sensitive to noise Use of smoothing, average gradient computation, large «tracking» neighborhoods.

21 Edge Detection Gradient-based example: morphological post-processing

22 Edge Detection Laplacian-based Edge 1st derivative 2nd derivative Zero Crossing I( x, y) I x, y I x, y 2 2 x y 2 2

23 Laplacian-based Edge Detection

24 Laplacian-based Edge Detection

25 Edge Detection Laplacian-based Laplacian operator on the image: Discrete implementation with convolution kernels: Set of closed connected contours but Very sensitive to noise! 2 convolution kernels

26 Edge Detection Laplacian-based: Laplacian of Gaussian (LoG) LoG I( x, y) I x, y G x, y I x, y G x, y 2 2 x G x, y G x, y I x, y I x, y 2 2 x y y Convolution kernel?

27 Edge Detection Laplacian-based: Laplacian of Gaussian (LoG) x y 2 2 x y x y G e e 2 2 Convolution kernel Band-pass Impulse response Transfer function

28 Edge Detection Laplacian-based Parameter controls the width of central peak : amount of smoothing. - Good approximation with Difference of Gaussians (DoG), with a ratio 2 / 1= 1.6. DoG separable in x and y : efficient implementation.

29 Edge Detection Laplacian of a Gaussian (LoG)

30 Edge Detection Laplacian of a Gaussian (LoG)

31 Edge Detection Analytical: CANNY: Hypothesis: 1D contours, staircase model, white Gaussian noise Edge detection via detection of local maxima of Linear Filtering. I x AH x n x O x0 I x fedge x0 x dx f x f x edge edge f edge?

32 Edge detection

33 Edge Detection Analytical CANNY: Performance Criteria: Good Detection 0 f edge x dx f 2 edge x dx Good Localization f ' edge f '2 edge 0 x dx Unique Response f 0 fedge x dx k f x dx f x dx ' edge ''2 2 edge edge 0

34 Edge Detection Analytical Canny: Optimization of the 3 criteria with Lagrange multipliers. b Max L x f f f x dx a b a ' '', edge, edge, edge f f f f dx 2 '2 '' edge edge edge edge 2 f 2 f 2 f 0 '' ''' edge edge edge

35 Analytical CANNY Solution x x x x f x a e sin wx a e cos wx a e sin wx a e cos wx edge ; w ; FIR on M, M : f f M f S f M w ' ' edge 0 0; edge 0; edge 0 ; edge

36 Edge Detection Analytical 1. CANNY s approximation with derivatives of Gaussian (IIR). fedge x xe 2 x 2 2, CANNY s approximation with IIR (DERICHE) f f f S f ' ' edge 0 0; edge 0; edge 0 ; edge 0 f x ce sin wx edge 2 ; x 2 w 2 2

37 Edge Detection Analytical DERICHE Deriche gaussian

38 Edge Detection In 2D : Derivation in one direction : Smoothing in the opposite direction : 2D expressions :

39 In 2D : Edge Detection

40

41 Edge detection Post-processing : Hysteresis : A high threshold to select very reliable edges A low threshold to select continuous edges Combination of both by connex component validation

42 Edge detection Post-processing : Edge closing by dynamic programming Hough transform

43 Outline Introduction Edge Detection Active Contours

44 Active Contours Formulations: Parametric Geometric Statistics Graph-cuts N-D Implementation Applications Conclusions and Perspectives

45 Active Contours Contour placed in the data space and deforming towards an optimal position and shape. Forces of Deformation: Internal forces: define intrinsic shape properties preserve shape smoothness during deformation. External forces: defined from the data. Control contour deformation according to data content (e.g: edge locations).

46 Active Contours Parametric contours. Geometric contours.

47 Parametric Active Contours Formulation of the Problem 1. Energy Minimization : Minimize the weighted sum of internal and external energies (Force potentials). Final contour position corresponds to an energy s minimum. 2. Dynamical Forces : Equilibrium between internal and external forces at each point on the contour.

48 Parametric Active Contours Definition of the Energy An active contour is a curve v(s) = [x(s), y(s)], where s [0, 1] is the arc length. s v(s) evolves towards a position minimizing the energy functional: E total = E internal (v(s)) + E external (v(s))

49 Parametric Active Contours E internal (v(s)) : Goal: Obtain a smooth contour Penalize the size of the object increased energy with high area and perimeter values. Penalize irregular contours minimize the contour curvature. Constrain the shape of the contour: circles, ellipses,... Minimization Method: optimization. Finite formulation: Exhaustive search of a minimum, global or with probabilistic algorithms. Infinite formulation: local minimum via progressive adaptation, gradient descent or other PDE solvers.

50 Parametric Active Contours E internal (v(s)) : E interne 1 0 s v s s 2 s 2 v s s 2 2 ds length of the contour tension controlled by the elasticity coefficient. Curvature of the contour rigidity controlled by rigidity coefficient. d dy ' ', tan, ds x y ds dx s

51 Parametric Active Contours E external (v(s)) : Standard formulation: Integral of a force potential. Eexterne 1 0 P v s ds Potential : low values on the contours in the image (e.g.: derived from image gradient) P x, y w I x, y 2 P x, y w G x, y * I x, y 2

52 Parametric Active Contours Energy Minimization Goal: find the contour v(s) that minimizes the global energy. Framework: Attract an initial contours towards contours of the image, while avoiding stretching and bending. Method 1: Variational problem formulated with the Euler-Lagrange equation: v s v s s s s s 2 2 P v 0 2 2

53 Parametric Active Contours Energy Minimization Method 2: Dynamic Deformable model: Minimization viewed as a static problem. Build a dynamic system that we evolve towards an equilibrium state according to a Lagrangian mechanical point of view. This dynamical model unifies the shape and motion descriptions, defining an active contour quantification of a shape evolution through time v(s,t).

54 Parametric Active Contours Energy Minimization Method 2: Dynamic Deformable model: Motion equation: according to the 2 nd law of Newton, : 2 v v v 2 2 v s 2 t t s s 2 s 2 s P v Equilibrium state: 2 v t v t 2 0

55 Parametric Active Contours Energy Minimization Methods 1 & 2: no analytical solution (due to external energy). Need to discretize : Finite Differences: each element of the contour is viewed as a point with individual mechanical properties. Finite Elements : sub-elements between nodes. N control points v = (v 1, v 2,..., v N ), distant with a spatial step h.

56 Parametric Active Contours Energy Minimization: Numerical Schemes Discretization of spatial derivatives 2 v v 2v v i i 1 i i s h Matrix Notation ( cst): 4 v v 4v 6v 4v v s i i 2 i 1 i i 1 i h v s 2 4 P v s v s s Av P v Penta diagonal matrix h h h h h h h A h h h h......

57 Parametric Active Contours Energy Minimization: Numerical Schemes Ax Px x, y v x, y, Problems Ay Py x, y Non-linear terms in the potential force. Matrix A non-invertible. Need for iterative numerical schemes Ax P x, y x x n n 1 n 1 n n 1 x Ay P x, y y y Internal forces n n 1 n 1 n n 1 y Inertia coefficient

58 Parametric Active Contours Energy Minimization: Parameters Spatial continuity spatial step smaller than pixel size. = elasticity (i.e. dof of points to move away from each others). β = rigidity. Temporal step set to have a maximum displacement of 1-2 pixels at each iteration.

59 Parametric Active Contours Energy Minimization: Conclusions Advantages Extraction of a locally optimal position via iterative deformation of a curve. Suited for contour extraction: open, closed (v 0 = v N ) With fixed extremities (v 0 et v N fixed) General Framework : several different types exist. Simple an efficient 2D implementation. Numerical stability wrt internal forces.

60 Parametric Active Contours Energy Minimization: Conclusions Limitations Instability wrt external forces: if spatial step too big, can miss some contours. Sensitive to local minima problems and initialization. Difficult parameterization. No change in topology allowed (i.e. division/fusion of objects). No simultaneous deformation of multiple objects.

61 Parametric Active Contours Formulation of the Problem 1. Energy Minimization : Minimize the weighted sum of internal and external energies (Force potentials). Final contour position corresponds to an energy s minimum. 2. Dynamical Forces : Equilibrium between internal and external forces at each point on the contour.

62 Parametric Active Contours Formulation with Dynamical Forces Dynamical problem with more general forces than potential forces. Newton s law : mass C 2 F internal C F external C F viscous C t 2 C t

63 Parametric Active Contours Formulation with Dynamical Forces Simplification: no mass C F internal C F external C t idem Superposition of forces

64 Parametric Active Contours F external : GRADIENT forces Properties of the Gradient Vectors: Point towards the contours (normals). Large modules near edges. Module around 0 in homogeneous regions. Problems: Weak attraction range (only close to edges) No force in homogeneous areas (nothing moves ). To solve : Initialization problems. Convergence towards concave regions.

65 Parametric Active Contours Example F external : Balloon [Cohen et Cohen] : A gradient fore to attract the contours towards edges. A pressure force to constrain the model to inflate/deflate: n +/-: inflate PC Fexterne C k k1n s PC Computational cost: image gradient and normals on the contours for each node. Need to control the dynamical behavior of the contour far from the edges ( weight of k 1 ).

66 Parametric Active Contours Example F externe : Potential Forces for Distances Distance map D(x, y) (Euclidian or Chamfer) for each pixel to the closest point on the contour field of potential forces. D(x, y) defines the potential energy : Pdistance x, y we and the field of forces D x, y Fexterne C Pdistance x, y Problem remaining: concave shapes 2 D x, y

67 Parametric Active Contours Example of F externe : GVF [Xu et Prince 1998] Vector field. Preserve gradient properties near the edges. Diffuse these properties in homogeneous regions via «gradient diffuse».

68 Parametric Active Contours Example of F externe : GVF GVF is a vector field : I x, y V x, y u( x, y) v( x, y) edge xy, V x, y is defined via energy minimization: Regularization on the overall domain E u x, y u x, y v x, y v x, y x y x y x, y V x, y x, y edge 2 2 edge dxdy edge xy, = Edge map of the image Data terms I x, y

69 Parametric Active Contours Example of F externe : GVF The GVF vector field is obtained by solving the Euler equations : 2 2 Iedge x, y Iedge x, y Iedge x, y u x, y u x, y 0 x x y 2 2 Iedge x, y Iedge x, y Iedge x, y v x, y v x, y 0 y x y Laplace Eq. Gradient data term Gradient of the Edge map

70 Parametric Active Contours GVF Final result, iter = 50 GVF force Final result, iter = 50 GVF force Final result, iter = 50 GVF force

71 Parametric Active Contours Example of F externe : GVF Numerical implementation in the computer lab

72 Parametric Active Contours Example of F externe : GVF

73 Parametric Active Contours Bibliography 1. Kass M, Witkin A and Terzopoulos D. Snakes: Active contour models. International Journal of Computer Vision 1987; 1-: Cohen LD and Cohen I. Finite-elements methods for active contour models and balloons for 2-D and 3-D Images. IEEE Transactions on Pattern Analysis and Machine Intelligence 1993; 15-11: Xu C and Prince JL. Snakes, shapes and gradient vector Flow. IEEE Transactions on Image Processing 1998:

74 Active Contours Parametric contours. Geometric contours.

75 Geometric Active Contours Introduction Theory of curve evolution and geometrical flows. The contour deforms with a speed made of 2 terms: Regularizing term (curvature-based motion). Expansion term or contraction to go towards image edges. The active contour is defined via a geometrical flow (PDE). the curve evolution must stop at locations of image edges corresponding to the object to segment.

76 Geometric Active Contours Geometric Active contours Numerical methods via level sets. Geodesic. Mumford-Shah.

77 Geometric Active Contours Curve Evolution Theory: Curve evolution through geometric measures (normal vectors to the curve, curvature,...) and independent of curve parameterization (e.g. derivatives). Let a curve X s, t x( s, t) y( s, t) with spatial parameter s and temporal parameter t The curve evolution in the normal directions controlled by the PDE: Propagation speed X s, t t Geometric measure of the curve V N s N

78 Geometric Active Contours Curve Evolution Theory 1. Constant speed: X s, t VN 0 t similar to a pression force (balloon). 2. Motion under curvature: X s, t t similar to an elasticity force. N, 0

79 Geometric Active Contours Numerical Methods with Level Sets Goal: Numerical methods to compute the spatial propagation of a curve in time: add a temporal dimension. Precise characterization of the geometric properties of the contour. Approach: Define a spatio-temporal function. Immerge the contour in a function as its 0-level (iso-contours). Extend the function to other levels. : 0,0 0, t 0

80 Geometric Active Contours Numerical Methods with Level Sets Definition of the level set function: x, y, t 0, x, y x, y, t 0, x, y x, y, t 0, x, y s, t st, is defined as the 0 level of x, y, t. st, v deforms with a speed applied on each point. How to control the level set motion? i o o st, i

81 Geometric Active Contours Numerical Methods with Level Sets Iterative Deformation Scheme: 1. Define a field of speed vectors (cf. theory of curve evolution). 2. Compute initial values of the level set function, based on the initial position of the contour to evolve. 3. Adjust the function in time, so that the level 0 corresponds to the segmentation solution. Evolution equation for the level set function?

82 Geometric Active Contours Numerical Methods with Level Sets Curve evolution theory: d, t, t cste 0 dt t Evolution equation of the level set function: V t, t N N s, t, t. 0 t st,

83 Geometric Active Contours Numerical Methods with Level Sets Evolution equation of the level set function: t, t V, t. N 0 N s, t st, Normal vector: N, t, t

84 Geometric Active Contours Numerical Methods with Level Sets Evolution equation of the level set function: st, N s, t, t t 0,0 V given, t 0

85 Geometric Active Contours Numerical Methods with Level Sets Geometrical properties of the level set curve: directly computed on the level set function! N, t, t N s, t st,., t 2, t 2 2 xx y x y xy yy x x y

86 Geometric Active Contours Numerical Methods with Level Sets What type of, t 1 function?: Most common choice: signed distance function N, t, t N s, t st, Careful! The solution of V t is not the signed distance function of the curve solution to: V N. t

87 Geometric Active Contours Numerical Methods with Level Sets What speed of propagation? Take into account: Image Information: zero on edges from the objets to segment. Geometric Information of the contour: smoothing via constraints on the countour. Particular case: Motion under curvature Each part of the model evolves in the normal direction, with a speed proportional to the curvature. points can move inward or outward, depending on the curvature s sign.

88 Geometric Active Contours Numerical Methods with Level Sets Speed of propagation? Defined only on the contour only. Extension of the speed over the whole domain : Natural Extension: e.g. with motion under curvature, computing the curvature on the overall level set function. Values of the nearest point from the 0-level. problems since the speed does not depend on f and we are not anymore in the H-J framework. t, t V, t 0 x, y, t t V x, y x, y, t

89 Geometric Active Contours Numerical Methods with Level Sets Reinitialization Why reinitialize? Unique correspondence between a curve and its level set function (convergence iif convergence) Preserve a constant gradient norm numerical stability. Methods : Direct evaluation: Detect the 0-level and re-compute the signed distance function. High computational cost! Iterative: Equilibrium state: norm of gradient =1. Opposite flows for negative and positive values. Problem: the 0-level can move during reinitialization! t sign 1

90 Geometric Active Contours Numerical Methods with Level Sets Narrow Band Only evolve level sets in a narrow band around the level zero. Reduce computational cost. No need to compute evolution speed far from the 0-level. less constraints on t for the CFL stability, limiting the maximum speed of deformation.

91 Geometric Active Contours Level Sets Advantages Change of topology Intrinsic geometric properties easy to compute (normals, curvatures). Extension to N-D straightforward : add new spatial variables to the evolution equation of the volume (x, y, z,, t). Numerical implementation: Discretization of (x, y, t) on regular grid (x, y). Standard numerical schemes for the spatial derivative.

92 Geometric Active Contours Level Sets Limitations 3 limitations related to the numerical implementation: Construction of an initial level set function (x, y, t = 0) from an initial 0-level (initial contour). Evolution equation only defined for the 0-level the speed function V is not defined in general for the other levels arbitrary spatial extension. Instable definition of the normal to the level set function (x, y, t = 0) reinitialization + smoothing.

93 Geometric Active Contours Geometric Active contours Numerical methods via level sets. Geodesic. Mumford-Shah.

94 Geometric Active Contours Geodesic Deformable Models [Caselles, Kimmel, Sapiro 1997] Geodesic curves in a Riemannian space Geodesic: path (locally) minimal between 2 points. Space with metrics defined from geodesics. Novel approach (equivalent) Min E C C ' s ds g I C s ds 0 0 Min 1 0 g I C s C ' s ds Geodesic computation

95 Geometric Active Contours Geodesic Deformable Models Euclidian Geodesics Length L E Motion under curvature C C ' s ds ds C t = N Euclidian Metric and curvature Motion under curvature provides fastest minimization of L E. Geodesics for deformable models 1 Lenght L C g I C s C ' s ds C t R =g I N- g.n N 0 LE 0 C g I C s C ' s ds Image characteristics S

96 Geometric Active Contours Geometric Active contours Numerical methods via level sets. Geodesic. Mumford-Shah.

97 Geometric Active Contours Mumford and Shah Variational method. Image I 0 segmentation, defined on the domain is provided by a pair CI, with: C contours in the image and I a smooth approximation of I 0 Energie associated with the segmentation: 1D, # of points on C 2D, perimeter of C 3D, surface of C 2 2 E C, I I d length C I I d \ C \ C 0

98 Geometric Active Contours Mumford and Shah Conjuncture There exists a minimal segmentation made of a finite set of curves C 1. [Morel-Solimini 1995, Aubert-Kornprobst 2000]: There exists a minimal segmentation. The minimal segmentation is not unique. The ensemble of solutions is a compact set. Contours are rectifiable (i.e. of finite length). All contours can be included in a single rectifiable curve.

99 Geometric Active Contours Mumford and Shah Particular case: I 0 is a cartoon-like image. The smooth approximation I of I 0 is a piecewise constant image with values c 1 et c 2 which are the mean values of I 0 in the object and the background. The contour C corresponds to the contours of the objects. E c, c, C Length C I c I c 1 2 inside C 1 outside C I 0

100 Geometric Active Contours Mumford and Shah Deformable models without Edges [Chan, Vese, IEEE TIP 2001] E C L C A C I c d I c d 2 1 inside C outside C Regularizing terms (cf. internal energy) Homogeneity constraints (cf. external energy) c 1 c 2 = mean value inside C = mean value outside C L(C) = length of C A(C) = area of C

101 Geometric Active Contours Mumford and Shah Deformable models without Edges [Chan, Vese, IEEE TIP 2001] E C L C A C I c d I c d 2 1 inside C outside C Insert the n-d curve C in a level set function (n+1)-d inside outside { N / f ( ) 0} { N / f ( ) 0} { N / f ( ) 0} C = x Î x = C = x Î x < C = x Î x > 2. Define a Heaviside function H( ) H( z) 3. Define a Dirac function ( ) ( z) d = í 1if z 0 = ï ì ï ïî 0if z ³ 0 dh ( z) dz

102 Geometric Active Contours Mumford and Shah Deformable models without Edges [Chan, Vese, IEEE TIP 2001] - Level set Function = distance to the 0- level - Heaviside Function H C x, y, z / x, y, z 0 Area C Area 0 H arctan 2 Length C H Length d 0 - Dirac Function H d d

103 Geometric Active Contours Mumford and Shah Deformable models without Edges [Chan, Vese, IEEE TIP 2001] E, c, c dx H I c H dx I c 1 H dx length area homogeneity homogeneity é æ f ö ù de ( f ) m Ñ 2 2 n l ( ) l ( ) div - - I - c + I - c = çèñ f ø êë úû de ( f) f ò (,, ) ( f (,, ) W = 0 on W, c0 ( f ) = Ñ f n ( f (,, ) Segmentation via inf E, c, c, c, c c 1 ( f ) = ò W I x y z H x y z dxdydz ò W H x y z dxdydz ( ( ) I ( x, y, z) 1 - H f ( x, y, z) dxdydz ò W ( 1 - H ( f ( x, y, z ) dxdydz

104 Geometric Active Contours Bibliography D. Mumford and J. Shah, "Boundary detection by minimizing functional," International Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA, pp , S. Osher and J. A. Sethian, "Fronts propagating with curvaturedependent speed: Algorithms based on Hamilton-Jacobi formulations," Journal of Computational Physics, vol. 79, No. 1, pp , V. Caselles, F. Catté, B. Coll, F. Dibos, A geometric model for edge detection, Num. Mathematik, 66, 1-31, R.Malladi, J.A. Sethian, Baba C. Vemuri: Shape modeling with front propagation: A level set approach '. IEEE Transactions on Pattern Analysis and Machine Intelligence 17(2): , T. F. Chan and L. A. Vese, "Active contours without edges," IEEE Transactions on Image Processing, vol. 10, No. 2, pp , 2001.

105 Active Contours Two families of N-D Active Contours: Parametric : Explicit representation of the contour. Compact representation allowing fast implementation (enable real-time applications). Changes of topology very difficult to handle (in 3D). Geometric: Implicit representation of the contour as the level 0 of a scalar function of dimension (N-D+1). Contour parameterization after the deformations. Flexible adaptation of the contours topology. Increase dimension of space search.