Dr. Ulas Bagci

Transcription

1 Lecture 9: Deformable Models and Segmentation CAP-Computer Vision Lecture 9-Deformable Models and Segmentation Dr. Ulas Bagci

2 Lecture 9: Deformable Models and Segmentation Motivation A limitation of active contours based on parametric curves of the form f(s) (snakes, b- snakes, ) is that it is challenging to change the topology of the curve as it evolves.

3 Lecture 9: Deformable Models and Segmentation Motivation A limitation of active contours based on parametric curves of the form f(s) (snakes, b- snakes, ) is that it is challenging to change the topology of the curve as it evolves. If the shape changes dramatically, curve reparameterization may also be required.

4 Lecture 9: Deformable Models and Segmentation Motivation A limitation of active contours based on parametric curves of the form f(s) (snakes, b- snakes, ) is that it is challenging to change the topology of the curve as it evolves. If the shape changes dramatically, curve reparameterization may also be required. An alternative representation for such closed contours is to use level sets (LS). LS evolve to fit and track objects of interest by modifying the underlying embedding function instead of curve function f(s)

5 Lecture 9: Deformable Models and Segmentation Basics of Level Sets Energy measures the consistency of the image statistics inside and outside the regions Color image Cremers, Rousson, and Deriche 7

6 Lecture 9: Deformable Models and Segmentation Image Segmentation with Level Sets Contour evolution (Sethian and Osher, 988) Level sets for closed contours Zero-crossing(s) of a characteristic function define the curve Fit and track objects of interest by modifying the underlying embedding function φ( xy, ) instead of the curve f(s) Efficient algorithm A small strip around the locations of the current zerocrossing needs to updated at each step Fast Marching Methods 6

7 Lecture 9: Deformable Models and Segmentation 7

8 Lecture 9: Deformable Models and Segmentation Moving Interfaces D Moving Curves D Moving Surfaces water air Ex: Interfaces between water and oil Propagating front of bush fire Deformable elastic solid 8

9 Lecture 9: Deformable Models and Segmentation Evolving Curves and Surfaces Tangential motion does not change the interface! Only velocity component normal to surface is important! 9

10 Lecture 9: Deformable Models and Segmentation Traditional Way (non-ls)

11 Lecture 9: Deformable Models and Segmentation The Problem

12 Lecture 9: Deformable Models and Segmentation So what is LS?

13 Lecture 9: Deformable Models and Segmentation Ex. Describe curve as Level Sets of a Function (x, y) =x + y = Isocontour is the unit circle (implicit represt.)

14 Lecture 9: Deformable Models and Segmentation Ex. Describe curve as Level Sets of a Function (x, y) =x + y = A few isocontours of two dimensional function (circle) Along with some representative normals. )

15 Lecture 9: Deformable Models and Segmentation Describe curve as Level Sets of a Function Then, unit normal (outward) is ~N =

16 Lecture 9: Deformable Models and Segmentation Describe curve as Level Sets of a Function Then, unit normal (outward) is ~N = On Cartesian grid, we need to approximate this equation (ex. Finite i+ i x 6

17 Lecture 9: Deformable Models and Segmentation Describe curve as Level Sets of a Function Then, unit normal (outward) is ~N = On Cartesian grid, we need to approximate this equation (ex. Finite i+ i x Mean curvature of the interface is defined as the divergence of the normal ~ N =(n,n ) apple = r. ~ @y = r.( r r ) 7

18 Lecture 9: Deformable Models and Segmentation Describe curve as Level Sets of a Function Then, unit normal (outward) is ~N = On Cartesian grid, we need to approximate this equation (ex. Finite i+ i x Mean curvature of the interface is defined as the divergence of the normal ~ N =(n,n ) apple = r. ~ @y = r.( r r ) 8

19 Lecture 9: Deformable Models and Segmentation Describe curve as Level Sets of a Function concave convex 9

20 Lecture 9: Deformable Models and Segmentation Variational Formulations and LS Transition from Active Contours: contour v(t) front γ(t) contour energy forces F A F C image energy speed function k I Level set: The level set c at time t of a function ψ(x,y,t) is the set of arguments { (x,y), ψ(x,y,t) = c } Idea: define a function ψ(x,y,t) so that at any time, γ(t) = { (x,y), ψ(x,y,t) = } there are many such ψ ψ has many other level sets, more or less parallel to γ only γ has a meaning for segmentation, not any other level set of ψ

21 Lecture 9: Deformable Models and Segmentation Level Set Framework Usual choice for ψ: signed distance to the front γ() - d(x,y, γ) if (x,y) inside the front ψ(x,y,) = on d(x,y, γ) outside ψ(x,y,t) ψ(x,y,t) γ(t)

22 Lecture 9: Deformable Models and Segmentation Level Set Framework ψ(x,y,t) no movement, only change of values the front may change its topology the front location may be between samples ψ(x,y,t+) = ψ(x,y,t) + ψ(x,y,t)

23 Lecture 9: Deformable Models and Segmentation Level Set Segmentation with LS: Initialise the front γ() Compute ψ(x,y,) Iterate: ψ(x,y,t+) = ψ(x,y,t) + ψ(x,y,t) until convergence Mark the front γ(t end )

24 Lecture 9: Deformable Models and Segmentation Recap: Variational Formulations and LS Transition from Active Contours: contour v(t) front γ(t) contour energy forces F A F C image energy speed function k I Level set: The level set c at time t of a function ψ(x,y,t) is the set of arguments { (x,y), ψ(x,y,t) = c } Idea: define a function ψ(x,y,t) so that at any time, γ(t) = { (x,y), ψ(x,y,t) = } there are many such ψ ψ has many other level sets, more or less parallel to γ only γ has a meaning for segmentation, not any other level set of ψ

25 Lecture 9: Deformable Models and Segmentation Front Propagation product of influences ψ(x,y,t+) - ψ(x,y,t) extension of the speed function k I (image influence) ψ t + ˆ k I F A + F G (κ) ( ) ψ = constant force (balloon pressure) smoothing force depending on the local curvature κ (contour influence) spatial derivative of ψ % κ = div ψ ( ' * & ψ ) link between spatial and temporal derivatives, but not the same type of motion as contours!

26 Lecture 9: Deformable Models and Segmentation Front Propagation Speed function: k I is meant to stop the front on the object s boundaries similar to image energy: k I (x,y) = / ( + I (x,y) ) only makes sense for the front (level set ) yet, same equation for all level sets extend k I to all level sets, defining ^k I possible extension: ^ k I (x,y) = k I (x,y ) where (x,y ) is the point in the front closest to (x,y) ^ ( such a k I (x,y) depends on the front location ) 6

27 Lecture 9: Deformable Models and Segmentation LS Algorithm ψ(x,y,t). compute the speed k I on the front extend it to all other level sets. compute ψ(x,y,t+) = ψ(x,y,t) + ψ(x,y,t). find the front location (for next iteration) modify ψ(x,y,t+) by linear interpolation

28 Lecture 9: Deformable Models and Segmentation Narrow Band Extension Weaknesses of algorithm update of all ψ(x,y,t): inefficient, only care about the front speed extension: computationally expensive Improvement: narrow band: only update a few level sets around γ other extended speed: k I (x,y) = / ( + I(x,y) )

29 Lecture 9: Deformable Models and Segmentation Narrow Band Extension Caution: extrapolate the curvature κ at the edges re-select the narrow band regularly: an empty pixel cannot get a value may restrict the evolution of the front

30 Lecture 9: Deformable Models and Segmentation Summary of LS Level sets: function ψ : [, I width ] x [, I height ] x N R ( x, y, t ) ψ(x,y,t) embed a curve γ: γ(t) = { (x,y), ψ(x,y,t) = } γ() is provided externally, ψ(x,y,) is computed ψ(x,y,t+) is computed by changing the values of ψ(x,y,t) changes using a product of influences on convergence, γ(t end ) is the border of the object Issue: computation time (improved with narrow band)

31 Lecture 9: Deformable Models and Segmentation Examples

32 Lecture 9: Deformable Models and Segmentation Summary of Boundary Based Methods Curves: object boundaries Intelligent scissors (live-wire) sketch in real time a curve that clings to object boundaries Snakes energy-minimizing, D spline curve that evolves towards image features such as strong edges Level set Evolve the curve as the zero-set of a characteristic function, Easily change topology and incorporate region-based statistics

33 Lecture 9: Deformable Models and Segmentation Applications: Contour Tracking and Rotoscoping Track facial features for performance-driven animation Track heads and people, moving vehicles Medical image segmentation (CT image) Rotoscoping Agarwala, Hertzmann, Seitz et al. ()

34 Lecture 9: Deformable Models and Segmentation

35 Lecture 9: Deformable Models and Segmentation Applications: Image Restoration SNR is approximately Original Image

36 Lecture 9: Deformable Models and Segmentation Applications: Image Restoration SNR is approximately Original Image Recovered Image!! 6

37 Lecture 9: Deformable Models and Segmentation Applications: Reconstruction of Surfaces from Unorganized Data Points Reconstruction of a rat brain from data of MRI slices 7

38 Lecture 9: Deformable Models and Segmentation ICCV : Level Set Person Segmentation and Tracking Localized appearance, top-down shape information and level set were combined to segment and track people in videos. 8

39 Lecture 9: Deformable Models and Segmentation CVPR : Level Set Segmentation (>8 citation) Ultrasound image segmentation. Chunming Li et al. LS Evolution without reinitialization: a new variational formulation. 9

40 Lecture 9: Deformable Models and Segmentation Application: Vein Segmentation

41 Lecture 9: Deformable Models and Segmentation Pedestrian Segmentation

42 Lecture 9: Deformable Models and Segmentation Motion Segmentation

43 Lecture 9: Deformable Models and Segmentation Yet another segmentation example

44 Lecture 9: Deformable Models and Segmentation Tumor Segmentation

45 Lecture 9: Deformable Models and Segmentation Slice Credits and References Osher and Paragios (), Paragios, Faugeras, Chan et al. (), Paragios and Sgallari (9) G. Strang, Lecture Notes, MIT. Malladi, Sethian, Vemuri. IEEE PAMI 99. R. Szelisky, Lecture Presentations. Sethian, JA. Fast Marching. PNAS 996. Osher and Fedkiw. Level set methods and dynamic implicit surfaces. Lim, Bagci, and Li. IEEE TBME [Willmore Flow and Level Set]