Actve Contours/Snakes Erkut Erdem Acknowledgement: The sldes are adapted from the sldes prepared by K. Grauman of Unversty of Texas at Austn
Fttng: Edges vs. boundares Edges useful sgnal to ndcate occludng boundares, shape. local entty Here the raw edge output s not so bad but qute often boundares of nterest are fragmented, and we have extra clutter edge ponts. Images from D. Jacobs
Fttng: Edges vs. boundares Detectng boundares requres groupng Gven a model of nterest, we can overcome some of the mssng and nosy edges usng fttng technques. Wth votng methods lke the Hough transform, detected ponts vote on possble model parameters.
Deformable contours a.k.a. actve contours, snakes Gven: ntal contour (model) near desred object Goal: evolve the contour to ft exact object boundary Man dea: elastc band s teratvely adjusted so as to be near mage postons wth hgh gradents, and satsfy shape preferences or contour prors [Snakes: Ac?ve contour models, Kass, Wtkn, & Terzopoulos, ICCV987] Fgure credt: Yur Boykov
Deformable contours: ntuton Image from http://www.healthlne.com/blogs/exercse_ftness/uploaded_mages/handband-795868.jpg
Deformable contours vs. Hough Lke generalzed Hough transform, useful for shape fttng; but ntal ntermedate fnal Hough Rgd model shape Sngle votng pass can detect multple nstances Deformable contours Pror on shape types, but shape teratvely adjusted (deforms) Requres ntalzaton nearby One optmzaton pass to ft a sngle contour
Why do we want to ft deformable shapes? Some objects have smlar basc form. but some varety n the contour shape.
Why do we want to ft deformable shapes? Non-rgd, deformable objects can change ther shape over tme, e.g. lps, hands Fgure from Kass et al. 987
Why do we want to ft deformable shapes? Non-rgd, deformable objects can change ther shape over tme. Fgure credt: Julen Jomer
Aspects we need to consder Representaton of the contours Defnng the energy functons External Internal Mnmzng the energy functon Extensons: Trackng Interactve segmentaton
Representaton We ll consder a dscrete representaton of the contour, consstng of a lst of d pont postons ( vertces ). ( x 0, y0 ) ( x, y 9 9) = ( x, y ), for = 0,,, n explct, parametrc representaton At each teraton, we ll have the opton to move each vertex to another nearby locaton ( state ).
Fttng deformable contours How should we adjust the current contour to form the new contour at each teraton? Defne a cost functon ( energy functon) that says how good a canddate confguraton s. Seek next confguraton that mnmzes that cost functon. ntal ntermedate fnal
Energy functon The total energy (cost) of the current snake s defned as: E = E + total nternal E external Internal energy: encourage pror shape preferences: e.g., smoothness, elastcty, partcular known shape. External energy ( mage energy): encourage contour to ft on places where mage structures exst, e.g., edges. A good ft between the current deformable contour and the target shape n the mage wll yeld a low value for ths cost functon.
External energy: ntuton Measure how well the curve matches the mage data Attract the curve toward dfferent mage features Edges, lnes, texture gradent, etc.
External mage energy How do edges affect snap of rubber band? Thnk of external energy from mage as gravtatonal pull towards areas of hgh contrast Magntude of gradent G x ( I) + Gy( I ) - (Magntude of gradent) Gx ( I) + Gy( I) ( )
External mage energy Gradent mages G x ( x, y) and ( x, y) G y External energy at a pont on the curve s: E ( ) = ( G ( ) + external x G y ( ) ) External energy for the whole curve: n Eexternal = Gx( x, y ) + Gy ( x, y = 0 )
Internal energy: ntuton What are the underlyng boundares n ths fragmented edge mage? And n ths one?
Internal energy: ntuton A pror, we want to favor smooth shapes, contours wth low curvature, contours smlar to a known shape, etc. to balance what s actually observed (.e., n the gradent mage).
Internal energy For a contnuous curve, a common nternal energy term s the bendng energy. At some pont v(s) on the curve, ths s: d d E nternal ( ( s)) = α + β ds d s Tenson, Elastcty Stffness, Curvature
For our dscrete representaton, Internal energy for the whole curve: 0 ), ( = = n y x v ds d + ) ( ) ( + + + = ds d = + + + + = 0 n nternal E β α Note these are derva-ves rela-ve to pos%on- - - not spa-al mage gradents. Why do these reflect tenson and curvature? Internal energy
Penalzng elastcty Current elastc energy defnton uses a dscrete estmate of the dervatve: Eelastc = n = 0 α + n = α ( x+ x ) + ( y+ y ) = 0 What s the possble problem wth ths defnton? Krsten Grauman
Penalzng elastcty Current elastc energy defnton uses a dscrete estmate of the dervatve: Instead: Eelastc = = n = 0 α α n = 0 + ) ( x x ) + ( y y d + + ) where d s the average dstance between pars of ponts updated at each teraton. Krsten Grauman
Dealng wth mssng data The preferences for low-curvature, smoothness help deal wth mssng data: Illusory contours found! [Fgure from Kass et al. 987]
Extendng the nternal energy: capture shape pror If object s some smooth varaton on a known shape, we can use a term that wll penalze devaton from that shape: Enternal where shape. ˆ } { + = α n = 0 ( ˆ ) are the ponts of the known Fg from Y. Boykov
Total energy: functon of the weghts external nternal total E E E γ + = ( ) = + + + + = 0 n nternal d E β α 0 ), ( ), ( y n x external y x G y x G E = + =
Total energy: functon of the weghts e.g., α weght controls the penalty for nternal elastcty large α medum α small α Fg from Y. Boykov
Recap: deformable contour A smple elastc snake s defned by: A set of n ponts, An nternal energy term (tenson, bendng, plus optonal shape pror) An external energy term (gradent-based) To use to segment an object: Intalze n the vcnty of the object Modfy the ponts to mnmze the total energy
Energy mnmzaton Several algorthms have been proposed to ft deformable contours: Greedy search Dynamc programmng (for d snakes)
Energy mnmzaton: greedy For each pont, search wndow around t and move to where energy functon s mnmal Typcal wndow sze, e.g., 5 x 5 pxels Stop when predefned number of ponts have not changed n last teraton, or after max number of teratons Note: Convergence not guaranteed Need decent ntalzaton
Energy mnmzaton: dynamc programmng v 3 v v v 4 v 5 v 6 Wth ths form of the energy functon, we can mnmze usng dynamc programmng, wth the Vterb algorthm. Iterate untl optmal poston for each pont s the center of the box,.e., the snake s optmal n the local search space constraned by boxes. Fg from Y. Boykov [Amn, Weymouth, Jan, 990]
E Energy mnmzaton: dynamc programmng Possble because snake energy can be rewrtten as a sum of par-wse nteracton potentals: total n =,, n) E (, + ) = ( Or sum of trple-nteracton potentals. E total n,, n) = E (,, + ) = (
Snake energy: par-wse nteractons ), ( ), ( ),,,,, ( y n x n n total y x G y x G y y x x E = + = ) ( ) ( n y y x x + + + = + α = = ) ( ),, ( n n total G E = + + n α ), (... ), ( ), ( ),, ( 3 n n n n total v v E v v E v v E E + + + = ) ( ), ( G E α + = + + where Re-wrtng the above wth : ( ) y x v, =
Aspects we need to consder Representaton of the contours Defnng the energy functons External Internal Mnmzng the energy functon Extensons: Trackng Interactve segmentaton
Trackng va deformable contours. Use fnal contour/model extracted at frame t as an ntal soluton for frame t+. Evolve ntal contour to ft exact object boundary at frame t+ 3. Repeat, ntalzng wth most recent frame. Trackng Heart Ventrcles (multple frames)
Jörgen Ahlberg http://www.cvl.sy.lu.se/scout/masters/papers/ex708.pdf 3D actve contours
Lmtatons May over-smooth the boundary Cannot follow topologcal changes of objects
Lmtatons External energy: snake does not really see object boundares n the mage unless t gets very close to t. mage gradents I are large only drectly on the boundary
Dstance transform External mage can nstead be taken from the dstance transform of the edge mage. orgnal - gradent dstance transform edges Value at (x,y) tells how far that poston s from the nearest edge pont (or other bnary mage structure)
Deformable contours: pros and cons Pros: Useful to track and ft non-rgd shapes Contour remans connected Possble to fll n subjectve contours Flexblty n how energy functon s defned, weghted. Cons: Must have decent ntalzaton near true boundary, may get stuck n local mnmum Parameters of energy functon must be set well based on pror nformaton
Summary Deformable shapes and actve contours are useful for Segmentaton: ft or snap to boundary n mage Trackng: prevous frame s estmate serves to ntalze the next Fttng actve contours: Defne terms to encourage certan shapes, smoothness, low curvature, push/pulls, Use weghts to control relatve nfluence of each component cost Can optmze d snakes wth Vterb algorthm. Image structure (esp. gradents) can act as attracton force for nteractve segmentaton methods.