Fitting: Deformable contours April 26 th, 2018

Transcription

1 4/6/08 Fttng: Deformable contours Aprl 6 th, 08 Yong Jae Lee UC Davs Recap so far: Groupng and Fttng Goal: move from array of pxel values (or flter outputs) to a collecton of regons, objects, and shapes. Slde credt: Groupng: Pxels vs. regons mage mage clusters on ntensty clusters on color By groupng pxels based on Gestaltnspred attrbutes, we can map the pxels nto a set of regons. ach regon s consstent accordng to the features and smlarty metrc we used to do the clusterng. 3

2 4/6/08 Fttng: dges vs. boundares dges useful sgnal to ndcate occludng boundares, shape. Here the raw edge output s not so bad Images from D. Jacobs but qute often boundares of nterest are fragmented, and we have extra 4 clutter edge ponts. Fttng: dges vs. boundares Gven a model of nterest, we can overcome some of the mssng and nosy edges usng fttng technques. Wth votng methods lke the Hough transform, detected ponts vote on possble model parameters. 5 Votng wth Hough transform Hough transform for fttng lnes, crcles, arbtrary shapes y (x, y ) y 0 (x0, y 0 ) b x 0 x mage space m Hough space In all cases, we knew the explct model to ft. 6

3 4/6/08 Today Fttng an arbtrary shape wth actve deformable contours 7 Deformable contours a.k.a. actve contours, snakes Gven: ntal contour (model) near desred object Slde credt: [Snakes: Actve contour models, Kass, Wtkn, & Terzopoulos, ICCV987] 8 Fgure credt: Yur Boykov Deformable contours a.k.a. actve contours, snakes Gven: ntal contour (model) near desred object Goal: evolve the contour to ft exact object boundary Man dea: elastc band s teratvely adjusted so as to be near mage postons wth hgh gradents, and satsfy shape preferences or contour prors Slde credt: [Snakes: Actve contour models, Kass, Wtkn, & Terzopoulos, ICCV987] 9 Fgure credt: Yur Boykov 3

4 4/6/08 Deformable contours: ntuton 0 Image from Deformable contours vs. Hough Lke generalzed Hough transform, useful for shape fttng; but ntal ntermedate fnal Hough Rgd model shape Sngle votng pass can detect multple nstances Deformable contours Pror on shape types, but shape teratvely adjusted (deforms) Requres ntalzaton nearby One optmzaton pass to ft a sngle contour Why do we want to ft deformable shapes? Some objects have smlar basc form but some varety n the contour shape. 4

5 4/6/08 Why do we want to ft deformable shapes? Non-rgd, deformable objects can change ther shape over tme, e.g. lps, hands Fgure from Kass et al Why do we want to ft deformable shapes? Non-rgd, deformable objects can change ther shape over tme, e.g. lps, hands 4 Why do we want to ft deformable shapes? Non-rgd, deformable objects can change ther shape over tme. Fgure credt: Julen Jomer 5 5

6 4/6/08 Aspects we need to consder Representaton of the contours Defnng the energy functons xternal Internal Mnmzng the energy functon xtensons: Trackng Interactve segmentaton 6 Representaton We ll consder a dscrete representaton of the contour, consstng of a lst of d pont postons ( vertces ). ( 0 x 0, y ) ( x 9, y9) x, y ), for ( 0,,, n At each teraton, we ll have the opton to move each vertex to another nearby locaton ( state ). 7 Fttng deformable contours How should we adjust the current contour to form the new contour at each teraton? Defne a cost functon ( energy functon) that says how good a canddate confguraton s. Seek next confguraton that mnmzes that cost functon. ntal ntermedate fnal Slde credt: 8 6

7 4/6/08 nergy functon The total energy (cost) of the current snake s defned as: total nternal external Internal energy: encourage pror shape preferences: e.g., smoothness, elastcty, partcular known shape. xternal energy ( mage energy): encourage contour to ft on places where mage structures exst, e.g., edges. A good ft between the current deformable contour and the target shape n the mage wll yeld a low energy. Slde credt: 9 xternal energy: ntuton Measure how well the curve matches the mage data Attract the curve toward dfferent mage features dges, lnes, texture gradent, etc. 0 Slde credt: xternal mage energy How do edges affect snap of rubber band? Thnk of external energy from mage as gravtatonal pull towards areas of hgh contrast Magntude of gradent G Slde credt: x( I ) Gy( I) - (Magntude of gradent) G x( I) Gy( I) 7

8 4/6/08 xternal mage energy Gradent mages G x ( x, y) and ( x, y) G y xternal energy at a pont on the curve s: external ( ) ( G ( ) G ( ) ) xternal energy for the whole curve: external x n Gx( x, y ) Gy ( x, y 0 y ) Internal energy: ntuton What are the underlyng boundares n ths fragmented edge mage? And n ths one? 3 Internal energy: ntuton A pror, we want to favor smooth shapes, contours wth low curvature, contours smlar to a known shape, etc. to balance what s actually observed (.e., n the gradent mage). 4 8

9 4/6/08 Internal energy For a contnuous curve, a common nternal energy term s the bendng energy. At some pont v(s) on the curve, ths s: d ds d d s nternal ( ( s)) Tenson, lastcty Stffness, Curvature 5 Internal energy For our dscrete representaton, d ds ( x, y ) 0 n d ds v ( ) ( ) Internal Note these energy are dervatves for the whole relatve curve: to spatal poston. nternal n 0 Why do these reflect tenson and curvature? 6 xample: compare curvature ( v ) curvature ( x x x ) ( y y y ) (,5) (,) (,) (3,) (,) (3,) 7 9

10 4/6/08 Penalzng elastcty Current elastc energy defnton: elastc n 0 n ( x x ) ( y y ) 0 What s the possble problem wth ths defnton? 8 Penalzng elastcty Current elastc energy defnton: Instead: elastc n 0 n 0 ( x x ) ( y y ) d where d s the average dstance between pars of ponts updated at each teraton. 9 Dealng wth mssng data The preferences for low-curvature, smoothness help deal wth mssng data: Illusory contours found! Slde credt: [Fgure from Kass et al. 987] 30 0

11 4/6/08 xtendng the nternal energy: capture shape pror If object s some smooth varaton on a known shape, we can use a term that wll penalze devaton from that shape: nternal n 0 ( ˆ ) where { ˆ } are the ponts of the known shape. 3 Fg from Y. Boykov Total energy: functon of the weghts total nternal external external nternal n 0 n Gx( x, y ) Gy ( x, y 0 ) d 3 Total energy: functon of the weghts e.g., weght controls the penalty for nternal elastcty large medum small Slde credt: 33 Fg from Y. Boykov

12 4/6/08 Recap: deformable contour A smple elastc snake s defned by: A set of n ponts, An nternal energy term (tenson, bendng, plus optonal shape pror) An external energy term (gradent-based) To use to segment an object: Intalze n the vcnty of the object Modfy the ponts to mnmze the total energy 34 nergy mnmzaton Several algorthms have been proposed to ft deformable contours. We ll look at two: Greedy search Dynamc programmng 35 Slde credt: nergy mnmzaton: greedy For each pont, search wndow around t and move to where energy functon s mnmal Typcal wndow sze, e.g., 3 x 3 pxels Stop when predefned number of ponts have not changed n last teraton, or after max number of teratons Note: Convergence not guaranteed Need decent ntalzaton 36

13 4/6/08 nergy mnmzaton Several algorthms have been proposed to ft deformable contours. We ll look at two: Greedy search Dynamc programmng 37 Slde credt: nergy mnmzaton: dynamc programmng v 3 v v v 4 v 5 v 6 Wth ths form of the energy functon, we can mnmze usng dynamc programmng, wth the Vterb algorthm. Iterate untl optmal poston for each pont s the center of the box,.e., the snake s optmal n the local search space constraned by boxes. 38 Fg from Y. Boykov [Amn, Weymouth, Jan, 990] nergy mnmzaton: dynamc programmng Possble because snake energy can be rewrtten as a sum of par-wse nteracton potentals: total n,, n) (, ) ( 39 3

14 4/6/08 Snake energy: par-wse nteractons n total ( x,, xn, y,, yn) Gx ( x, y ) Gy ( x, y ) Re-wrtng the above wth v x, y : n total (,, n) G( ) total n ( x x ) ( y y ) n (,, n) ( v, v) ( v, v3)... n ( vn, vn) where (, ) G( ) 40 Vterb algorthm () for each possble poston (state) of vertex, fnd cost of optmal path arrvng there, and optmal poston of predecessor. () backtrack from best state for last vertex. total ( v, v) ( v, v3)... n ( vn, vn ) states m vertces ( v, v) ( v, v3) 3( v3, v4) v v v3 v4 ( ) ( m) 4 3 m 4 ( v 4, v n ) vn () 0 () 3 () 4 () n () () () 0 3 () 4 () n () (3) 0 (3) 3 (3) 4 (3) n (3) ( ) 0 ( m) m n (m) Complexty: O( nm ) vs. brute force search? 4 xample adapted from Y. Boykov nergy mnmzaton: dynamc programmng v 3 v v v 4 v 5 v 6 Wth ths form of the energy functon, we can mnmze usng dynamc programmng, wth the Vterb algorthm. Iterate untl optmal poston for each pont s the center of the box,.e., the snake s optmal n the local search space constraned by boxes. Slde credt: 4 Fg from Y. Boykov [Amn, Weymouth, Jan, 990] 4

15 4/6/08 nergy mnmzaton: dynamc programmng DP can be appled to optmze an open ended snake ( v, v) ( v, v3)... n ( vn, vn) n For a closed snake, a loop s ntroduced nto the total energy. ( v, v) ( v, v3)... n ( vn, vn) n( vn, v) Slde credt: n n 4 3 Work around: ) Fx v and solve for rest. ) Fx an ntermedate node at ts poston found n (), solve for rest. 43 Aspects we need to consder Representaton of the contours Defnng the energy functons xternal Internal Mnmzng the energy functon xtensons: Trackng Interactve segmentaton 44 Slde credt: Trackng va deformable contours. Use fnal contour/model extracted at frame t as an ntal soluton for frame t+. volve ntal contour to ft exact object boundary at frame t+ 3. Repeat, ntalzng wth most recent frame. Trackng Heart Ventrcles (multple frames) 45 5

16 4/6/08 Trackng va deformable contours Vsual Dynamcs Group, Dept. ngneerng Scence, Unversty of Oxford. Applcatons: Traffc montorng Human-computer nteracton Anmaton Survellance Computer asssted dagnoss n medcal magng 46 Lmtatons May over-smooth the boundary Cannot follow topologcal changes of objects Slde credt: 47 Lmtatons xternal energy: snake does not really see object boundares n the mage unless t gets very close to t. mage gradents I are large only drectly on the boundary 48 6

17 4/6/08 Dstance transform xternal mage can nstead be taken from the dstance transform of the edge mage. orgnal -gradent dstance transform edges Value at (x,y) tells how far that poston s from the nearest edge pont (or other bnary mage structure) 49 Deformable contours: pros and cons Pros: Useful to track and ft non-rgd shapes Contour remans connected Possble to fll n llusory contours Flexblty n how energy functon s defned, weghted Cons: Must have decent ntalzaton near true boundary, may get stuck n local mnmum Hyper-parameters of energy functon must be set well 50 Summary Deformable shapes and actve contours are useful for Segmentaton: ft or snap to boundary n mage Trackng: prevous frame s estmate serves to ntalze the next Fttng actve contours: Defne terms to encourage certan shapes, smoothness, low curvature, push/pulls, Use weghts to control relatve nfluence of each component cost Can optmze snakes wth Vterb algorthm 5 7

18 4/6/08 Questons? See you Tuesday! 5 8