n Proc. Internatonal Conference on Pattern Recognton, Venna, 1996 Hermte Deformable Contours D.M. Gavrla Computer Vson Laboratory, CfAR, Unversty of Maryland College Park, MD 074, U.S.A. gavrla@umacs.umd.edu http://www.umacs.umd.edu/users/gavrla/ Abstract We propose the Hermte representaton for deformable contour ndng. Ths representaton compares favorably n terms of versatlty and controlablty wth other local contour representatons that have been used prevously for ths purpose. The Hermte representaton allows a compact representaton of curved shapes, wthout the smoothng out of corners. It s also well suted for both nteractve and trackng applcatons. The Hermte representaton s used to formulate the contour ndng problem as an optmzaton problem usng a maxmum a posteror energy crteron. Optmzaton s performed by dynamc programmng. Our approach tocontour trackng decouples the eects of transformaton and deformaton, usng a template matchng strategy to robustly account for the transformaton eect. We demonstrate these deas on a varety of mages from derent domans. 1 Introducton Image segmentaton by boundary ndng s one of the central problems n computer vson. Ths s because amongst features that can be used to dstngush objects from ther backgrounds, such as color and texture, shape s usually the most powerful. For detectng nstances of objects wth xed and known shape, the Hough-transform or a template matchng technque s well suted (see [3]). For cases where there exsts some exblty n the object shape (ether w.r.t. a prevous frame n a trackng applcaton, or w.r.t. a user suppled shape n an nteractve object delneaton settng) deformable contour models have found wdespread use. Deformable contours (also called actve contour models, or "snakes") are energy-mnmzng models for whch the mnma represent solutons to contour segmentaton problems. They can overcome problems of tradtonal bottom-up segmentaton methods, such as edge gaps and spurous edges, by the use of an energy functon that contans shape nformaton n addton to terms determned by mage features. The addtonal shape nformaton can be seen as a regularzaton term n the ttng process. Once placed n mage space, the contour deforms to nd the most salent contour n ts neghborhood, under the nuence of the generated potental eld. An extensve amount ofwork has been reported on deformable contours snce ther emergence n the late eghtes among others [1], [4]-[6], [8]-[1], [14]-[16]. A useful way to characterze the derent approaches s along the followng dmensons: contour representaton energy formulaton (nternal and external) contour propagaton mechansm (spatal and temporal) We revew the varous contour representatons that have been used n Secton. A new local representaton s proposed for the deformable contour framework, based on Hermte nterpolatng cubcs, see Secton 3. Its use has several advantages, as wll become apparent. The man plus s that t handles both smooth and polygonal curves naturally. We formulate the soluton to the contour ndng problem by amaxmum a posteror (MAP) crteron. Ths leads to an nternal energy formulaton whch contans squared terms of devatons from the expected Hermte parameter values. The external energy terms descrbe the typcal mage gradent correlatons. See Secton 4.1. The resultng energy mnmzaton s performed by dynamc programmng whch gves the optmal soluton to contour ndng for a certan search regon, see Secton 4.. One of the well-known lmtatons of deformable contours s that ther ntal placement has to be close to the desred object boundary n order to converge. In trackng applcatons, ths assumpton mght be volated. To keep the problem computatonally tractable, we propose to decouple the eects of transformaton and deformaton, see Secton 4.3. Experments on a varety of mages are presented n Secton 5, after whch we conclude n Secton 6. Contour Representatons - A Revew Contour representatons can be roughly dvded nto two classes, dependng on whether they are global or local. Global representatons are those where changes n one shape parameter aect the entre contour, and conversely, local change of the contour shape aects all parameters. These representatons are typcally compact, descrbng shape n terms of only a few parameters. Ths s an advantage n a recognton context,.e. when tryng to recover these parameters from mages, because of lower complexty. A useful class of shapes easly modeled by a few global parameters are the super-quadrcs [15], whch are general-
zatons of ellpses that nclude a degree of "squareness". To these shapes, one can add global deformatons, such as taperng, twstng and bendng []. A more general global representaton s the Fourer representaton [14]. It expresses a parametrzed contour n terms of a number of orthonormal (snusodal) bass functons. Arbtrary contours can be represented n any detal desred, gven a sucent number of bass functons. Local representatons control shape locally byvar- ous parameters. Ths exblty makes local representatons well suted n a shape reconstructon context, as s the case when deformng a contour to t mage data. The smplest contour representaton s an ordered lst of data ponts. More compact representatons descrbe contours n terms of pecewse polynomals. Each segment of the parametrzed contour (x (t) y (t)) s descrbed by a polynomal n t. The lowest-degree nterpolatng polynomal s of degree one, leadng to a contour representaton by polylnes and polygons. More exblty s possble by the use of hgher order polynomals, generally cubc polynomals they are the lowest degree polynomals for whch dervatves at the endponts can be spec- ed. Hgher order polynomals tend to bounce back and forth n less controlable fashon and therefore are used less frequently for nterpolaton purposes. Natural cubc splnes are pecewse thrd degree polynomals whch nterpolate control ponts wth C 0, C 1 and C contnuty. The natural cubc splne parameters depend on all control ponts, whch makes t a global representaton. B-splnes on the other hand, have a local representaton, where contour segments depend only on a few neghborng control ponts. Ths comes at a prce of not nterpolatng the control ponts. The same C 0, C 1 and C contnuty as natural splnes s now acheved at the jon ponts of connectng segments. By replcatng control ponts, one can force the B-splne to nterpolate the control ponts. A last nterestng property sthat the B-splne can be speced such that t performs a least-squares t on the avalable data ponts. In prevous work, three local representatons have been used for deformable contour ndng: pont representatons, polygonal chans and unform B-splnes. These representatons have the followng dsadvantages when used for the contour ndng task. Manpulatng contours on the ne scale oered by pxel-by-pxel representatons leads typcally to hgh computatonal cost (for example, note the hgh complexty ncurred n [8]). The ncorporaton of a-pror shape nformaton n ths featureless representaton s dcult. If, on the other hand, a contour s represented by a few (feature) ponts, and contour ndng only consders mage data n the local neghborhood of these ponts,no use s made of data at ntermedate locatons whch makes the approach prone to mage nose. The polygonal chan representaton [6] overcomes some of these problems. However, t s not well suted to represent curved objects well, requrng many control ponts to be adequate. In an nteractve object delneaton settng, ths s tedous. For trackng applcatons, the placement ofcontrol ponts close to each other, typcal also of pont representatons, leads to stablty problems. Ths s because for most contour ndng approaches usng local representatons, a-pror shape nformaton s encoded for each control pont w.r.t. to ts neghborng control ponts (.e. curvature [9][1] [16], ane coordnates [10]). If control ponts are close together, small devatons due to mage nose or contour propagaton wll result n large changes of local shape propertes. B-splnes present anecent and natural way to represent smoothly curved objects. For objects wth sharp corners they are less suted the C contnuty smooths out any regons of hgh curvature of a contour. The fact that B-splnes do not nterpolate the control ponts can be consdered a drawback n an nteractve object delneaton settng (thnk of a physcan pontng to specc locatons n medcal mages). The before mentoned use of control pont duplcaton can take care of ths, but then straght lne segments appear around the newly C 0 contnuous control pont. Wthout user nterventon, when to duplcate control ponts becomes a dcult decson for example, Menet [11] duplcates control ponts n regons where after M steps of contour deformaton, the curvature s hgher than a user-suppled threshold. 3 The Hermte Representaton The prevous consderatons lead us to propose the Hermte representaton for deformable contour ndng. Hermte contours are pecewse cubc polynomals, whch nterpolate the control ponts p 0 ::: p N. In each nterval, the Hermte cubc Q(s t) = [x(s t) y(s t)] s speced by the postons p ;1, p and tangent vectors ;1 +, ; at the endponts. Let Q be an arbtrary cubc polynomal where T =[t 3 t t 1 1] C = wth tangent vector Q 0 (t) Q = T C (1) 6 4 a x b x c x d x a y b y c y d y 3 7 5 Q 0 = T 0 C =[3t t 10] C () Gven hermte parameter matrx H =[h x h y ]=[p ;1 p + ;1 ; ]T (3) the correspondng Hermte coecent matrx C H can be derved as [7] C H = 6 4 ; 1 1 ;3 3 ; ;1 0 0 1 0 1 0 0 0 3 7 5 H
We collect all the hermte parameters n state vector H for later use H = 6 4 ; 0 p 0 + 0 ::: ; N p N + N 3 7 5 (4) When consderng the same crtera of usefulness for the contour ndng problem as dscussed n prevous secton for the pont-, polygon- and splne-based representatons, we note that the Hermte representaton can ecently represent both smooth and sharp contours. Ths s because smooth contours are well represented by the Hermte nterpolatng cubcs, whle at the same tme, arbtrary sharp corners can be easly generated at the control ponts by the adjustment of the left and rght tangent vector parameters nterpolates the control ponts s explct n those features that can be measured from mage data: poston and drecton of gradent at control ponts. Ths allows to prune the search space durng contour ndng, as we wll see n next secton. 4 Contour Detecton 4.1 MAP formulaton A maxmum a posteror (MAP) crteron s formulated for the soluton of the contour ndng problem. The am s to nd from all possble contours the contour whch matches the mage data best, n a probablstc sense. Let d(x y) be the orgnal normalzed mage and t H (x y) be the mage template correspondng to the Hermte parameters H. We want to nd H MAP whch maxmzes the probablty that t H occurs gven d, e.g. P (t HMAP jd). t HMAP s then the maxmum a posteror soluton to the problem. Bayes rule gves P (t HMAP jd) = max H P (t H jd) P (djt = max H ) P (t H ) (5) H P (d) where P (djt H ) s the condtonal probablty ofthe mage gven the template, and P (t H )andp(d) are the pror probabltes for template and mage, respectvely. Takng the natural logarthm on both sdes of eq.(5) and dscountng P (d), whch doesnot depend on H, leads to an equvalent problem of maxmzng objectve functon U U(t HMAP d) = max H U(t H d) = max H (lnp(t H )+lnp(djt H )) (6) The above equaton descrbes the trade-o between a-pror and mage-derved nformaton. If the mage s consdered as a nose corrupted template wth a addtve and ndependent nose that s zero-mean Gaussan, we have P (djt H )=P (djt H + n) =P (njd ; t H ), thus Y 1 P (djt H )= p e ; (d(x y);t H (x y)) n (7) n t H (x y) and X (d(x y) ; t ln P(djt H(x y)) H )= constant+ t H (x y) (8) Ths last term can be replaced by correlaton term d t H,approxmatng jjdjj and jjt H jj by constants. For jjdjj = 1 and jjt H jj =1we obtan max H ln P(djt H ) mn H (1;dt H ) = mn H Eext (9) A smlar dervaton was descrbed by Rosenfeld and Kak [13] and Stab and Duncan [14]. We model the pror probablty for a Hermte contour H as P (H) =P (HjH) = constant Y ; (H[];H[]) e (10) where H represents an expected contour. H s typcally obtaned as the sample mean of contours generated n a tranng phase, or as the contour obtaned by predcton durng trackng. acts as a weghng measure for the varous dmensons. In case of an open contour, we set the 's of ; 0 and N + equal to nnty. In trackng applcatons the contour typcally undergoes a tranformaton T (for example, translaton, rotaton and scale) for whch one does not wanttopenalze. The above modelng assumes that any transformaton on the contour whch one does not want to penalze has already been performed, before eq.(10) s appled. Any further contour change s consdered as deformaton from an expected contour and thus penalzed. Takng the natural logarthm gves max H lnp(t H )=mn H X (H[] ; H[]) 4. Dynamc Programmng =mn H Ent (11) There are many ways to solve the resultng mnmzaton problem mn H E = mn (Ent + Eext): (1) H Varatonal calculus methods have been used extensvely for contnuous parameter spaces where dervatve nformaton s avalable [5] [9] [11] [1] [14] [15].
For dscrete search spaces one possblty stouse A.I. search technques (e.g. best-rst, smulated annealng, genetc algorthms). We wll use a dscrete enumeraton technque based on dynamc programmng (DP) whch was popularzed by Amnet al. [1], and used snce by [8] [10]. The advantages of dynamc programmng w.r.t. varatonal calculus methods are n terms of stablty, optmalty and the possblty to enforce hard constrants [1]. For dynamc programmng to be ecent compared to the exhaustve enumeraton of the possble solutons, the decson process should be Markovan. Ths s typcally the case f the a pror-shape component E nt contans a summaton of terms whch only depend on parameters whch can be derved locally along the contour. For the case of open contours, our objectve functon can be wrtten as E = E 1(p 0 + 0 ; 1 p 1 ) + ::: + E N (p N;1 + N;1 ; N p N ) (13) Applyng the dynamc programmng technque to our formulaton nvolves generatng a sequence of functons of two varables, s wth = 0::N ; 1, where for each s a mnmzaton s performed s over two dmensons. s are the optmal value functons dened by s 0( 1 ; p 1 ) = p mn E 1(p 0 0 + ; 1 p 1 ) 0 0 + s ( +1 ; p +1 ) = p mn (s ;1(p + )+ + E (p + ; +1 p +1 )) = 1 :: N ; 1 (14) If p and ; ( + ) range over NP and NT values at each ndex, the complexty of the proposed algorthm s O(NNP NT ). The above formulaton s for open contours. For closed contours, where the rst and last control pont are dened equal, we apply the same algorthm as for the open contour case, yet repeat t for all N P possble locatons of the rst (last) control pont, whle keepng track of the best soluton. The complexty ncreases to O(NNP 3 NT ). Speed-up can be acheved by a mult-scale approach. Here contour ndng s rst done on a lower resoluton mage to nd an approxmated contour. Ths can be done wth a coarse dscretzaton of the parameter space (.e. requrng smaller N P and N T for the same parameter range). At the ner level, the orgnally desred dscretzaton can be acheved by decreasng the parameter range to le around the soluton found at the coarse level. At the same scale, the algorthm can be sped up by dscountng unprobable control pont locatons before startng the DP search. A measure of \unprobablty" can be speced n terms of weak mage gradent strength or dot product between measured and expected gradent drectons (the latter are explct n the Hermte representaton). If all the canddate control pont locatons are rated smlarly (e.g. standard devaton of ratngs below a threshold), t s more robust to consder all. In addton, for closed contours, one can use only a sngle pass DP for closed contour and to optmze for the remanng E 0(p N N + ; 0 p 0 ) whle assgnng to p 0 and p N the optmal values found for the open contour case. Of coarse, all these speed-up procedures loose the optmalty propertyofdp.neverthe- less, the last two methods whch were mplemented performed satsfactory n practce. 4.3 Contour Trackng The hgh computatonal cost of dynamc programmng, and of other search methods whch do not get stuck n the closest local mnmum, makes search only feasble n a lmted neghborhood. For nteractve contour delneaton ths s ne, snce the user s lkely to place well-postoned control ponts, very close to the desred contour. In trackng applcatons ths requrement s often unrealstc. On the other hand, t s our observaton that the eects of deformaton are often small from frame to frame once rgd moton s accounted for. We therefore decouple the eects of moton and deformaton on the contour, searchng rst for transformaton parameters T =[t s]wtht, and s denotng translaton, rotaton and scalng. T s found w.r.t. the undeformed contour, after whch search contnues for the deformaton parameters. The rst stage s robustly performed by template matchng (or Hough Transform [10]) on a Gaussan-blurred gradent mage. The second stage s the DP approach descrbed earler. Both stages use moton predcton methods template matchng at tme t +1 searches n a parameter range centered around predcted transformaton T (t + 1) usng predcted template H(t + 1). H(t + 1) s also the ntal contour of DP search. For smplcty, wecurrently use T (t +1) = T (t) and H(t + 1) = H(t). More general, T (t + 1) = p(t+1) where p s a best ttng n-th order polynomal at (t-m, T(t-M))..., (t-1, t). Smlar consderaton holds for H(t + 1). If the tme-span M n whch a n-th order model holds s large t s ecent tousea recursve predctor such as the Kalman lter. 5 Experments We have performed experments wth the proposed combnaton of Hermte representaton and templateplus-dp search n both nteractve astrackng settngs. The assocated template matchng parameters were range and dscretzaton of the transformaton parameters (translaton, rotaton and scale). DPrelated parameters ncluded the ntal values of the Hermte parameters, ther range and dscretzaton, as well as the weghng parameters. The locatons consdered around control pont p led on a rectangular grd wth x-axs perpendcular to p +1 ;p ;1.
The Hermte gradents were descrbed n terms of length l and drecton.typcally, N =5, N P =9 (N P = 4 after prunng), N l =3,N =9. Fgure 1 demonstrates versatlty of the Hermte representaton. Derent ntal contours are placed by the user as shown n Fgure 1a. Fgure 1b shows the search regon covered by DP for the ntal control pont placement for each contour segment the Hermte cubcs are shown correspondng to ( max, +1max ) and ( mn, +1mn ) for xed (ntal) control pont locatons and l = l max. Many derent Hermte contours whch le wthn ths search regon are not dsplayed. Fgure 1c shows the result of contour ndng by DP. One observes awdeva- rety of shapes that have been accurately descrbed by the Hermte representatons, from the smoothly varyng contour of the mug rm to the sharp corners of the square pattern, wth a curved horzontal segment jonng at the corner. It compares favorably wth a possble representaton by polygonal chans, splnes or Fourer descrptors. For completeness, we also show n Fgure 1d the condtoned Sobel gradent mage, whch s used by the DP algorthm. We use a condtoned mage nstead of the orgnal Sobel mage n order to amplfy weak but probable edges. Ths s done based on local consderatons, takng nto account mean and standard devaton n a n n neghborhood. A lnear remappng s appled on the mage data at (x y) f s greater than a user speced threshold. Fgure shows derent nstances of ntal placement and contour detectons on a MR mage of the human bran. Fnally, Fgure 3 shows a trackng sequence of a head usng the proposed combnaton of coarse-scale template matchng and DP. 6 Concluson We have advocated the use of the Hermte representaton for deformable contour ndng. It was shown to have advantages over pont-, polygonal- and splne-based representatons n terms of versatlty, stablty and controlablty. A decoupled approach to contour trackng was proposed based on template matchng on a coarse scale to account for moton effects, and a DP formulaton on a ner scale to account for the deformaton eects. 7 Acknowledgements The author thanks Larry Davs for hs contnued support. [] A. Barr, \Global and local deformatons of sold prmtves," Comput. Graphcs, vol.18, pp.1-30, 1984. [3] D.H. Ballard and C.M. Brown, Computer Vson, Prentce-Hall, Eaglewood Cls, 198. [4] A. Blake, R. Curwen and A. Zsserman, \A Framework for Spatotemporal Control n the Trackng of Vsual Contours," Int. J. Computer Vson, vol.11, nr., pp.17-145, 1993. [5] A. Chakraborty, M. Worrng and J.S. Duncan, \On Mult-Feature Integraton for Deformable Boundary Fndng," Proc. ICCV, pp.846-851, 1995. [6] P. Delagnes, J. Benos and D. Barba, \Actve contours approach to object trackng n mages sequences wth complex background," Pattern Recognton Letters, vol.16, pp.171-178, 1995. [7] Foley et al., Computer Graphcs, Addson- Wesley, 1996. [8] D. Geger et al., \Dynamc Programmng for Detectng, Trackng, and Matchng Deformable Contours," IEEE Trans. on Pattern Analyss and Machne Intellgence, vol.17, no.3, 1995. [9] M. Kass, A. Wtkn and D. Terzopoulos, \Snakes: Actve Contour Models," Int. J. Comp. Vs., pp.31-331, 1988. [10] K.F. La and R.T. Chn, \Deformable Contours: Modelng and Extracton," Proc. IEEE Computer Vson and Pattern Recognton, pp.601-608, 1994. [11] S. Menet, P. Sant-Marc and G. Medon, \Bsnakes: mplementaton and applcaton to stereo," Proc. Image Understandng Workshop, pp.70-76, 1990. [1] P. Radeva, J. Serrat and E. Mart, \A Snake for Model-Based Segmentaton," Proc. ICCV, pp.816-81, 1995. [13] A. Rosenfeld and A. Kak: Dgtal Pcture Processng, Academc Press, 198. [14] L.H. Stab and J.S. Duncan, \Boundary Fndng wth Parametrcally Deformable Models," IEEE Trans. on Pattern Analyss and Machne Intellgence, vol.14, no.11, pp.1061-1075, 199. [15] D. Terzopoulos and D. Metaxas, \Dynamc 3D Models wth Local and Global Deformatons: Deformable Superquadrcs," IEEE Trans. on Pattern Analyss and Machne Intellgence, vol.13, no.7, pp.703-714, 1991. [16] N. Ueda and K. Mase, \Trackng Movng Contours Usng Energy-Mnmzng Elastc Contour Models," Proc. ECCV, 199. References [1] A.A. Amn, T.E. Weymouth and R. Jan, \Usng Dynamc Programmng for solvng varatonal problems n vson," IEEE Trans. on Pattern Analyss and Machne Intellgence, vol.1, no.9, pp.855-867, 1990.
(a) (b) (c) (d) Fgure 1: Mug mage (a) contour ntalzaton (b) search regon (c) contour detecton (d) condtoned Sobel mage (a) (b) Fgure : Bran MR mage (a) contour ntalzaton (b) contour detecton Fgure 3: A trackng sequence of a head (t = 0, 8, 4, 30)