Moel Driven Ege Detection P. Fua an Y. G. Leclerc SI International 333 avenswoo Avenue, Menlo Park, CA 9425 (fua@ai.sri.com leclerc@ai.sri.com) Machine Vision an Applications, 3, 199 Abstract Stanar ege etectors fail to n most relevant eges, ning either too many or too few, because they lack a geometric moel to guie their search. We present a technique that integrates both photometric an geometric moels with an initial estimate of the bounary. The strength of this approach lies in the ability of the geometric moel to overcome various photometric anomalies, thereby ning bounaries that coul not otherwise be foun. Furthermore, eges can be score base on their gooness of t to the moel, thus allowing one to use semantic moel information to accept or reject the eges. 1 Introuction In real-worl images, object bounaries cannot be etecte solely on the basis of their photometry because of the presence of noise an various photometric anomalies. Thus, all metho for ning bounaries base on purely local statistical criteria are boun to err, ning either too many or too few eges base on arbitrary threshol. To supplement the weak an noisy local information, we consier the geometric constraints that object moels can provie. We o this by escribing a bounary as an elastic curve with a eformation energy erive from the geometric constraints, as suggeste previously (Terzopoulos 1987; Kass et al. 1988). Local minima in this energy correspon to bounaries that match the object moel exactly. We incorporate the photometric constraints by ening photometric energy as the average along the curve of a scalar energy el erive from the photometric moel of an ege. Local minima in this energy correspon to bounaries that match the photometric moel exactly. We n a caniate bounary by eforming the curve insuchaway as to minimize its total energy, which is the sum of the eformation an photometric energies. We call this eformation process \optimizing" the curve. Once a curve has been optimize, i.e., once it has settle in a local minimum of the total energy, thus eecting a compromise between the two constraints. We can then utilize the object moels to etermine if the curve actually correspon to an object bounary. Such \energy minimizing curves" have twokey avantages: The geometric constraints are irectly use to guie the search for a bounary. The work reporte here was partially supporte by the Defense Avance esearch Projects Agency uner contracts DACA76-85-C-4 an MDA93-86-C-84. We wish to thank Demetri Terzopoulos for the many iscussions we have ha concerning this work. 1
The ege information is integrate along the entire length of the curve, proviing a large support without incluing the irrelevant information of points not on the curve. Taken together, they allow energy minimizing curves to n photometrically weak bounaries that local ege etectors simply coul not n without also ning many irrelevant bounaries. Energy minimizing curves, as we call them, were originate by Terzopoulos, Kass, an Witkin as \snakes" (Terzopoulos 1987; Kass et al. 1988). This paper was motivate by the nee to solve two major problems with snakes. First, the precise relationship between the local minima foun by these snakes an the stanar enitions of an ege ha not been eluciate. Secon, snakes were basically esigne for use in interactive environments. Consequently, they require fairly extensive parameter ajustments from one image to the next, an even from one part of an image to the next. In this paper, we oer three major contributions. (1) We show the precise relationship between the local minima of the energy minimizing curve an the stanar enitions of an ege. (2) We present a proceure for automatically ajusting the parameters of these curves. (3) We briey show how the relationship an the proceure can be combine in an automate system that extracts roa an builings from aerial images, escribe extensively in (Fua an Hanson 1988). 2 A Photometric Moel: Step Eges Several authors (Haralick 1984; Canny 1986; Torre an Poggio 1986) ene step-ege points as those for which the rst irectional erivative of image intensity, in the irection of the graient, is extremal. That is, an ege point (x ;y ) satises where @ 2 I(x; y) @g 2 @ @g = g r; (x;y)=(x ;y ) g = g(x ;y )= ri(x ;y ) jri(x ;y )j ; =; (1) an I(x; y) represents the image intensities after convolution with a twice ierentiable kernel (typically a Gaussian). 1 It can be shown that Equation (1) is equivalent to @ jri(x; y)j =: @g (x;y)=(x ;y ) In other wor, the Haralick et al. criterion for an ieal step-ege point is equivalent to the criterion that the magnitue of the image intensity graient be maximal in the graient irection. 1 This enition is a goo approximation to the position of iscontinuities in the unerlying intensity function when the iameter of support of the operator is much smaller than the raius of curvature of the ege. In particular, it is not applicable at corners an junctions of eges. 2
However, for noisy non-ieal step eges the graient irection may become very unreliable, especially at some istance from the ege itself. We therefore use the following, somewhat more general enition of an ege: Denition An ege is a curve C whose points have a graient magnitue that is maximal in the irection normal to the curve. That is, all points along the curve (calle ege points) satisfy @ jri(f(s))j @n(f(s)) =; (2) where s is the arc-length of C, f(s) isavector function mapping the arc-length s to points (x; y) in the image, an n(f(s)) is the normal to C. Note that this enition is equivalent to the Haralick et al. enition for ieal noise-free step eges. To incorporate this photometric moel in the energy minimizing curve approach, we ene the photometric energy of the curve C as E P (C) =, 1 jcj jcj jri(f(s))j where jcj is the length of C. For simplicity of notation, the limits on the above integral will not be repeate in the remainer of the text. To unerstan the relationship between optimize curves an eges, consier the following. When C is either an open or close curve that has been optimize, i.e., when it is a local minimum of E P with respect to innitesimal eformations of the curve, we prove in the Appenix that: @jri(f(s))j @n(f(s)) = (s) 1 jri(f(s))j, jcj jri(f(s))j ; (3) where (s) is the curvature of C. Furthermore, we prove that when C is an open curve that has been optimize, jri(f())j = jri(f(jcj)j = 1 jcj jri(f(s))j (4) also hol. These two equations have the following consequences. Equation (3) implies that the points along an optimize curve are all ege points when either its curvature is zero (i.e., it is a straight line segment), or the magnitue of the graient along the curve is constant. Consequently, optimize curves whose curvature is small or for which the magnitue of the graient is approximately constant are goo approximations to eges. In particular, we have foun that replacing jri(x; y)j by log jri(x; y)j (which has no eect for ieal step eges) yiel more stable solutions because the secon term in Equation (3) is smaller for non-constant ege strengths, an hence the approximation is generally better. For an open-ene curve, Equation (4) must also be satise. This equation implies that the curve is stable only when the graient magnitue at the en points equals the average. In other wor, an open ene energy minimizing curve place exactly along an ege will shrink or expan, following the ege, until this conition is met. When the graient magnitue is constant, of course, this conition is always met, an curves therefore remain unchange. Thus, again, using log jri(x; y)j yiel more stable solutions because this equation is more nearly satise for non-constant ege strengths. An important note is 3
that if we ha chosen to use the integral of graient magnitue, as oppose to the average, this equation woul not hol: curves woul simply expan without boun. After optimization, caniate eges can be score on how well they match the photometric moel by computing the proportion of points along the curve that satisfy Equation (2) to within some tolerance. We use this score in our applications to accept or reject any given ege. 3 A Geometric Moel: Smooth Curves 3.1 Theory We have seen in the previous section that energy minimizing curves match eges well wherever the curves have alow curvature. To ensure stable results we ene the eformation energy of such curves so that their curvature remains small an the minimum energy state is close to the photometric ege. This can be achieve simply by ening the eformation energy of a curve C as: E D (C)= (s) 2 where (s) is the curvature of C. The total energy of such a curve is then E(C) =E P (C)+ E D (C), where E P (C) is the photometric energy ene in the previous section. To elineate a photometric ege using one of these curves, one must provie an initial estimate of the location of the curve an then optimize it using the proceure escribe in the next subsection. Previously (Kass et al. 1988), was chosen interactively. Wehave foun that a single value of, however carefully chosen, is inaequate for use across a wie variety of images. Instea, we compute in the following aaptive manner. Let C i be the initial curve an C f be the curve after optimization. Let i enote the operator i = ; C=Ci C that is, i is the variational operator evaluate at the initial curve. Dene f similarly for the nal curve. Since, by enition, f E(C) =, then = j f E P (C)j j f E D (C)j : When the initial estimate is known to be close to the nal answer, we choose = j i E P (C)j j i E D (C)j so that we remain as close to the initial estimate as possible. If instea we want to smooth the initial estimate, we can use higher values of. This is in fact the simplest way of imposing a geometric moel, in this case smoothness, upon the ata. 3.2 Implementation In the actual implementation, the curves are escribe as polygons with n equiistant vertices X = f(x i y i );i=1;:::;ng an the eformation energy can be mae iscrete as E D = X i (2x i, x i,1, x i+1 ) 2 +(2y i,y i,1,y i+1 ) 2 (5) 4
when (s) is small. To perform the optimization we coul use a simple graient escent technique (Leclerc an Fua 1987), but it woul be extremely slow for curves with a large number of vertices. Instea, weembe the curve in a viscous meium an solve the equation of ynamics @E + @X X t =; where E = E P + E D an is the viscosity of the meium (Terzopoulos 1987). Since the eformation energy E D in Equation 5 is quaratic, its erivative with respect to X is linear an therefore @E D @X = KX; where K is a pentaiagonal matrix. Thus, each iteration of the optimization amounts to solving the linear equation: KX t + (X t, X t,1 )= @E P @X Xt,1 : (6) Figure 1 Because K is pentaiagonal, the solution to this set of equations can be compute eciently in O(n) time using LU ecomposition an backsubstitution. Note that the LU ecomposition nee be recompute only when changes. When is constant, only the backsubstitution step is require. For a curve with 1 vertices, the LU ecomposition step takes.5 secon on a 3675 Symbolics Lisp Machine with a Floating Point Accelerator, an the backsubstitution step takes only.12 secon. Previously (Terzopoulos 1987), was chosen interactively, an remaine xe throughout the optimization process. Since this has proven to be inaequate for use across a variety of images, we use the following aaptive proceure for computing. We start with an initial step size an compute the viscosity so that p 2n = @E @X ; where n is the number ofvertices. This ensures that the initial isplacement of each vertex is on the average of magnitue. Because of the non-linear term, we must verify that the energy has ecrease from one iteration to the next. If, instea, the energy has increase, the curve is reset to its previous position, the step size is ecrease, an the viscosity recompute accoringly. This is repeate until the step size becomes less than some threshol value. In most cases, because of the presence of the linear term that propagates constraints along the whole curve in one iteration, it only takes a small number of iterations to optimize the initial curve. For example, going from the initial estimate of the close curve with about 2 vertices shown in Figure 1a to the optimize result shown in Figure 1b took 2 iterations an 3.9 secon on 3675 Symbolics Lisp Machine with a Floating Point Accelerator. Only 18 iterations an 2.8 secon were require for the open curve. Note that the time actually spent solving the equations of the ynamics represents only a relatively small fraction of the total time require to perform the complete optimization; most of the time is actually spent upating the snakes an fetching the image values. 3.3 Evaluation Criterion Figure 2 To illustrate the utility of our ege-point criterion (Equation 2), consier the synthetic image shown in Figure 2, wherein one bounary of the central region is smooth while the 5
Figure 3 other is jagge. Two energy minimizing curves were entere interactively (Figure 3b) an optimize using the smoothness constraint above (Figure 3c). emoving all points along the optimize curves that o not satisfy Equation 2 yiel Figure 3. While most of the pixels belonging to the smooth bounary satisfy the ege criterion, very few of those belonging to the jagge bounary o. Also note that a small part of the topmost curve faile to settle correctly along the ege because of noise. However, this failure is easily ientie because it is the only part of the curve for which a signicant fraction of the points fails to satisfy the criterion. Thus, computing the proportion of ege points in the optimize eges is an ecient an reliable way to measure how well the photometric bounaries match the smoothness constraint. 4 Applications In this section we present two applications of our energy-minimizing curves. First we escribe an application of our moel to roa elineation in an interactive context, where the user provies a rough initial estimate of the location of the contours. This interactive proceure has been integrate in the SI Cartographic Moeling Environment (Hanson an Quam 1988). It allows a cartographer to roughly sketch features of interest, an then interactively ajust the resultant optimize contours. We then outline an application of our moel to roa an builing elineation in the context of the fully automate system escribe in etails in (Fua an Hanson 1988). 4.1 Interactive oa Delineation Figure 4 Figure 5 We can apply the smooth curve moel of the previous section irectly to the problem of ning roa bounaries in aerial images. We moel roa as ribbons whose smoothly curve eges are approximately parallel. A ribbon is implemente as a polygonal curve forming the center of the roa. Associate with each vertex i of this curve is a with w i that enes the two curves that are the caniate roa bounaries. The photometric energy is ene as the sum of the photometric energies of the two bounary curves, an the eformation energy is the sum of the smoothness term escribe above an an aitional term that enforces the parallel constraint: X (w i, w i,1 ) 2 : i We have applie this moel to two roa segments in the aerial image of Figure 4a. Figure 4b shows the initial estimates for two roa segment bounaries an Figure 4c shows the nal optimize bounaries. This image is an excellent example of a situation where a local ege etector such as the Canny ege etector is insucient, by itself, to n the relevant bounaries. Because the roa are irt roa with ill-ene eges, no single ege-strength threshol yiel a satisfactory set of bounaries: when the threshol is too high, most of the roa eges are lost, whereas when the threshol is too low, there is a plethora of irrelevant eges (see Figure 5). In the next section, we briey escribe an automate system that exploits the power of energy minimizing curves to overcome this problem. 6
4.2 Automate oa an Builing Delineation In our automate system (Fua an Hanson 1988), roa are moele as above, while builings are moele as polygons whose eges meet at right angles. We begin by using the synthetic image of Figure 2 to emonstrate how the system uses energy minimizing curves to extract smooth an straight eges from a set of Canny ege maps. Since a full escription of the system is beyon the scope of this paper, we can only briey outline the subsequent steps that lea to the generation of roa an builing outlines, an show the results of this system applie to an aerial image. Figure 6 Figure 7 Figure 8 Figure 9 Figure 1 Smooth Eges: The system begins by applying the Canny ege etector to the image using several threshol (Figure 6). The pixels in each of these maps are then linke using a linker (Fischler an Wolf 1983). Each of these linke segments is use as the starting point for an energy minimizing curve. After optimization, the curves are broken into segments wherever sharp ben are foun. Those segments for which at least 7% of the points satisfy the ege criterion are accepte as caniates for the next step (Figure 7a,b,c). This 7% gure has been use for all examples in this paper, an inee for every other application of the automate system. Experimentally, curves with less than 5% of the points satisfying the ege criterion are almost always spurious, while curves with more than 9% are almost always correct. We chose 7% as a compromise that rejects most of the spurious eges without losing a signicant number of correct ones. Since the same ege may be iscovere, in whole or in part, in several of the Canny ege maps, the system must choose an appropriate subset of these curves. The basic goal of this selection process is to choose the longest possible curves that o not overlap an have goo ege quality. This goal is achieve by selecting the largest subset of non-overlapping curves that maximize the number of pixels satisfying the ege criterion minus the number of pixels that o not. The result of this selection process is shown in Figure 7. This technique provies an eective way of selecting the smooth bounaries in the image. Furthermore by combining the output of the Canny ege etector using several arbitrary threshol, we have eliminate the necessity of carefully choosing an appropriate threshol. This is a substantial avantage since, as iscusse previously, choosing this threshol is in general a very icult task. Straight Eges: The system begins with the same linke Canny ege images as before, n points of high curvature in the linke segments, an ts straight lines between those points. The straight lines are use as the starting points for energy minimizing curves with only two vertices. As previously, the curves are optimize an only those with 7% of points satisfying the ege criterion are retaine (Figure 8a,b,c). A subset of these curves is selecte using the same proceure as above (Figure 8). Note that, in this particular case, all of the nal curves coul have been extracte from a single Canny ege map. This rarely occurs in practice because the noise across a real image is never as uniform as it is in this synthetic image. Automatic oa Delineation: Applying the smooth ege technique to the aerial image of Figure 9 yiel the eges shown in Figure 1a. These eges are then groupe into sets of parallel eges, each of which is use as the starting point for a ribbon, as escribe in Section 4.1. After optimization, the system uses a selection criterion to retain only the best 7
roa caniates (Figure 1c). The groups from which these caniates arose are shown in Figure 1b. Figure 11 Automatic Builing Delineation: Applying the straight ege technique to the same aerial image yiel the eges shown in Figure 11a. These eges are then groupe into rectilinear ege structures, an close using a simple ege tracker. Finally, the outline of these rectilinear contours is optimize by using an energy minimizing curve an a rectilinearity constraint. Once again, the system uses a selection criterion to retain only the best builing caniates (Figure 11c). The groups from which these caniates arose are shown in Figure 11b. 5 Summary an Conclusion We have presente a technique for ning object bounaries that integrates both photometric an geometric moels with an initial estimate of the bounary. The moels are incorporate by ening an energy function that is minimal when the moels are satise exactly. The initial estimate is use as the starting point for ning a local minimum of this energy function, by embeing the initial curve in a viscous meium an solving the equations of ynamics. The strengths of this \energy minimizing curve" approach are that the geometric constraints are irectly use to guie the search for a bounary, an that the ege information is integrate along the entire length of the curve, thereby proviing a large support. We have shown the precise relationship between optimize curves an a stanar enition of an ege, an that this relationship can be use to etermine when an optimize curve shoul be accepte as a caniate ege for further processing. We have also presente metho for automatically choosing the optimization parameters an how they shoul change over time. We foun that these contributions were crucial to our ability to embe energy minimizing curves in the automate system briey iscusse here an presente in etail elsewhere (Fua an Hanson 1988); these metho have provie us with the ability to n photometrically weak bounaries that local ege etectors simply coul not n without also ning many irrelevant bounaries. Appenix In this appenix, we show that a necessary an sucient conition for an open curve C to be a local extremum of E(C) = G(f(s)) (1) with respect to all innitesimal eformations is that = (s) G(f(s)), G(f(s)) n(s) G(f(s)) (2) an G(f()) = G(f(jCj)) = G(f(s)) : (3) where the curve C is parameterize by its arc-length s; f(s) is a vector function mapping the arc-length s to points (x; y) along the curve; n(s) is the normal to the curve; t(s) isthe 8
tangent to the curve; an (s) is its curvature. Furthermore, eq. (2) alone is the necessary an sucient conition for close curves. To prove this result, consier eformations of the curve C, which we shall call C, such that the mapping from arc-length s to points (x; y) is of the form f (s) =f(s)+((s)n(s)+(s)t(s)) ; (4) where (s) an (s) are arbitrary continuous an ierentiable function. When C is a close curve, (s) an (s) are constraine to be perioic of perio jcj. Since C is a local extremum of E(C) if an only if E(C ) = =; (5) for all (s) an (s), we now prove the following theorem. Theorem: Equation (5) hol for all (s) an (s) if an only if eq. (2) an (3) above are satise. Proof: By enition, E(C )= G(f (s )) ; (6) where s is the arc-length of C.We can rewrite this in terms of s by using the equality Thus, E(C )= an therefore, to within the non-zero factor of 1= ( jf (s)j )2 : E(C ) = =, G(f (s)), = f (s) : (7) G(f (s)) jf (s)j f (s) ; (8) G(f (s)) f (s) f (s) G(f (s)) f (s) f (s) f (s) + G(f (s)) jf (s)j G(f (s)) f (s) jf f (s) (s)j : (9) The rst term we nee to evaluate in eq. (9) is jf (s)j =, for which we nee the following equalities: Therefore, f(s) t(s) n(s) = f (s) = t(s) = t (s) = (s)n(s) = n (s) =,(s)t(s): f (s) = f (s)+ (s)n(s)+(s)n (s)+ (s)t(s)+(s)t (s) = t(s)+ (s)n(s),(s)(s)t(s)+ (s)t(s)+(s)(s)n(s) = t(s) 1+( (s),(s)(s) +n(s) ( (s)+(s)(s)) : (1) 9
Since t(s) an n(s) are, by enition, orthogonal unit vectors, we have an therefore f (s) = q[1+( (s),(s)(s))] 2 +[( (s)+(s)(s))] 2 ; (11) jf (s)j = [1 + ( (s), (s)(s))]( (s), (s)(s)) + ( (s)+(s)(s)) q[1 2 : (12) + ( (s), (s)(s))] 2 +[( (s)+(s)(s))] 2 The secon term we nee to evaluate is G(f (s)) = ((s)n(s)+(s)t(s)) rg(f (s)) = (s) G(f (s)) n(s) + (s) G(f (s)) : (13) t(s) Substituting (11), (12), an (13) into (9), evaluating at =, an multiplying by the non-zero factor 1= ( jf (s)j )2 : E(C ) = = =, (s) G(f(s)) n(s) G(f(s)) + (s) G(f(s)) t(s) Diviing by an rearranging terms, we have G(f(s)) = (s), (s) n(s) + (s) G(f(s)) t(s) ( (s), (s)(s)): + (s) G(f(s)), G(f(s)), + G(f(s))( (s), (s)(s)) G(f(s)) G(f(s)) : A necessary an sucient conition for the rst integral to be zero for all (s) is that the term multiplying (s) be zero, which is eq. (2). Integrating the secon integral by parts: therefore, (s) G(f(s)) + (s) t(s) G(f(s)), G(f(s)) jcj, (s) G(f(s)) =; (14) t(s) G(f(s)) (jcj) G(f(jCj)), G(f(s)), () G(f()), =: (15) A necessary an sucient conition for this equality to hol for all (s) is that both terms be zero, which is eq. (3). Thus, a necessary an sucient conition for an open curve C to be a local extremum of E(C) is that both equations (2) an (3) hol. Furthermore, for close curves, () (jcj) an G(f()) G(f(jCj)), hence eq. (15) is always satise. Therefore, a necessary an sucient conition for a close curve C to be a local extremum of E(C) is that eq. (2) hol. Q.E.D. 1
eferences [1] Canny J (1986) A computational approach to ege etection. IEEE Transactions on Pattern Analysis an Machine Vision, (8)(6):679{698 [2] Fischler M, Wolf H (1983) Linear elineation. Proceeings of the IEEE Conference on Computer Vision an Image Processing, Washington, D.C., 351{356. [3] Fua P, Hanson A (1988) Extracting Generic Shapes Using Moel-Driven Optimization. Proceeings of the 1988 DAPA Image Unerstaning Workshop, Boston, MA, 994{ 14 [4] Haralick M (1984) Digital step eges from zero crossings of secon irectional erivatives. IEEE Transactions on Pattern Analysis an Machine Vision, (6)(1):58{68 [5] Hanson AJ an Quam L(1988), \Overview of the SI Cartographic Moeling Environment," in Proceeings of the Image Unerstaning Workshop, Boston, MA, 576{582. [6] Kass M, Witkin A, Terzopoulos D (1988) Snakes: active contour moels. International Journal of Computer Vision, (1)(4):321{331 [7] Leclerc YG, Fua P (1987) Fining object bounaries using guie graient ascent. Topical Meeting on Machine Vision, Technical Digest Series, Optical Society of America, Washington, D.C., (12):168{161 [8] Terzopoulos D (1987) On matching eformable moels to images. Topical Meeting on Machine Vision, Technical Digest Series, Optical Society of America, Washington, D.C., (12):16{167 [9] Torre V an Poggio T (1986) On ege etection. IEEE Transactions on Pattern Analysis an Machine Vision, (8)(2):147{163 11