A G E NG BUDAPES 00 EurAgEng Paper Number: 0-AE-004 tle: Weed Classfcaton by Actve Shape Models Authors: Søgaard, H..(*); Hesel,. Dansh Insttute of Agrcultural Scences, Department of Agrcultural Engneerng, Research Centre Bygholm, P.O. Box 536, DK-8700 Horsens, Denmar. el.: +45 769 608. Fax: 769 600. Emal: hennngt.sogaard@agrsc.d. Dansh Insttute of Agrcultural Scences, Department of Crop Protecton, Research Centre Flaebjerg, DK-400 Slagelse, Denmar. Emal: torben.hesel@agrsc.d. Summary: he objectve s to present a new method for classfcaton of weed speces by mage processng based on actve shape models. Young weed seedlngs wth up to four leaves and wthout mutual overlappng wth other leaves are to be dentfed. A database contanng mage examples of more than 0 of the most mportant weed speces n Dansh agrcultural felds has been establshed. he mages have been used as tranng data for the constructon of an actve shape model for each speces. On the bass of these models, an algorthm for dentfcaton of weed speces n dgtal mages has been developed. Prelmnary results have shown that the performance rate (rate of correctly dentfed weed seedlngs) of the algorthm s about 80% or more.
. Introducton It has been wdely accepted that the use of herbcdes wthn agrculture must be reduced n order to protect the envronment and the resources of drnng water. One way to obtan such a reducton s to perform precson applcaton of herbcdes n the feld, whch means usng the rght mxtures and dose rates of herbcdes at the rght tme and place, accordng to the dstrbuton and growth stages of occurrng weed speces. he sprayng technology and decson support systems for precson applcaton of herbcdes exst, and potentals for herbcde savngs between 30 and 75% have been demonstrated (Hesel et al., 999). However, precson sprayng presupposes that the dstrbuton of dfferent weed speces has been mapped by countng of plants; a job whch, untl now, has manly been done by way of tme-consumng manual surveyng. hough, some methods and systems for automatc dentfcaton and mappng of weed speces by machne vson have been proposed (e.g. Manh et al., 00; Söefeld et al., 000).. Objectves he objectve s to present a new method for classfcaton of weed speces by mage processng based on actve shape models. Young weed seedlngs wth up to four leaves and wthout mutual overlappng wth other leaves are consdered. 3. Method of approach he project s a part of the large Dansh project, Autonomous Platform and Informaton system for regstraton of crops and weeds (API, see http://www.cs.auc.d/~ap/). he overall objectve of the project s to develop and construct a small autonomous robotc vehcle for collecton of nformaton about weeds and crops n the feld. he frst job for the robot wll be to acqure and process mage nformaton that can provde the bass for relable weed maps. he mages wll be acqured n a grd pattern (e.g. a 0 0 m grd), and by the use of geostatstcal methods, weed maps for the weed speces of nterest wll be produced. he weed speces were recognsed by ther shapes by use of actve shape models (ASM) (Cootes et al., 994). he frst step of ths approach was to develop an ASM for each weed speces from a set of representatve tranng mages. he next step was to develop software that could classfy new unnown weed seedlngs by comparng them wth the weed models. 3.. Image database he tranng mages for shape modellng were acqured n the feld. Colour mage scenes representng more than 0 of the most mportant weed speces n Dansh agrcultural felds were collected n an mage database (more than 0 examples of each speces at early growth stages). Each mage scene covers approx. 50 00 mm wth a resoluton of about 0 pxels per mm. he mages were taen vertcally from above wth a Canon Powershot G Dgtal Camera (048 536 pxels). o ensure dffuse llumnaton and a fxed camera heght above the ground, the camera was mounted on top of a transparent plastc cylnder ( 50 mm) covered wth cloth. From each mage scene, one or more sub-mages contanng one weed seedlng each were "cut out". As the ASM method requres grey-scale mages (not colour mages), the sub-mages were subsequently converted to grey-scale mages by the operaton green-red-blue for each pxel (plus some approprate scalng). hs operaton has
proven to greatly enhance the green vegetaton n contrast to the bacground (Woebbece et al., 995). he grey-scale sub-mages and the assocated speces names were stored n a database and used as tranng mages for the model buldng process. 3.. Modellng shapes of weed seedlngs he ASM technques were orgnally appled to mage problems n the medcal doman, e.g. for modellng and measurng vertebrae n the spne for ostepeross dagnoss. However, ASM represents a general modellng technque that s applcable to all nds of shapes exhbtng a certan degree of stochastc varaton (Cootes et al., 994). An ASM conssts of a flexble shape template, descrbng how the object shapes can vary. In ths project, the modellng and dentfcaton of weed seedlngs was done by means of Matlab 6 (Release ; MathWors, 000) together wth the Actve Shape Model oolt (Vsual Automaton Ltd., 998). he ASM oolt provdes a lbrary of basc Matlab functons for modellng and recognton of structures and shapes n dgtal mages. o buld an ASM, the shape of each of the tranng objects must be represented by a set of ponts. For the weed seedlngs, the ponts were placed on the boundares of the leaves (an open contour for each leaf). In each tranng mage, 3 ponts were placed on the leaf boundares (Fg. ). he ponts were labelled wth numbers from 0 to 3. he placement and labellng of ponts were mportant, as each pont was to represent the same part of the seedlng from one tranng example to another. For example, one of the cotyledons should always be represented by ponts numbered 0-3, wth three of the ponts placed at ey postons: ponts zero and 3 placed at the begnnng of the leaf near the stem, and pont 6 at the tp of the leaf. Several speces had some characterstc ncsons at the true leaves, whch could also be used as dentfable ey postons. he ponts between the ey postons were spaced evenly along the leaf boundares. 8 6 66 98 0 65 3 33 99 3 49 5 Fgure. Placement of labelled ponts on the leaf boundares of some examples of whte goosefoot (Chenopodum album). 3
he co-ordnates of the ponts were stored and analysed statstcally, speces by speces, to extract characterstc shape varatons of seedlngs wthn the same speces. Before the real analyss of shape varatons could tae place, t was necessary to algn the set of tranng shapes. he algnment was acheved by scalng, rotatng and translatng the tranng shapes so that they corresponded as closely as possble wth each other. he algnment was performed n a way that mnmsed the sum of squared dstances between equvalent ponts on dfferent tranng shapes. he analyss of the varaton n shape across the N algned tranng shapes from a gven speces was based on a prncpal component analyss, as descrbed by Cootes et al., (994). Each of the algned tranng shapes gave rse to a vector descrbng the n boundary ponts: x x, y, x, y, K, x n, y n ), =,, N = (, 0,0,,,, K where (x,j, y,j ) s the j th pont of the th shape. he mean shape, x, s calculated as x = N N = x he modes of varatons,.e. the ways n whch the ponts of the shape tend to move together, can be found by applyng a prncpal component analyss to the devatons, x = x x ( =, K, N ), from the mean. From these devatons, the n by n covarance matrx can be calculated: N S = x x N = he modes of varaton of the pont of the shape can be descrbed by the n unt egenvectors, p,, p n, of S. he egenvectors are defned by Sp = λ p and p p = ( =, K,n) where λ,, λ n are the correspondng n egenvalues of S ( λ λ L ). λn he th prncpal component correspondng to the vector x s defned as a weghted sum of the elements of ths vector: b, = p ( x x), =, K, N, =, K, n he prncpal components represent a lnear ndependent decomposton of the varaton of the tranng shapes. he frst prncpal component, whch s assocated wth the largest egenvalue, λ, descrbes the largest part of the shape varaton (the frst mode of varaton). In fact, the proporton of the total shape varance descrbed by the th prncpal component s equal to the λ. Most of the varaton can usually be represented by a small number of prncpal components, say t (t < n). he value of t can, for nstance, be chosen n such a way that the frst t prncpal components explan a suffcently large proporton (e.g. 99%) of the total varance, λ = Σ λ, of the shapes. 4
Any shape n the algned tranng set can be approxmated as a sum of the mean shape and a weghted sum of the frst t egenvectors: ( t x x + Pb, =, K N (), where P = p p L p ) s a matrx of the frst t egenvectors, and b = s a vector of weghts (prncpal components), whch s calculated as b = P ( x x), =, K, N (, b,l b, t b ) Eqn. () permts generaton of a new shape example by replacng b by a new vector of weght values, b = ( b b L b t ). Provded that the weghts are not to far from zero, the new synthetc example wll be smlar to those n the tranng set, as the change n shape wll be determned by the modes of varaton represented by the tranng shapes. Sutable lmts for the weght vector, b, are derved by examnng the dstrbutons of the weght values to generate the tranng set. If Gaussan dstrbutons are assumed, the set of weghts can be chosen, so that the Mahalanobs dstance, D, from the mean shape s less than a sutable value, D max : m D m = b t h h= λh D max Fgure llustrates the three frst modes of varaton for whte goosefoot (Chenopodum album), as found by analysng examples le those gven n Fgure. he varaton around the mean model shape s shown by varyng b, b and b 3 one by one (D max = ) whle eepng the other b-weghts at zero. he frst and most domnatng mode corresponds to varyng growth stages and accounts for 59% of the total shape varance n the tranng set. he second and thrd modes account for 7 and 0% of the total shape varance, respectvely, and correspond to V- postons of the cotyledons and the two frst true leaves, respectvely. Frst mode of varaton: λ λ 0 + λ + λ b Second mode of varaton: λ λ 0 + λ + λ b hrd mode of varaton: λ 3 λ3 0 + λ3 + λ3 b 3 Fgure. Illustraton of the frst three modes of varaton for whte goosefoot (Chenopodum album). 5
3.3. Modellng grey-level appearance he actve shape model of a gven weed speces should be used for locatng new examples of ths speces. For ths purpose, not only the shape, but also the typcal grey-level appearance near the edges of the leaves are mportant. hs s accounted for by examnng the grey levels n the regon around each of the labelled model ponts. Snce a gven pont represents a partcular part of the seedlng, the grey-level pattern about that pont n mages of dfferent examples wll often be smlar. For each pont, a one-dmensonal grey-level profle normal to the model curve (contour) passng through ths pont s consdered. he profles are charactersed by ther mean and by ther varaton to gve a statstcal descrpton of the expected profles about each pont. he detaled calculaton procedure has been presented by Cootes et al., (994). 3.4. Seedlng recognton n mage scenes On the bass of results from the ASM analyss, an algorthm for locatng and dentfyng weed seedlngs n mage scenes was developed. In the frst step of ths algorthm, the mage scene s searched for green objects, whch are potental weed seedlngs. Only objects havng smlar sze as weed seedlngs are selected for further processng. In the second step, each of the selected objects s nvestgated to determne the weed speces. hs step nvolves an teratve search procedure n whch each speces model s refned gradually untl the best ft to the object boundares s obtaned. Before the teratve procedure s started, a prelmnary model algnment wth respect to pose (poston and orentaton) and scale taes place. After that, the shape refnng teratons are carred out by repeatng the followng three steps (see Fg. 3):. Compare the regon of the mage around each model pont wth the grey-level models to calculate the dsplacement of the pont requred to move t to a better locaton.. From these dsplacements calculate the optmal adjustments to the translaton, rotaton and scale and to the shape parameters (b-weghts as defned above). 3. Update the model parameters and the pont postons correspondngly (under the restrctons mposed on the shape by the t possble modes of varaton n the model). hese three steps are repeated untl the changes of the parameters between two successve teratons are suffcently small. he result from fttng four dfferent models to a whte goosefoot seedlng (Chenopodum album) appears from Fg. 4. (a) (b) Fgure 3. Illustraton of a sngle teraton of the object search procedure used for actve shape models (before (a) and after (b) movng the ponts). he green arrows ndcate the desred dsplacements of the model ponts. 6
Snce there are only t (< n) modes of varaton avalable n the model, whle the requred dsplacements of the ponts represents n degrees of freedom, the fnal model can only be an approxmaton to the object shape. When a model search has been carred out for all weed speces searched for, only one step remans, namely the classfcaton step. In ths step each object s classfed accordng to the speces model that gves the best ft to the object boundary. he assessment of the degree of ft for a gven speces model s based on a combned crteron that taes two aspects (sub-crtera) nto account: a. How well the deformed model shape resembles the object shape, and b. How much the model needs to be deformed to acheve the best ft. A model whch fts the object very well and only requres a moderate deformaton of the mean model shape wll result n a hgh ft score. he resemblance between the model and the object (sub-crteron a.) can be measured as the rato, α (0 α ), of pxels covered both by the model and the object (ntersecton of sets) to the number of pxels covered by the model and/or the object (unon of sets). he model coverage s defned as pxels wthn the four polygons made up by the leaf contours. he rato wll tend to one n case of close resemblance. o measure how close the deformed model s to the mean model shape (sub-crteron b.), one can use Hotellng's test for two samples (Anderson, 958) to test the hypothess that the set of shape parameters s an outcome of the multvarate Gaussan dstrbuton of shape parameters n the tranng set. he test wll result n a sgnfcance level, α (0 α ), and hgh values wll ndcate low degree of deformaton. A combned crteron can be calculated as α = α φ α θ (0 α, φ > 0, θ > 0). Prelmnary nvestgatons have ndcated that the resemblance sub-crteron s more mportant than the deformaton sub-crteron. In most cases φ =.5 together wth θ = 0.44 seem to result n α > 0.5 f the model corresponds to the same weed speces as the object (Fg. 4). he object s therefore classfed accordng to the model that Model: dead nettle (Lamum spp.) Model: charloc (Snapss arvenss) Model: whte goosefoot (Chenopodum album) Model: speedwell (Veronca spp.) Before deformaton (mean model shape) After deformaton α = 56% α = 70% α = 35% α = 58% α = 5% α = % α = 84% α = 59% α = 6% α = 50% α = 4% α = 4% Fgure 4. Result from fttng four dfferent models to a whte goosefoot seedlng (Chenopodum album). 7
results n the hghest α-value. However, f the α-values for all of the models are less than 0.5, the object cannot be dentfed as one of the speces represented by the models. 4. Results and dscusson Untl now, models for weed speces have been generated, and the weed dentfcaton procedure has been tested prelmnarly for sx weed speces. However, the number of tests performed wll not be suffcent to put forward accurate statements about the performance rate of the dentfcaton procedure (lelhood that a gven weed seedlng wll be correctly dentfed), although prelmnary results ndcate that the rate of correctly dentfed weed seedlngs s around 80% or more. In comparson wth other deformable models, e.g. snaes (Manh, 00), one of the advantages of the ASM technque s that the models do not only tae leaf shapes nto account, but also the overall geometry of the seedlngs. hs aspect maes t easer to dscrmnate between speces. he model parameters assocated wth the ASM of a gven weed speces are suffcent for reconstructon of the shapes of real weed seedlngs. Each parameter corresponds to a mode of shape or pose varaton wth an ntutvely understandable nterpretaton, whch may help assessng whether the model s suffcent to model the natural varaton. he algorthm for dentfcaton of weeds has not yet been desgned and optmsed for real-tme use. herefore, the speed of the process s stll rather low (several seconds to dentfy a weed seedlng when usng a common Wndows PC of today). However, the ASM oolt, whch was used for ths study s probably not the most speed effcent mplementaton of the ASM approach, one of the reasons beng that the toolt runs under Matlab. An alternatve and probably faster C++ mplementaton of ASM and AAM (Actve Appearance Models) has been developed at the echncal Unversty of Denmar (see http://www.mm.dtu.d/~aam/). 5. References Anderson,.W. (958). An Introducton to Multvarate Statstcal Analyss. John Wley & Sons, New Yor, 958. Cootes,.F., Hll, A., aylor, C.J. & Haslam, J. (994). he Use of Actve Shape Models for Locatng Structures n Medcal Images. Image and Vson Computng (6): 355-366. Hesel,., Chrstensen, S. & Walter, A.M. (999). Whole-feld experments wth ste-specfc weed management. Proceedngs of the Second European Conference on Precson Agrculture (ed. JV Stafford), Part : 759-768. Manh, A.-G., Rabatel, G., Assemat, L. & Aldon, M.-J. (00). Weed leaf mage segmentaton by deformable templates. Journal of Agrcultural Engneerng Research 80 (), 39-46. MathWors (000). Matlab. he Language of echncal Computng. he MathWors Inc., Massachusetts, USA. Söefeld, M., Gerhards, R. & Kuhbauch, W. (000). Ste-specfc weed control from weed recordng to herbcde applcaton. Proceedngs 0th German Conference on Weed Bology and Weed Control, Stuttgart-Hohenhem, Germany, 4-6 March, 000. Vsual Automaton (998). Actve Shape Model oolt. Software Users Gude. Verson.0. Vsual Automaton Ltd, Unversty of Manchester, UK. Woebbece, D.M., Meyer, G.E., Bargen, K. von & Mortensen, D.A. (995). Color ndces for weed dentfcaton under varous sol, resdue, and lghtng condtons. ransactons of the Amercan Socety of Agrcultural Engneers 38: 59-69. 8