Real-Time Crop Row Image Reconstruction for Automatic Emerged Corn Plant Spacing Measurement

Transcription

1 Agricultural ad Biosystems Egieerig Publicatios Agricultural ad Biosystems Egieerig 2008 Real-Time Crop Row Image Recostructio for Automatic Emerged Cor Plat Spacig Measuremet Lie Tag Iowa State Uiversity, Lei F. Tia Uiversity of Illiois at Urbaa-Champaig Follow this ad additioal works at: Part of the Agriculture Commos, ad the Bioresource ad Agricultural Egieerig Commos The complete bibliographic iformatio for this item ca be foud at abe_eg_pubs/334. For iformatio o how to cite this item, please visit howtocite.html. This Article is brought to you for free ad ope access by the Agricultural ad Biosystems Egieerig at Iowa State Uiversity Digital Repository. It has bee accepted for iclusio i Agricultural ad Biosystems Egieerig Publicatios by a authorized admiistrator of Iowa State Uiversity Digital Repository. For more iformatio, please cotact digirep@iastate.edu.

2 Real-Time Crop Row Image Recostructio for Automatic Emerged Cor Plat Spacig Measuremet Abstract I-field variatios i cor plat spacig ad populatio ca lead to sigificat yield differeces. To miimize these variatios, seeds should be placed at a uiform spacig durig platig. Sice the ability to achieve this uiformity is directly related to plater performace, itesive field evaluatios are vitally importat prior to desig of ew platers ad curretly the desigers have to rely o maually collected data that is very time cosumig ad subject to huma errors. A machie visio-based emerged crop sesig system (ECSS) was developed to automate cor plat spacig measuremet at early growth stages for plater desig ad testig egieers. This article documets the first part of the ECSS developmet, which was the real-time video frame mosaickig for crop row image recostructio. Specifically, the mosaickig algorithm was based o a ormalized correlatio measure ad was optimized to reduce the computatioal time ad ehace the frame coectio accuracy. This mosaickig algorithm was capable of recostructig crop row images i real-time while the samplig platform was travelig at a velocity up to 1.21 m s-1 (2.73 mph). The mosaickig accuracy of the ECSS was evaluated over three 40 to 50 m log crop rows. The ECSS achieved a mea distace measuremet error ratio of -0.11% with a stadard deviatio of 0.74%. Keywords Cor plat spacig measuremet, Image mosaickig, Machie visio, Real-time Disciplies Agriculture Bioresource ad Agricultural Egieerig Commets This article is from Trasactios of the ASABE 51, o. 3 (2008): This article is available at Iowa State Uiversity Digital Repository:

3 REAL TIME CROP ROW IMAGE RECONSTRUCTION FOR AUTOMATIC EMERGED CORN PLANT SPACING MEASUREMENT L. Tag, L. F. Tia ABSTRACT. I field variatios i cor plat spacig ad populatio ca lead to sigificat yield differeces. To miimize these variatios, seeds should be placed at a uiform spacig durig platig. Sice the ability to achieve this uiformity is directly related to plater performace, itesive field evaluatios are vitally importat prior to desig of ew platers ad curretly the desigers have to rely o maually collected data that is very time cosumig ad subject to huma errors. A machie visio based emerged crop sesig system (ECSS) was developed to automate cor plat spacig measuremet at early growth stages for plater desig ad testig egieers. This article documets the first part of the ECSS developmet, which was the real time video frame mosaickig for crop row image recostructio. Specifically, the mosaickig algorithm was based o a ormalized correlatio measure ad was optimized to reduce the computatioal time ad ehace the frame coectio accuracy. This mosaickig algorithm was capable of recostructig crop row images i real time while the samplig platform was travelig at a velocity up to 1.21 m s 1 (2.73 mph). The mosaickig accuracy of the ECSS was evaluated over three 40 to 50 m log crop rows. The ECSS achieved a mea distace measuremet error ratio of 0.11% with a stadard deviatio of 0.74%. Keywords. Cor plat spacig measuremet, Image mosaickig, Machie visio, Real time. It is well kow to cor producers that ueve plat spacig ad emergece may reduce yield potetial. Nielse (2005) reported that ueve cor plat spacig withi the row decreased yield up to two bushels per acre for every ich icrease of stadard deviatio of platto plat spacig. The performace of plater meterig mechaisms directly determies how uiformly the seeds ca be placed at a appropriate spacig. Cosequetly, plater performace will have substatial ifluece o crop establishmet ad fial yield. Thus, the uiformity of plat spacig is a importat parameter that plater maufacturers use to evaluate plater performace. Extesive field experimets thus are carried out by maufacturers durig early crop growth stages ad over differet soil coditios. However, acquirig maual measuremets of plat spacig requires substatial labor ad time resources. Maual measuremets ot oly limit the quatity of data, but also itroduce huma errors. This situatio has made a automated ad accurate early growth stage cor plat spacig sesig system highly desirable. Developig the sesig techology to automatically idetify each cor plat before harvestig has bee a Submitted for review i November 2005 as mauscript umber IET 6068; approved for publicatio by the Iformatio & Electrical Techologies Divisio of ASABE i May Preseted at the 2002 ASAE Aual Meetig as Paper No The authors are Lie Tag, ASABE Member Egieer, Assistat Professor, Departmet of Agricultural ad Biosystems Egieerig, Iowa State Uiversity, Ames, Iowa; ad Lei F. Tia, ASABE Member Egieer, Associate Professor, Departmet of Agricultural ad Biological Egieerig, Uiversity of Illiois at Urbaa Champaig, Urbaa, Illiois. Correspodig author: Lie Tag, Departmet of Agricultural ad Biosystems Egieerig, Iowa State Uiversity, 203 Davidso Hall, Ames, IA 5001; phoe: ; fax: ; e mail: lietag@ iastate.edu. challegig problem i agricultural research. Machie visio techologies have bee widely adopted to replace huma labor i various ispectio ad measuremet applicatios, ad have bee applied to these types of problems. Jia et al. (1991) ad Shrestha ad Steward (2003, 2005) ivestigated machie visio approaches for cor plat sesig. To acquire plat spacig data through machie visio, the locatio of each idividual plat eeds to be determied. I additio, sice a crop row is captured i a series of sequetial images, i.e., video frames, the locatios of plats eed to be tracked or retaied across these frames i order to calculate the plat spacig whe two adjacet plats are separated by frames. There are two possible approaches to this problem: (1) plat shape feature based trackig, ad (2) soil backgroud based mosaickig. I the plat shape feature based matchig approach, the locatios of a idividual plat i two sequetial images are first foud, ad the those images are coected at these locatios. This approach was adopted by Sachiz et al. (1996) for trackig cabbage plats i a series of sequetial images. I the research reported herei i this curret article, the first versio of a cor plat spacig sesig system was based o the plat feature based trackig approach, but the system failed at field tests uder two frequetly occurrig coditios: widy weather ad large plat to plat gaps betwee two sequetial frames. This failure occurred because the wid caused cor plat caopy movemet betwee two sequetial frames, thus makig plats utrackable across the frames. Large iter frame plat gaps could essetially break the trackig process as well whe o plat appeared withi the overlapped regio of two sequetial frames. The video frame mosaickig approach relies o searchig the spatial cotiuity of sequetial frames i the image backgroud cosistig of soil ad residue. I this way, two sequetial frames are coected at a matched locatio foud Trasactios of the ASABE Vol. 51(3): America Society of Agricultural ad Biological Egieers ISSN

4 i the image backgroud, ad wid iterferece ad iterframe plat gaps will ot cofoud the process. Shrestha ad Steward (2003) took this approach ad developed a machie visio based cor plat populatio sesig system. I their research, a image mosaickig algorithm was developed i which the miimum value of the sum of absolute errors over matchig patches betwee two sequetial frames was foud. To improve the mosaickig speed, Shrestha et al. (2004) later proposed a gradiet ascet method of the frame correlatio surface with Kalma filter predictio ad achieved a algorithm that was te times faster tha the miimum error method. They also idicated, however, that the success of this algorithm depeded o precise shift predictio ad the characteristics of the correlatio surface. Therefore, for the overall objective of developig a machie visio based emerged crop sesig system (ECSS) for cor plat spacig measuremet, the first task was to develop a mosaickig algorithm that ca accurately ad reliably retai the spatial iformatio of a sequece of overlapped crop row video frames. Uder this overall objective, oe importat requiremet of the ECSS was real time performace, i.e., the mosaickig process must keep up with the velocity of the image recordig platform whe samplig alog a crop row sice this will greatly ehace the system efficiecy. For the ECSS, the targeted samplig speed was set about 1.11 m s 1 (2.50 mph), which is similar to typical huma walkig speed. Aother importat requiremet of the ECSS was the distace measuremet accuracy. Ideally, plater egieers would require that the average cor plat spacig measuremet error be smaller tha ±5 mm. Typically, cor plat populatios are aroud 30,000 plats per acre i the Midwester U.S. With a 0.76 m (30 i.) row spacig, this populatio would result i a 0.18 m (6.97 i.) cor plat spacig. Thus, a ±5 mm distace error requires that the distace measuremet error be about 3% or less of the true distace. Cosequetly, the specific objectives of this research were to (1) develop a video frame mosaickig algorithm that is capable of accurately recostructig cor plat row images i real time, ad (2) evaluate the performace of the mosaickig algorithm i terms its speed ad accuracy. MATERIALS AND METHODS There were two mai tasks i the crop row image recostructio process. The first task was image acquisitio ad camera calibratio, while the secod task was video frame mosaickig. IMAGE ACQUISITION AND PREPROCESSING Image Acquisitio Uder Outdoor Lightig Coditios To precisely measure crop plat spacig, high resolutio images are ecessary. I this research, video was recorded usig a 3CCD digital video camcorder (DCR VX2000 NTSC, Soy, Japa). A customized computer with dual 400 MHz Petium CPU ad Microsoft Widows NT 4.0 OS was used for video processig. All algorithms were developed by usig Microsoft Visual C Video frames were acquired usig a color frame grabber (PXC200, Imageatio, Beaverto, Ore.) via a S video iput. The camcorder was mouted either o a tractor or a bicycle ad was aimed vertically dowward to capture the top view of a crop row. The wide dimesio of the camcorder field of view was orieted parallel to the crop row to provide larger video coverage alog the samplig directio. The camera height varied slightly to accommodate differet crop caopy sizes at differet growth stages, but remaied at 1 m from the groud. Cosequetly, the spatial resolutio of the images was i a rage of 1.0 to 1.4 mm per pixel. To overcome the motio effects due to travel ad camera vibratio, oly oe field of a video frame was used for image aalysis. Thus, each processed video frame had a size of pixels. The camcorder shutter speed was set at 1/500 to 1/1000 depedig o the light itesity. The camcorder was maually focused durig the system setup ad fixed for the rest of the video recordig process. The auto focus fuctio was itetioally disabled because it was too slow to hadle the rapid focal distace chages geerated by camera vibratio. Uder atural outdoor lightig coditios, direct sulight casts shadows that create itesity ad color variatios withi the image ad across sequetial video frames. This icosistecy i image formatio ca be detrimetal for the image mosaickig techique, which is based o correspodece searchig across sequetial images. Aother problem associated with direct exposure to sulight is that plat leaves with waxy surfaces ca have a strog specular reflectio, thus causig saturated pixels. Tia (1995) idicated that, although a traditioal polarizig filter could reduce some of the glare, it also slightly chaged the hue of the image. This color chage ca reduce the accuracy of color based image segmetatio, which was importat for both image mosaickig ad subsequet plat idetificatio. To overcome these difficulties, the video samples used i this research were acquired either o cloudy days or with a light diffuser o suy days. A video recordig platform usig a modified bicycle is illustrated i figure 1. Camera Calibratio To recover accurate spacig iformatio betwee plats, machie visio based measuremet requires a spatial trasformatio that ca relate image coordiates i the 2D image plae back ito its origial 3D world coordiate system. A iverse perspective trasformatio was derived by usig a perspective trasformatio matrix techique described by Gozalez ad Woods (1992). I this calibratio process, a uique calibratio matrix ca be determied from a miimum of four pixels. Because this calibratio techique is a least square based approach, a larger umber of pixels is desired so that the matrix is over determied for better Figure 1. Video recordig usig a modified bicycle uder a suy day TRANSACTIONS OF THE ASABE

5 calibratio accuracy. I the ECSS, the calibratio algorithm was desiged to take eight calibratio poits, which were maually selected from images of a calibratio pael. IMAGE MOSAICKING Correlatio Based Matchig The image mosaickig algorithm was developed to compute iterframe displacemet so that two spatially overlapped sequetial video frames acquired at differet times could be coected to form a mosaicked image. A commo approach for producig mosaicked images is to compare the liear relatioship of itesity profiles i the eighborhood of potetial matches usig correlatio measure as a similarity criterio (Forsyth ad Poce, 2002). This matchig process ivolves a pixel wise search, ad the coectig poit is give by the locatio of a matched widow that maximizes the similarity criterio withi a search regio (Trucco ad Verri, 1998). If a matchig widow W i the previous frame is defied to have a size of p pixels, where p =, ad to be cetered p at positio (u, v), the the vector ( u, v) of widow W ca be obtaied by readig i the itesity values of the pixels withi the widow row by row. Give a potetial match occurrig at positio (u + d u, v + d v ) i matched widow W i the curret frame, the secod vector ω (u + d u, v + d v ) ca be costructed i the same way (fig. 2). Subsequetly, the correspodig ormalized correlatio fuctio (correlatio coefficiet) ca be defied as: 1 1 C( d u, d v ) = (1) where ω ad ω are the ormalized vectors with zero mea, which are defied as: = (2) = (3) where ad are vectors of dimesio p ad cosist of the mea values of the elemets i vectors ω ad ω, respectively, ad are calculated by: 1 = p 1 = p p i i= 1 1 p p 1 1 M (4) i i= 1 1 p 1 1 M (5) The output of the ormalized correlatio fuctio rages from 1 to +1, ad it reaches its maximum value whe the image itesity profiles of the two widows are related by a affie trasformatio A = Aλ + μ for some costats λ ad μ with λ > 0. The ivariace of C to affie trasformatios of the itesity fuctio affords correlatio based matchig techiques some degree of robustess over the itesity chage across two sequetial frames. For matchig widows with zero mea ad uit Frobeius orm, maximizig the correlatio ad miimizig the sum of squared differeces (SSD) are equivalet. Mathematically, the correlatio fuctio calculates the cosie of the agle betwee vector ω ad vector ω, whereas the SSD measures the 2 orm or the Euclidea legth of vector ω ω. The correlatio ad SSD measures for vectors ω ad ω are preferable to the sum of absolute differece (SAD) because SAD does ot itrisically measure the cross correlatio ad the liear relatioship betwee the vectors. I the ormalized correlatio based method, a exact copy of the matched patter ca hardly be expected i the search area because some part of the patter is usually corrupted by oise, geometric distortio, or occlusio. Therefore, istead of lookig for a absolute match, it is more appropriate to seek the maximum of C over a search area i the curret frame. Computatioal Optimizatio of Image Mosaickig Sice the video equipmet ofte moved alog the row at a varyig velocity, the size of the search area eeded to be adequately large. The maximum allowable travel velocity is determied by the computatioal requiremets of the image mosaickig algorithm ad the computatioal capacity of the computig hardware. Obviously, a larger search area will substatially icrease the computatioal time for a pixel Matchig Widow (u, v) W Search Area Matched W Widow (u+d u,v+d v ) m m Previous Frame Curret Frame Figure 2. Durig the correlatio based matchig process, a matchig widow W cetered at positio (u, v) i the previous frame is matched with a widow W cetered at positio (u + d u, v + d v ) i the curret frame by fidig the maximum correlatio of the pixels i W across each possible locatio of W i the search area. I this illustratio, the search area is of size m m pixels ad both widows W ad W have a size of pixels. Vol. 51(3):

6 wise correlatio based matchig algorithm. It is, thus, importat to desig a computatioally efficiet implemetatio to improve the travel velocity ad to allow the use of a larger search area. This efficiecy was realized through the use of recursio ad lookup tables. I a typical implemetatio of ormalized correlatiobased image mosaickig, oe computes C(d ) (eq. 1), where d is a 2 dimesioal shift vector defied as (d u, d v ), at each pixel for each possible shift d ad fids the shift for which C(d ) is maximum. Although geeratig ω ad of the pre selected matchig widow of size pixels i the previous frame oly requires O( 2 ) subtractio calculatios, the search for a potetial match i the curret frame ca be computatioal expesive because ω ad must be updated o a pixel wise basis. If the size of the search area is m m, the stadard implemetatio of equatio 1 requires O(m 2 2 ) subtractios to obtai ω ad i order to cover every possible ceter positios of the widow areas i the search area. However, it ca be show that: 2 ( )( ) = I I where I ad I are two scalar variables represetig the average pixel itesity values of widows W ad W, respectively. Similarly, the followig equatio ca be derived: ( I ) 2 (6) = (7) Usig equatios 6 ad 7, the direct computatio of vector ω, i.e.,, is avoided, ad I ca be computed recursively. To do this, I is computed for a shift d usig the I L of its immediate left eighbor matchig widow W L. More specifically, the cotributio to I L ca be subtracted from the leftmost colum of W L ad the cotributio from the colum immediately to the right of W L ca be added. This recursive process ca be formulated as the equatio below: 2 L WL i= 1 i= 1 2 ( i,1) + W ( i, ) I I = (8) This recursive calculatio ca be applied both horizotally ad vertically across the search area, which sigificatly reduces the required additios from O(m 2 2 ) to O(2m 2 ). Although the use of equatios 6 to 8 ca reduce the additios, O(m 2 2 ) multiplicatio operatios are still required to compute ω ω ad ω ω. Sice a sigle multiplicatio operatio takes sigificatly more CPU clock cycles tha a sigle assigmet operatio, the algorithm ca be made eve faster if all possible values of ω ω ad ω ω are pre computed ad stored i a 2 dimesioal lookup table. Because the itesity values of pixels are represeted by 8 bit umbers, the lookup table is easy to implemet ad oly takes about 64K bytes of memory. Durig implemetatio, both stadard ad optimized mosaickig algorithms were coded, ad tests were coducted to quatify the effects of this computatioal optimizatio o ru time. Whe the mosaickig algorithm was implemeted, the memory required for the mosaicked image was pre allocated accordig to the maximum available free memory. Preallocatio saved time over dyamically reallocatig memory for each pair of sequetial video frames ad helped maitai a costat frame processig time. This costat processig time was required to specify a maximum allowable travel velocity. IMPLEMENTATION OF FRAME TO FRAME MATCHING Overall Setup of Frame to Frame Matchig I the ECSS implemetatio, two spatially overlapped video frames were matched to fid their spatial correspodece. I discussig the processig of the icomig video sigal, the most recetly acquired video frame is called the curret frame, while the oe immediately before it is called the previous frame. After each mosaickig operatio usig the previous ad curret frames, the curret frame was copied ito the memory cotaiig the previous frame, ad a ew frame was acquired to replace the curret frame i its memory locatio. Hece, the updatig speed of the previous ad curret frames was determied by the operatioal time of mosaickig, ot by the stadard NTSC video frame rate (30 frames per secod). Video frame elemets used i the desig of a correlatio based matchig procedure for both previous ad curret frames are depicted i figure 3. Amog those elemets, the key elemet was the matchig widow area, which was first selected i the previous frame ad the used as a matchig widow to fid its best match (a matched widow) i the curret frame. Withi the curret frame, a search area was defied to be startig from ad vertically symmetric aroud the mapped locatio of the matchig widow defied i the previous frame (fig. 3). The ormalized correlatio measure associated with every possible matched widow withi the search area was calculated. I additio, the matchig widow area selected from the previous frame was surrouded by a 5 pixel wide vegetatio buffer zoe o all four sides, which i tur produced a larger widow, called a buffered matchig widow. The vegetatio buffer zoe was used to better exclude vegetatio from the matchig process, as wid could possibly blow the adjacet vegetative objects such as leaves ito the widow area durig the time iterval betwee the acquisitio of the previous ad curret frames, thus causig icorrect matchig. The ormalized correlatio fuctio (eq. 1) eeds to be computed for all (N I/2 1) (M H/2 1) locatios of all possible matched widows withi the search area to fid the best match. There were two frame boudary buffer zoes located at the upper ad bottom edges of the 3rd quarter area of the previous frame, ad each had a size of pixels. A matchig widow i the previous frame was selected from the 3rd quarter area. Because of the possibility of mior rotatio ad vertical displacemet betwee sequetial frames, the frame boudary buffer zoes were excluded from the selectio of the matchig widow i the previous frame to better esure the existece of a match i the curret frame. Usually, image itesity values are used i correlatio calculatio. The camcorder used i this research had red, gree, ad blue sigal bads geerated by three CCD sesors. Although the itesity value at each pixel could be calculated by averagig its red, gree, ad blue values, a better approach was to directly use the red bad sigal for correlatio calculatio. Nitsch et al. (1991) idicated that soil ad residue reflectivity curves had a costat to slightly 1082 TRANSACTIONS OF THE ASABE

7 J 640 ÖÖÖÖÖ 243 y Travel Directio H K ÔÔÔ Matchig Widow ÔÔÔ I Vegetatio Buffer Zoe x 2 d Quarter Area 3 rd Quarter Area Previous Frame Frame Boudary Buffer Zoe N y M Matched Widow Search Area Matchig Widow x 2 d Quarter Area 3 rd Quarter Area Curret Frame Figure 3. Relatioship betwee frame area, search area, widow area, vegetatio buffer zoe, ad frame boudary zoes (ot to scale.) icreasig tred over the 400 to 900 m wavelegth regio. This spectral property implies that the red bad sigal is very likely to improve the texture iformatio associated with image backgroud (soil ad residues) where the correlatio based match was sought. Therefore, the red color bad was used to calculate the ormalized correlatio measure, which also improved the computatioal efficiecy by elimiatig pixel wise itesity calculatios. Matchig Widow Selectio To esure that a correlatio based matchig algorithm performs well, the selectio of a matchig widow i the previous frame is critical. Pratt (1974) stated two basic problems with the simple correlatio measure. First, the broadess of the correlatio fuctio makes detectio of the peak difficult sice the simple correlatio measure igores the spatial relatioship of poits i the image. Secod, image oise may lead to a false peak correlatio. To alleviate these problems, a spatially matched filterig process or a statistical measure ca be icorporated to decorrelate or white the image before the correlatio based matchig process. I the case of objects i images, a whiteig filter resembles a high pass filter. Sice images of atural scees have a sigificat amout of atural spatial correlatio, the statistical correlatio measure utilizes edge outlie comparisos betwee the two scees. A edge detector type of spatial filter ca icrease the accuracy of correlatio based matchig if the motio of the scee or the camera is purely traslatioal. If a mior rotatio does occur, high pass filterig before correlatio aalysis ca reduce the accuracy i fidig the optimum peak. Although the ECSS had limited camera rotatio, mior rotatios did exist. So, i ECSS, high pass filterig techiques were ot implemeted before the correlatio aalysis. Istead, the matchig widows with the greatest high frequecy cotet ad strogest textural iformatio were used to correlate sequetial frames (Trucco ad Verri, 1998). Specifically, the pixel itesity variace withi a matchig widow was calculated at each possible locatio by scaig through the 3rd quarter area of the previous frame. Durig this scaig process, a coarser step size (five rows per vertical step ad 30 colums per horizotal step) was used to save time. Sice wid ca geerate chages i vegetative object shapes across two matchig frames, vegetatio free areas were prioritized i the process of matchig widow selectio. Cosiderig both strog texture ad vegetatio free criteria, the followig rules were adopted for the matchig widow selectio process: If there were vegetatio free matchig widow ad buffered zoe areas, the the matchig widow was the widow havig the maximum variace ad residig i a vegetatio free matchig widow ad buffered zoe area. Else if there were vegetatio free matchig widows, the the matchig widow was the widow havig the maximum variace ad without vegetatio. Else the matchig widow was the widow havig the maximum variace. I practice, the above matchig widow selectio process could robustly detect ad select areas cotaiig residues, rock edges, soil cracks, ad rough soil surfaces. Cosequetly, the selected widow areas provided more stable ad distiguishable texture features for a more accurate iterframe displacemet calculatio. Vol. 51(3):

8 (a) (b) Figure 4. Illustratio of the segmetatio results: (a) origial image, ad (b) segmeted image. Vegetatio was detected usig a color segmetatio approach developed by Steward ad Tia (1998). Their algorithm utilized a EGRBI (excess gree, red blue, itesity) color trasformatio, K meas clusterig, ad Bayes classificatio. A segmetatio example is give i figure 4. The performace of a correlatio based matchig is also sesitive to the size of the matchig widow. Jai ad Jai (1981) foud that the accuracy of the area correlatio method was poor whe the widow size was small. However, icreasig the widow size ca substatially icrease the computatioal time. Therefore, whe determiig the size of the matchig widow, a tradeoff is eeded such that both the matchig accuracy ad the real time mosaickig objectives ca be achieved. I the ECSS, the matchig widow area size of (I = 51 ad H = 11) pixels was foud to provide adequate textural iformatio for a cosistet mosaickig performace while meetig real time requiremets. Search Area Size Determiatio I geeral, icreasig the size of the search area allows a larger displacemet betwee the previous ad curret frames. However, a larger search area also icreases the time required for a frame coectio to be made. I the ECSS, a larger search area is preferable because it allows frame segmets of larger size to be coected ito the mosaicked image, which meas fewer frame coectig operatios for a fixed legth of crop row, thus reducig potetial distace measuremet errors due to possible imperfect mosaickig. O the other had, implemetig a larger search area ievitably icreases the time eeded for a frame coectio operatio, resultig i a greater chage i viewig agle for the same object i two mosaicked frames, assumig a costat travel velocity. A agled view causes geometric distortios ad object occlusios, which ca potetially degrade the iterframe widow matchig accuracy. Therefore, the size of the search area must be a compromised solutio, ad through trial ad error, the size of the search area was defied to be (N = 280 ad M = 40) pixels. Mosaicked Image Geeratio Oce video iterframe distace was foud usig the correlatio measure, the video frame sequece was coected (fig. 5). Whe implemetig this image mosaickig process, the curretly acquired frame was matched with the previous frame. The distace icremets alog the X ad Y axes after the ith mosaickig operatio were: Δx i = x pi x ci (9) Δy i = y ci y pi (10) where (x pi, y pi ) is the positio of the matchig widow ceter i the previous frame, ad (x ci, y ci ) is the positio of the matched widow ceter i the curret frame. The mosaicked image grew as image mosaickig proceeded. I the mosaicked image, the cumulative displacemets at the ith image i the X ad Y directios after i mosaickig operatios were: i i 1 x x pi X = = + (11) i i Y = =1 y (12) Sequetial frames were coected with detected traslatioal motio parameters {Δx i, Δy i, X i, Y i } at every mosaickig poit where the matchig widow i the previous frame was cetered. Whe the absolute value of Y i became large, cor plats could be shifted out from either the upper or lower edge of the mosaicked image. I other words, the displacemet alog the lateral directio to the crop row, Y, could accumulate ad evetually drive the scee out of the vertical view rage (243 pixels) of the mosaicked image. This would evetually happe whe either the agle betwee the crop row directio ad the video recordig course (a) or the camera iitial orietatio (b) was ot zero (fig. 5). The shaded areas from every frame i figure 5 costituted the actual scee beig coected ito a mosaicked image. Assumig that a crop row is straight ad that it origiated from a poit o the horizotal middle lie of the first frame, 1084 TRANSACTIONS OF THE ASABE

9 Figure 5. Illustratio of the problem of crop row vaishig. The shaded areas from every frame costituted the actual scee beig coected ito a mosaicked image. ad defiig the height of the frames as H ad the legth of the mosaicked image at frame i to be L i, the costrait for prevetig the crop row from vaishig from the mosaicked image at frame i was: L i ta(max{, }) < H / 2 (13) I the actual implemetatio of the image mosaickig procedure, the followig rule was eforced to prevet crop row from vaishig: If Y i > 40 pixels, the Y i = Y i 40 pixels (14) where the umber 40 was the maximum allowable offset o the Y axis. Whe this rule was satisfied, a discotiuity occurred i the mosaicked image. These types of discotiuities were called mosaickig breakpoits. A mosaickig breakpoit could potetially split a cor plat, which would make the automated plat idetificatio for spacig measuremet more difficult. This problem was later solved i the plat idetificatio algorithm. A mosaicked image cosisted of may fragmets from a sequece of image frames. The coectig poits of these fragmets were called mosaickig poits. Sice these fragmets origiated from differet portios of origial frames, their positios relative to the sesig uit varied. To precisely compute the spacig across these fragmets i a mosaicked image, every fragmet i the mosaicked image was marked with a mosaickig poit or a mosaickig breakpoit alog with its image coordiates i its origial sigle frame image (fig. 6). I this way, the camera calibratio matrix, which was useful to correct the oliear distortio caused by the les, could remai valid ad could be used for plat spacig calculatios from the mosaicked image. MOSAICKING PERFORMANCE TESTS First, real time performace of the image mosaickig algorithm was ivestigated by calculatig the mea processig time of a mosaickig operatio over 100 mosaickig operatios. This time was used to estimate the allowable travel velocity of the video recordig platform. Mosaickig accuracy was tested by usig the video tapes recorded i experimetal fields i differet states icludig Texas, Kasas, Illiois, ad Iowa. The algorithm was repeatedly tested over sample paths about 40 to 50 m log. These sample paths had various weed ifestatio, soil tillage, ad crop growth stage coditios. I particular, a careful validatio test was coducted by usig three crop row videos recorded i Texas ad Iowa. The cor plats i these three sampled crop rows were mostly at V3 growth stage. The crop row i Texas had a legth of m, while the other two crop rows i Iowa were m ad 41.0 m log, respectively. I total, there were 12 video clips recorded from these crop rows; for every sampled crop row, there were two paths recorded i oe directio ad two paths recorded i the opposite directio. The true legth of each tested crop row was maually measured by layig a measurig tape alog the crop row. The camera was calibrated o site immediately after the system was set up for recordig. The total legth measured from the mosaicked image was the compared with the true legth to evaluate the accuracy of the mosaickig algorithm. Specifically, the distace measuremet error ratio was calculated as: Figure 6. Sub segmet from mosaicked images of Texas corfields, where idicates mosaickig poits, ad X represets a mosaickig breakpoit. Vol. 51(3):

10 (a) (b) Figure 7. Sample segmets of mosaicked images of corfields i (a) Iowa ad (b) Illiois. Err EL TL ER = = (15) TL TL where ER, EL, ad TL are the measuremet error ratio, the estimated crop row legth, ad the tape measured crop row legth, respectively. Mea compariso of distace measuremet error ratios from these three crops rows was coducted usig oe way aalysis of variace (ANOVA) provided by JMP 6 statistical software (SAS Istitute, Ic., Cary, N.C.). RESULTS With a matchig widow size of pixels ad a search area of pixels, the stadard implemetatio of the ormalized correlatio based matchig algorithm required o average 0.43 s to coect two frames. Sice the camera's horizotal view was 762 mm (2.5 ft), the correspodig maximum allowable travel velocity (video recordig speed) was 0.60 m s 1 (1.34 mph). I cotrast, the frame coectio time was oly 0.25 s after employig the recursive techique based o equatios 6 to 8 ad decreased further to 0.21 s whe a lookup table was icorporated to calculate ω ω ad ω ω used i these equatios. This improved computatioal process icreased the overall mosaickig speed by 51%, ad the maximum allowable video recordig velocity was the correspodigly icreased from 0.60 m s 1 (1.34 mph) to 1.21 m s 1 (2.73 mph). Examples of mosaicked image sub segmets created by usig the developed image mosaickig procedure are provided i figure 7. Through ispectig the cotiuity of objects at the mosaickig poits, such as residues, plat leaves, ad rocks, the image mosaickig algorithm was show to work well visually. The otable ueve spacig of the mosaickig poits i figure 7 was largely due to variable movig velocity of the samplig platform ad the fact that every mosaicked fragmet was coected at the ceter positio of the matchig widow i the previous frame, whose locatio was determied by the matchig widow selectio rules described earlier. Row Locatio Table 1. Mosaickig algorithm field test results. Err (m) ER (%) Mea (STD) of ER (%) Texas (0.34) Iowa (0.40) Iowa (0.98) Overall 0.11 (0.74) Whe the mosaickig accuracy of the algorithm was tested over three sample crop rows, the algorithm achieved a mea crop row legth measuremet error ratio of 0.11% with a stadard deviatio (STD) of 0.74% (table 1). There was o evidece of sigificat differeces i the error ratio across three test rows (F 2,9 = 2.86, P = 0.11), idicatig that the mosaickig algorithm performed cosistetly over those experimetal rows i spite of differeces i soils ad residue cover. However, the mea ER values from the Texas ad Iowa rows were biased positively ad egatively, respectively, implyig that differet camera calibratios most likely caused measuremet error, sice camera calibratio error will result i a cumulative measuremet error. CONCLUSIONS A algorithm for real time cor crop row image recostructio was developed ad evaluated accordig to computatioal time ad image mosaickig accuracy specificatios. From this research, we ca coclude: Crop row mosaickig ca be doe meetig the real time requiremets of a typical field data collectio system. Specifically, the algorithm will allow a 1086 TRANSACTIONS OF THE ASABE

11 maximum data collectio speed of 4.4 km h 1 (2.7 mph) whe tested o a 400 MHz dual Petium CPU computer. Crop row mosaickig accuracy uder typical cor field coditios meets the requiremets for cor plater performace testig. A mea mosaickig distace measuremet error ratio of 0.11% with 0.74% stadard deviatio was observed i this research. ACKNOWLEDGEMENT This research of the Iowa Agriculture ad Home Ecoomics Experimet Statio, Ames, Iowa, Project No. 0475, was supported by Hatch Act ad State of Iowa fuds. This research has also bee supported by the Illiois Coucil of Food ad Agricultural Research (C FAR), project umber IDACF 01 DS 3 1 AE ad the Uiversity of Illiois. Ay opiios, fidigs, ad coclusios or recommedatios expressed i this publicatio are those of the author(s) ad do ot ecessarily reflect the views of the Iowa State Uiversity ad the Uiversity of Illiois. REFERENCES Forsyth, D., ad J. Poce Computer Visio: A Moder Approach. Eglewood Cliffs, N.J.: Pretice Hall. Gozalez, R. C., ad R. E. Woods Digital Image Processig. Readig, Mass.: Addiso Wesley. Jai, R. J., ad A. K. Jai Displacemet measuremet ad its applicatio i iterframe image codig. IEEE Tras. Comm. 29(12): Jia, J., G. W. Krutz, ad H. W. Gibso Cor plat locatig by image processig. I Proc. SPIE 1379: Optics i Agriculture, J. A. DeShazer ad G. E. Meyer, eds. Belligham, Wash.: SPIE. Nielse, R. L Effect of plat spacig variability o cor grai yield. Extesio Research Report. West Lafayette, Id.: Purdue Uiversity, Agroomy Departmet. Available at: Accessed 22 July Nitsch, B. B., K. V. Barge, ad G. E. Meyer Visible ad ear ifrared plat, soil ad crop residue reflectivity for weed sesor desig. ASAE Paper No St. Joseph, Mich.: ASAE. Pratt, W. K Correlatio techiques of image registratio. IEEE Tras. Aerospace ad Electroic Systems 10(3): Sachiz, J. M., F. Pla, J. A. Marchat, ad R. Brivot Structure from motio techiques applied to crop field mappig. Image ad Visio Computig 14(5): Shrestha, D. S., ad B. L. Steward Automatic cor plat populatio measuremet usig machie visio. Tras. ASAE 46(2): Shrestha, D. S., ad B. L. Steward Shape ad size aalysis of cor plat caopies for plat populatio ad spacig sesig. Applied Eg. i Agric. 21(2): Shrestha, D. S., B. L. Steward, K. R. Thorp, ad L. Bo A rapid video frame correspodece algorithm for agricultural video field surveyig. ASAE Paper No St. Joseph, Mich.: ASAE. Steward, B. L., ad L. F. Tia Real time weed detectio i outdoor field coditios. I Proc. SPIE 3543: Precisio Agriculture ad Biological Quality, G. E. Meyer ad J. A. DeShazer, eds. Belligham, Wash.: SPIE. Tia, L. F Kowledge based machie visio system for outdoor plat idetificatio. Chapter 3. Upublished PhD diss. Davis, Cal.: Uiversity of Califoria at Davis. Trucco, E., ad A. Verri Itroductory Techiques for 3 D Computer Visio, , , Eglewood Cliffs, N.J.: Pretice Hall. Vol. 51(3):

12 1088 TRANSACTIONS OF THE ASABE