PhenoBot: A Robot System for Phenotyping in the Greenhouse Using 3D Light-Field Technology

Size: px

Start display at page:

Download "PhenoBot: A Robot System for Phenotyping in the Greenhouse Using 3D Light-Field Technology"

Bertha Blair
6 years ago
Views:

1 PhenoBot: A Robot System for Phenotyping in the Greenhouse Using 3D Light-Field Technology IEEE TC Agricultural Robotics - AgRA Webinar #20 August 27, Gerrit Polder, Jan Willem Hofstee, Wageningen UR.

2 Outline! Introduction (PhenoBot = SpySee II)! Autonomous Robot! 3D Light field Camera System! Experiments Lab Greenhouse! Discussion

SpySee (EU project Spicy) Song,&Y.,&Glasbey,&C.&A.,&van&der&Heijden,&G.&W.&A.&M.,&Polder,&G.,&&&Dieleman,&J.&A.&(2011).

&Presented&at&the&17th&Scandinavian&Conference&on&Image&Analysis,& SCIA,&Ystad,&Sweden.& & Song,&Y.,&Glasbey,&C.&A.,&horgan,&G.&W.

&doi:10.1016/ j.biosystemseng.2013.12.008&van&der&heijden,&g.,&song,&y.,&horgan,&g.,&polder,&g.,&dieleman,& A.,&Bink,&M.,&et&al.

&Func3onal) Plant)Biology,&39(11),&870 &877.&doi:10.1071/FP12019& & Horgan,&G.&W.,&Song,&Y.,&Glasbey,&C.&A.,&van&der&Heijden,&G.&W.&A.&M.

3 SpySee (EU project Spicy) Song,&Y.,&Glasbey,&C.&A.,&van&der&Heijden,&G.&W.&A.&M.,&Polder,&G.,&&&Dieleman,&J.&A.&(2011).& Combining&stereo&and&EmeFofFflight&images&with&applicaEon&to&automaEc&plant&phenotyping& (Vol.&6688,&pp.& ).&Presented&at&the&17th&Scandinavian&Conference&on&Image&Analysis,& SCIA,&Ystad,&Sweden.& & Song,&Y.,&Glasbey,&C.&A.,&horgan,&G.&W.,&Polder,&G.,&Dieleman,&J.&A.,&&&van&der&HEIJDEN,&G.&W.& A.&M.&(2014).&ScienceDirect.&Biosystems)Engineering,&118(C),& &doi: / j.biosystemseng &van&der&heijden,&g.,&song,&y.,&horgan,&g.,&polder,&g.,&dieleman,& A.,&Bink,&M.,&et&al.&(2012).&& & SPICY:&towards&automated&phenotyping&of&large&pepper&plants&in&the&greenhouse.&Func3onal) Plant)Biology,&39(11),&870 &877.&doi: /FP12019& & Horgan,&G.&W.,&Song,&Y.,&Glasbey,&C.&A.,&van&der&Heijden,&G.&W.&A.&M.,&Polder,&G.,&Dieleman,&J.& A.,&et&al.&(2014).&Automated&esEmaEon&of&leaf&area&development&in&sweet&pepper&plants& from&image&analysis.&func3onal)plant)biology.&doi: /fp14070& 4* IR, Colour, Range (ToF) cameras

4 SpySee combining stereo and ToF Stereo&pair&+&ToF&range&image&"&detailed&range&image&

5 SpySee - limitations! For use in large greenhouses an autonomous platform is needed! Blue background behind the plants! Cumbersome and time consuming 3D reconstruction! The resolution of the ToF camera (64 x 40 px) was to low for using the depth information for segmentation! Occlusions hampers accurate estimation of leaf and fruit features! Limited depth of focus (DOF) range (difficult to measure objects close to the camera, or far in the background)

PhenoBot = SpySee II SpySee (limitations) Hand driven Colour + Time of Flight camera Cumbersome sensor fusion for pixel registration of depth and colour image Small depth of focus Blue background

6 PhenoBot = SpySee II SpySee (limitations) Hand driven Colour + Time of Flight camera Cumbersome sensor fusion for pixel registration of depth and colour image Small depth of focus Blue background screen needed in adjacent lane Occlusion due to stereo approach PhenoBot Autonomous Robot 3D Light field Camera Camera promises to output direct registered depth and colour image Image is in focus over whole depth range No background screen needed. Reduced occlusion due to micro lenses

7 Robot Trolley! Adapted spraying robot from Berg Hortimotive

8 Robot Trolley (adapted spraying robot)! Spraying boom replaced by camera stand! Adapted software start stop trigger - distance

9 3D Light Field Camera! Placing a micro lens array in front of an image sensor transforms a normal camera into a single lens 3D camera, which also allows the user to change the focus and the point of view after a picture has been taken! While the concept of such plenoptic cameras is known since 1908, only recently the increased computing power of low-cost hardware and the advances in micro lens array production, have made the application of plenoptic cameras feasible

10 3D Light Field Camera! Commercial available cameras: LYTRO raytrix

11 3D Light Field Camera Technology* 3D reconstruction and extended depth-of-field based on only one snapshot and a single-lens camera - calibration-free monocular camera - robust & space-saving setup - down to micron resolution - extended depth-of-field by software re-focus - captures fast moving objects by single shot Light field engine 4D light field raw image data Micro-lens array (MLA) optics perspective shift 3D 3D depth map in total focus 3D view re-focus plane on eye re-focus plane on nose US-Pat.-No.: 2012/ A1, CHIP-Award 2012: Innovation of the year Copyright& &2013&by&Raytrix&GmbH,&Germany.&All&rights&reserved.&Design,&features,& and&specificaeons&are&subject&to&change&without&noece.&

12 3D Light Field Camera (Raytrix)! One camera, one lens, one shot for 3D and 2D data! Up to 7MP effective resolution! 6x extended depth of field! Minimized occlusion! Passive system! Data capture will not interfere with the plants natural behaviour. (no laser scanning or IR pattern projection)! Robust

Cylinder occluded in one of the images, no matching possible!

13 Occlusion 1 2 Example of a setup that is prone to occlusion! Due to the large baseline a stereoscopic system suffers from occlusion. Cylinder occluded in one of the images, no matching possible! Due to thousands of micro lens images, with about the same perspective through the object lens, occlusion is exceptionally reduced

14 PhenoBot First experiments! Lab experiments: Compare hand measurements with 3D imaging Fruit and stem size! Experiments in Enza greenhouse Simultaneously manual measurements of phenotypic characteristics: Stem thickness Leaves length, width and angle Amount of flowers per bunch Amount of ripe and raw fruits per bunch

15 PhenoBot Lab experiments! Relation between depth value and real distance.! Relation between real and measured size: Cherry tomato Beefsteak tomato Truss tomato Tomato stem

16 PhenoBot Lab experiments! Results:

17 PhenoBot Greenhouse measurements

18 PhenoBot Greenhouse measurements

19 PhenoBot Greenhouse measurements! Example data

20 PhenoBot Greenhouse measurements

21 PhenoBot Greenhouse measurements Find red tomatoes based on colour only

22 PhenoBot Greenhouse measurements Find red tomatoes based on colour and depth

23 PhenoBot discussion advantages! One camera, one lens, one shot for 3D and 2D data No 3D reconstruction needed Exposure times similar to standard 2D cameras Image capture with up to 6fps (for R29) 3D data calculation from images with up to 6fps Up to 7MP effective resolution for R29 camera! Robust Industrial camera! Passive system No laser scanning or IR pattern projection No interference with standard plant behaviour

24 PhenoBot discussion advantages! Extended depth of field No need to change main lens focus for different plant growth states! Minimized occlusion find sharp depth discontinuities at leaf borders! Adaptable system Interchange the main lens to adapt to specific depth ranges Large range of lenses available for different depth ranges ( > 40 mm, see cons)

25 PhenoBot discussion - disadvantages! New technique facing a number of teething troubles Software (updates, Halcon, 64 bit, ) Documentation Price! No calibration for 3D reconstruction, but calibration of the micro lens array (grey reference) appeared to be very sensitive to saturated pixels Possible to apply new grey calibration on old data

26 PhenoBot discussion - disadvantages! For main lens only one aperture setting (f/11) Flash illumination can only be controlled with electronic gain! Wide field of view lenses not possible (> 40 mm) SpySee - 75 FOV PhenoBot - 50 FOV! Heavy computational load for calculating depth image. Big computer on robot or offline processing! Large file size (~ 40 Mb lossless compressed)

27 PhenoBot future plans! Search for optimal viewing angle! Image stitching! Illumination / flash! Data/processing flow Offline/online Compression! Feature extraction! QTL analysis Based on features Based on direct image information

28 Questions? Acknowledgments: Dick Lensink (Enza) Bram Veldhuisen and Menno Hutting (students Wageningen UR) Arne Erdmann (Raytrix) Jeroen Gerritse (Berg Hortimotive) This project was co-financed by the Consortium for Improving Plant Yield (CIPY), which is part of the Netherlands Genomics Initiative

Glas robotprojecten + (PicknPack, Phenobot, Trimbot & Sweeper) Erik Pekkeriet

Glas robotprojecten + (PicknPack, Phenobot, Trimbot & Sweeper) Erik Pekkeriet 1 Glas 4.0 4 robotprojecten + (PicknPack, Phenobot, Trimbot & Sweeper) Erik Pekkeriet PicknPack3 4 Globale strategie Yield & quality Stress Gezonde planten Oogst prognose Groei snelheid(homogeneity) Rijheid