4D Crop Analysis for Plant Geometry Estimation in Precision Agriculture

Transcription

1 4D Crop Analysis for Plant Geometry Estimation in Precision Agriculture MIT Laboratory for Information & Decision Systems IEEE RAS TC on Agricultural Robotics and Automation Webinar #37

2 Acknowledgements Frank Dellaert Jing Dong Andrew Melim Stefano Fenu 2 Glen Rains Gary McMurray Zsolt Kira David Jensen

3 The Borg Lab at Georgia Tech working at the boundary between robotics and computer vision 3 Inference, Sensor fusion, Simultaneous Localization and Mapping (SLAM) [Carlone, Dong, Dellaert: visual-inertial navigation]

4 The Borg Lab at Georgia Tech theory & algorithms for Simultaneous Localization and Mapping (SLAM) first global convergence results [Carlone & Dellaert: Duality theory for SLAM] first incremental SLAM solver [Kaess & Dellaert: isam - incremental smoothing and mapping] 4

5 The Borg Lab at Georgia Tech 5 Structure from Motion, sparse and dense 3D reconstruction [Beall, Dellaert: dense stereo reconstruction]

6 The Borg Lab at Georgia Tech Distributed estimation in multi robot systems [Choudhary, Carlone, Dong, Indelman, Christensen, Dellaert: distributed mapping, active sensing, exploration] 6

7 Outline Ag Sensing 3D reconstruction 4D crop analysis Current work

8 Outline Ag Sensing 3D reconstruction 4D crop analysis Current work

9 motivations World needs 75% more food by 2050 In North America, yield loss approaches 25% of harvest biotic stresses (pests, insects) abiotic stresses (moisture and nutrients imbalance) In the state of Georgia, yield losses exceed a billion dollars a year 9

10 motivations Crop monitoring is mainly performed by human operators: time and cost prohibitive covers small area of field (average field is 642 acres in Georgia) not consistent between scouting intervals must collect leaf and soil samples to identify source of stress 10

11 Use robotics and computer vision to make monitoring and early disease detection automatic and cost-effective. Deploy relatively cheap robots that: operate for extended time quickly cover large areas have repeatable performance our objective 11

12 challenges What sensing is required to detect biotic and abiotic stresses? How do you collect the data (which robots)? How do you interpret the data (which algorithms)? How do you visualize results / provide situational awareness to farm manager? 12

13 Ag Sensing Project 13 NRI: Mul)purpose robo)c pla2orm for field scou)ng and sampling

14 Ag Sensing Project 4D mapping: ground and aerial robots for monitoring and early disease detection using visual data Soil and leaf sampling: robotic arm to collect soil and leaf samples estimate plant geometry (height, crown radius) and appearance evaluate how geometry evolves over time (growth rate, discoloration) autonomous rover design of leaf sampling system (finger tips, visual servoing for grasping, image processing) design of soil sampling system 14 NRI: Mul)purpose robo)c pla2orm for field scou)ng and sampling

15 4D mapping overview Output: interpreted info to operator Input: sensor data 3D reconstruction week 1 15 week 2 Plant segmentation 4D crop analysis week 3

16 Outline Ag Sensing 3D reconstruction 4D crop analysis Current work

17 why 3D reconstruction? from webinar 32 - Dr. Carrick Detweiler: [slide courtesy of Carrick Detweiler] 2D laser: cost ~ 1k USD power ~ 2.5W weight ~ 160g range ~ 5m sparse data 17

18 our sensors PointGrey Chameleon color camera Inertial Measurement Unit (IMU) GPS (APM autopilot) low cost, low power, dense data 18

19 robotic platforms Data collection for entire growing season (planting through harvest) sensor data from tractor twice a week sensor data from UAV once a month AscTec Pelican 19

20 remote data collection remote control of sensors in Atlanta, Georgia data collection in Tifton, Georgia 20

21 3D reconstruction pipeline GPS & IMU data Simultaneous Localization and Mapping (isam2) Dense multiview stereo (PMVS2) images Feature extraction and matching Three main blocks: Feature extraction and matching: extract visual features and correspondences. SLAM (Simultaneous Localization and Mapping): use isam2 to estimate camera poses. Dense multi-view stereo: use PMVS2 to get a dense 3D reconstruction. [Dong, Carlone, Rains, Coolong, Dellaert, 4D Mapping of fields using autonomous ground and aerial vehicles, CSBE/ASABE Joint Meeting, 2014] 21

22 22

23 Feature extraction and matching 1.Image is converted to gray scale, and rectified: rectification 2.SIFT features are extracted 3.FLANN and 8-point RANSAC are used for matching 23

24 t=0 t=1 SLAM t=2 t=.. SLAM: joint estimation of camera poses over time (robot trajectory) and position of 3D features (map) 24 Challenges: 1.integration of fast and slow data streams 2.want very accurate (and fast) estimation 3.large scale estimation 4. noise, outliers

25 SLAM isam2: Maximum a posteriori estimation - compute trajectory and map via nonlinear optimization GTSAM: Georgia Tech Smoothing and Mapping Library 25 [Forster, Carlone, Dellaert, Scaramuzza, IMU Preintegration on Manifold for Efficient Visual-Inertial Maximum-a-Posteriori Estimation, RSS, 2015 (best paper finalist)]

26 SLAM in action Robot trajectory estimation from isam2 versus GPS >1k camera poses >1 million features 26

27 SLAM in action isam2 produces relatively sparse 3D reconstruction: sparse 3D reconstruction of crops during multiple sessions of data acquisition 27

28 Multi-view Stereo Multi-view stereo takes: a set of images camera parameters (intrinsics, poses) and provides a dense 3D reconstruction of the scene Patch-based Multi-view Stereo Software (PMVS2): state of the art implementation of multi-view stereo isam2 PMVS2 [Furukawa, Ponce, Accurate, Dense, and Robust Multi-View Stereopsis PAMI, 32(8), pp , 2010] 28

29 Multi-view Stereo dense point cloud from PMVS2 29

30 3D reconstruction is not enough We need algorithms to interpret large scale data! 30 Point clouds provide great visualization of large fields, but: they are difficult to manipulate for the non-expert it s difficult to extract patterns (e.g., subnormal growth rates, discoloration) by visual inspection

31 Outline Ag Sensing 3D reconstruction 4D crop analysis Current work

32 what is 4D crop analysis? Set of algorithms that takes as input: dense 3D reconstructions (3D snapshots of the crop at different instants of time) 4D crop analysis and produces as output: high-level statistics describing geometry (height, crown radius) and appearance (color) of each plant over time 32

33 4D crop analysis: overview Input: sequence of 3D reconstructions Preprocessing: ground floor extraction clustering 4D analysis: estimate a bounding box for each plant 33 results: evolution of plants height and size over time

34 Canopy segmentation (1/3) Discard points on the ground and allow treating each canopy in isolation divide field in patches (subset of points) fit a local plane to each patch using RANSAC normalize height of points cluster canopies 34

35 Canopy segmentation (2/3) Discard points on the ground and allow treating each canopy in isolation divide field in patches (subset of points) fit a local plane to each patch using RANSAC normalize height of points cluster canopies 35

36 Canopy segmentation (3/3) Discard points on the ground and allow treating each canopy in isolation divide field in patches (subset of points) fit a local plane to each patch using RANSAC normalize height of points cluster canopies Euclidean clustering, available in the Point Cloud Library (PCL) 36

37 Canopy segmentation A useful byproduct: crop height across the field 37

38 Crop analysis Estimate height and crown radius for each plant over time we fit a bounding box to each plant we use Expectation-Maximization (EM) to perform joint inference over the labels (association of 3D points to plants) and bounding box parameters Expectation: associate each 3D point to a bounding box Maximization: find the bounding box that best fits the selected set of points 38

39 4D Crop analysis example [Carlone, Dong, Fenu, Rains, Dellaert, Towards 4D Crop Analysis in Precision Agriculture: Estimating Plant Height and Crown Radius over Time via Expectation-Maximization, ICRA, Workshop on Robotics in Agriculture, 2015]

40 4D Crop analysis results Simulations Field tests in Tifton, Georgia

41 Outline Ag Sensing 3D reconstruction 4D crop analysis Current work

42 Current work improve quality of sensor suite plant segmentation benchmarking against RTK GPS benchmarking against manual measurements vision-based 4D alignment sensor fusion: 4D mapping + soil and leaf samples active sensing Old System New System CPU 1x ARM A8 1.0 GHz 4x ARM A9 2.0 GHz Video Rate 1.0 Hz (fixed) 7.5 Hz (user-defined) Resolution 1280 x x

43 Current work improve quality of sensor suite plant segmentation benchmarking against RTK GPS benchmarking against manual measurements vision-based 4D alignment sensor fusion: 4D mapping + soil and leaf samples active sensing 43

44 44 Preliminary results on plants segmentation

45 Current work improve quality of sensor suite plant segmentation benchmarking against RTK GPS benchmarking against manual measurements vision-based 4D alignment sensor fusion: 4D mapping + soil and leaf samples active sensing 45 GPS RTK-GPS isam2

46 Current work improve quality of sensor suite plant segmentation benchmarking against RTK GPS benchmarking against manual measurements vision-based 4D alignment sensor fusion: 4D mapping + soil and leaf samples active sensing Manual height measurements for benchmarking: added measuring sticks regularly spaced across the field 46

47 Current work improve quality of sensor suite plant segmentation benchmarking against RTK GPS benchmarking against manual measurements vision-based 4D alignment sensor fusion: 4D mapping + soil and leaf samples active sensing 47

48 Current work improve quality of sensor suite plant segmentation benchmarking against RTK GPS benchmarking against manual measurements vision-based 4D alignment sensor fusion: 4D mapping + soil and leaf samples active sensing 48 red: critical areas with slow growth rate

49 Conclusion We described a robotic system that monitors an agricultural field for early disease detection: ground / aerial robot equipped with camera, IMU, and GPS (low cost, low power, high accuracy) We proposed and demonstrated a set of algorithms to: 1. perform 3D reconstruction of the field combination of state-of-the-art methods from robotics and computer vision (isam2, PMVS2) 2. interpret a set of 3D reconstructions collected over time, to provide high-level statistics about the geometry of each plant and its evolution over time (4D crop analysis) preprocessing + Expectation-Maximization 49 Working solution, with minimal manual intervention Many opportunities for improvement Thank you!