Analysis of Off-The-Shelf Stereo Camera System Bumblebee XB3 for the Fruit Volume and Leaf Area Estimation

Transcription

1 Ref: C0265 Analysis of Off-The-Shelf Stereo Camera System Bumblebee XB3 for the Fruit Volume and Leaf Area Estimation Dejan Šeatović, Vincent Meiser, Martin Brand 1 and Daniel Scherly, Institute of Mechatronic Systems (IMS), ZHAW, Technikumstrasse 5, CH-8401 Winterthur, Switzerland Abstract The paper describes a set of simulated and real-world experiments to determine the total leaf area and fruit volume of a fruit tree using 3-dimensional data gathered from colour stereo images. Initially, the accuracy of the stereo algorithms is determined, and then the method for leaf area computation is developed and tested on particular trees with a known leaf area. Finally, leaf area computation is carried out on a number of images of real fruit trees obtained in the ICT-AGRI project 3D-Mosaic. Furthermore, limitations of the system are shown using the example of fruit volume and leaf area computation from orchard data. Several stereo algorithms are compared with regard to their accuracy and performance. The experimental system consisting of several sensors and software components is briefly explained, enabling rebuilding of the system with minimum effort with respect to construction and software development. Lastly, recommendations are proposed for the scientific and industrial teams interested in similar problems on selecting sensors and open-source and BSD software components to successfully approach this complex problem. Keywords: visual perception, sensor evaluation 1 Introduction The Point Grey Bumblebee XB3 is an IEEE-1394b complete stereo camera system that includes software and hardware. The system allows the user, without prior photogrammetric knowledge, to acquire and process stereo images in order to obtain 3D point clouds (Point- GREY, 2014). The main motivation for the purchase and analysis of this system was its relatively low price ($3500) and off-the-shelf processing software and SDK, promising manageable implementation effort. It is intended to be used for the automatic monitoring and analysis of orchard trees, see 3D-Mosaic, (Zude, 2014). In the project, the main goal of the computer vision work package was to develop software that is able to distinguish and localise two main objects on an orchard tree: the leaf and the fruit. The analysis of the camera system succeeded in the lab and was then applied in the field. The goal of the experiments described here was to determine the total leaf area of a tree using 3D data gathered from colour stereo images. For this, the accuracy of the stereo algorithms had to be determined first. Then, the method for the leaf area computation had to be tested on a tree with a known leaf area. Finally, leaf area computation was carried out on a number of images of real fruit trees obtained in Potsdam in August Martin Brand is former employee of Institute of Mechatronic Systems and MSE student. The focus of his master thesis was analysis of dense stereo algorithms and their performance. His research is one of the fundamentals of this paper. Proceedings International Conference of Agricultural Engineering, Zurich, /8

2 Figure 1: Bumblebee XB3 Stereo Camera System. Source: (PointGREY, 2014) The problem can be divided into two independent problems, the first is point cloud matching. Matching of two point clouds requires not only the estimation of the rigid transformation (rotation matrix and translation vector) but also the estimation of the scale factor. The goal of these experiments was to find an algorithm to automatically find these transformation parameters to match the two point clouds obtained from colour and infrared images. The second is the segmentation of 3D point clouds and 2D intensity images. To achieve a robust segmentation between the fruits and leaves it is necessary to obtain the object observation from as many different sensors as are available. The approach is motivated through remote sensing applications such as SPOT and LANDSAT, which allow separation of objects within the intensity images based on their reflectivity in different spectra parts. 2 Accuracy of Stereo Algorithms The analysis of the Triclops (manufacturer SDK) stereo algorithms was abandoned shortly after the camera was purchased and the basic functionality was tested. Unfortunately, the system does not allow the user to perform the calibration of the camera system. The vendor offers the calibration service for the end user which requires that the sensor is sent to the vendor s service point, however due to the time pressure and lack of openness of the system, the decision was made to abdicate the service and proceed with own system evaluation based on OpenCV (G. Bradski, 2000; Gary Bradski & Kaehler, 2008). The decision was supported by the fact that our own Infrared stereo camera system should be used for more reliable distinction between leaves, fruits and other parts of the tree (branches and stem). Dense stereo is a state-of-the-art research topic, for more details see (Tola, Lepetit, & Fua, 2010; Wu & Wang, 2006). 2.1 Description The experiment was designed to serve as a test of the accuracy of the stereo algorithms using the fact that the 3D information of the scene objects is known. The test scene consists of different arrangements of two geometric objects, Blocks and Wall, which are both made of Styrofoam and painted with an irregular colour pattern to facilitate the detection of correspondences in image pairs. Wall is a planar object (dimensions 448 x 448 x 20mm) chosen to verify the planarity of the resulting 3D points. Blocks consists of two cubes (200 x 200 x 200mm and 150 x 150 x 150mm) mounted such that two of their sides become coplanar. This shape provides different combinations of planes to verify the accuracy of plane angles in reconstructed points Stereo Processing To reconstruct the point clouds from the stereo images, two baselines of the Bumblebee camera were used: the short left baseline using the left and the center camera and the long baseline using the left and right camera. Minimum and maximum disparity were chosen as 160 and 255 pixels for the short baseline LC and 327 and 523 pixels for the long baseline LR, corresponding to a distance range of 0.75 to 1.20 m. For Wall one plane was fitted through parts of the image that are visible in the left image of both the short baseline LC and the long baseline LR. For Blocks there were four planes fitted. All Proceedings International Conference of Agricultural Engineering, Zurich, /8

3 implemented stereo algorithms (see axis of abscissas in Figure 4), except for Kosov s algorithm, were tested. Figure 2: Left: "Wall" object with indicated area for plane fitting. Right: "Blocks" object with indicated areas for plane fitting Results The results of this experiment indicate that from the tested algorithms, the two algorithms with the best reconstruction accuracy are the OpenCV implementation of Hirschmüller s algorithm, see (Hirschmueller, 2001; Hirschmuller, Innocent, & Garibaldi, 2002; Hirschmuller, 2008), and the normalized cross correlation (NCC) block matching. This can be inferred from the fact that the reconstructed 3D data contained the least number of outliers and the angles between the fitted planes were reconstructed most accurately by these two algorithms. Figure 3: Reconstructed "Blocks" object. Top row left to right: AD census, OpenCV Hirschmüller. Bottom row left to right: NCC blockmatching, SAD blockmatching. Errors such as outliers and holes are marked in red. 2.2 Leaf Area Computation for a Test Tree For this experiment, a small tree was imaged using a Point Grey Bumblebee XB3 at distances of 1, 2 and 3m at 12 orientations each differing by 30, resulting in a total of 36 images. In order to test the results achieved using the stereo images, all the leaves of the tree were scanned at a resolution of 300 dpi after the stereo images had been acquired. The surface area of the whole tree was then computed by counting all the pixels belonging to a leaf and multiplying this number by the area per pixel. This resulted in a total area of m 2 from a total of 445 leaves. Proceedings International Conference of Agricultural Engineering, Zurich, /8

4 Figure 4: Deviation from 90 angle in [ ] for reconstructed planes on the "Blocks" object for the tested algorithms and two baselines LC and LR Image Pre-processing Since the images obtained with the Bumblebee XB3 tend to be too dark, the images used in both experiments have been pre-processed in the following way to improve the results obtained by the stereo algorithms: 1. Convert the RGB image to HSV (Hue, Saturation, Value) 2. Change Saturation 3. Change Value 4. Convert back to RGB Figure 5: Test tree used in experiment. Left: Original image, Right: Pre-processed image Stereo Processing To reconstruct the point clouds from the stereo images, two baselines of the Bumblebee camera were used: the short baseline using the left and the center camera and the long baseline using the left and right camera. All implemented algorithms, except for Kosov s algorithm and the OpenCV implementation of SAD block matching, were tested. A full list of parameters and their description can be found in the appendix of Martin Brand s Master Thesis. Proceedings International Conference of Agricultural Engineering, Zurich, /8

5 2.2.3 Segmentation In this step all pixels belonging to leaves were selected to act as a mask limiting the leaf area calculation to the region of interest. For a pixel to belong to a leaf, the following criteria had to be met: Pixels had to be within a manually defined rectangle., for RGB values, with, for the hue value to select green pixels Leaf Area Computation In a first step, the range image (the z-data) was filtered using a 3x3 median filter and the x- and y-data was then reconstructed from the changed z-data using the rectification information for the respective baseline (LC or LR). In the next step, a Delaunay triangulation was carried out on the segmented data with the constraint that the 2D distance in pixels must be below and the 3D distance must be below 3cm for two points to form one side of a triangle. On the triangulated mesh, the sum of all individual areas from all triangles was computed as the leaf area of the tree, where and are two vectors of a single triangle Results The experiment was designed to infer a consistent factor to estimate the complete leaf area of a tree from one stereo image pair. A straight forward assumption would be to expect to see 50% of the leaf area in one view. The algorithm with the best performance over all distances is Hirschmüller s algorithm, which has consistent values for both the leaf area as well as the standard deviation at distances of 2 and 3m for both baselines LC and LR, as can be seen in Figure 6. Figure 6: Results for test tree leaf area using Hirschmüller's algorithm When looking at the average percentage of the computed leaf area in relation to the scanned tree over all 12 orientations of the images, a clear effect of the pre-processing step on the amount of leaf area can be seen for most algorithms. This effect is visible for all distances and both baselines. However, not only is the computed leaf area larger but so is the standard deviation across the images. It can also be seen that the closer the camera is located to the tree, the lower the leaf area becomes. This could be attributed to the relative increase in size of the leaves when moving closer, leading to a more sparse disparity image. Increasing the window size for the algorithms could help here but that would also introduce errors at leaf Proceedings International Conference of Agricultural Engineering, Zurich, /8

6 edges. Using the longer baseline instead of the shorter one also results in a lower computed leaf area. It was also found that the AD-census algorithm needs much smaller parameters for the building of the cross based regions (see appendix A.1 of Martin Brand's master thesis) since the leaves of the tree all have a very similar colour, leading to excessive smoothing. The zero mean (Z***) variants of the algorithms as well as RANK and SSD block-matching lead to the lowest values for the leaf area while NCC and SAD block-matching, Hirschmüller s algorithm and AD-census consistently result in larger leaf areas. The results of OpenCV s block-matching algorithm are not included, since the input images are only grayscale and no selection of green pixels has taken place other than the manually selected rectangular region, leading to incorrect results. 2.3 Leaf Area Computation for Fruit Trees in an Orchard The data for this experiment was collected in Potsdam, Germany on the 28th and 29th of August, 2012 in an Orchard consisting of 180 trees arranged in 6 rows of 30 trees. Each tree was imaged from both sides, denoted as A1 and A2 respectively. The camera used was a Point Grey Bumblebee BBX3 stereo camera. The images were pre-processed in the same manner as in the previous experiment with the test tree Stereo Processing Based on the results obtained in the two previous experiments, only the short baseline LC was used in this experiment. This choice was made since the longer baseline introduced too much distortion at the distances used in this application. Also, only normalized cross correlation (NCC) block-matching and the OpenCV implementation of Hirschmüller s algorithm were chosen since these performed best in the previous experiments regarding reconstruction accuracy and consistency in leaf area computation. The following table lists the parameters for the two chosen algorithms which were used in this experiment. The disparity range of 4 to 95 pixels corresponds to a distance range of 2 to 50m. Table 1: Stereo Processing Parameters Parameters NCC block-matching OpenCV Hirschmüller Minimum Disparity 4 4 Maximum Disparity Window Size 7x max-diff pre-filter cap uniqueness ratio speckle window size speckle range full DP - true Segmentation As in the previous experiment, only pixels that belonged to leaves were considered. For a pixel to belong to a leaf, the following criteria had to be met:, to eliminate points belonging to objects further in the background. This is also the reason why the disparity range during stereo processing was chosen to include pixels as far away as 50 m., for RGB values, with W = 250 being a threshold to eliminate bright pixels belonging predominantly to the sky or specular reflections. Also, only connected components of a size were selected in this step. Proceedings International Conference of Agricultural Engineering, Zurich, /8

7 , for RGB values, with K = 40 being a threshold to eliminate pixels belonging predominantly to the black cloth in the background. Using the white and the black pixels as mask, edges in that mask were used to find lines using the Hough transform. This was done because the black cloth behind the tree has long and straight edges which separate the background behind the cloth from the tree in front of it. Lines that had a length and were within +/- 10 of being either horizontal or vertical were used to eliminate pixels on the outside of the background cloth., for RGB values, with, for the hue value to select green pixels Leaf Area Computation The computation of the leaf area was done in the same manner as in the previous experiment, except for the use of the different pixel selection criteria mentioned above to find pixels belonging to a leaf Results The leaf areas were computed for both views for all 180 trees of which only two had ground truth data of their real leaf area available. The assumption made in the previous experiment with the test tree, that 50 % of the leaf area would be visible in one view, could not be confirmed in this experiment. The actual percentage is actually lower. Reasons for this are: Dynamic range of images is low, since the sensors don t allow more sophisticated adjustment; amount of false matches is greater than expected 10% (in average). This leads to errors in the segmentation and stereo processing. Several trees were not automatically distinguishable, so their leaf areas were not accurately determined at all. Values were manually corrected. 2.4 Colour and Infrared Stereo Point Clouds These point clouds were generated from images captured in Potsdam in August 2012, using stereo algorithms implemented and tested during the course of the 3D-Mosaic project. The sources were colour images from a Point Grey Bumblebee XB3 and two Point Grey Grasshopper 2 cameras with infrared filters to block out visible light. Figure 7: Input data set obtained from measurement in Potsdam. The infrared point cloud has been coloured orange for a better contrast. 3 Conclusions The Bumblebee XB3 as stereo camera system has several advantages when compared to high-end systems, such as price, and also when compared to systems assembled using offthe-shelf cameras. Unfortunately, the system is closed to the user and important parameters cannot be accessed (camera calibration). Furthermore, due to its Bayer pattern sensor, the system delivers only a fraction of the spectra and therefore stereo algorithms discover many similar pixels, which lead to very noisy point clouds, see Figure 7. At the end of the project, Proceedings International Conference of Agricultural Engineering, Zurich, /8

8 the implementation effort exceeded the estimated time by a factor of three. Due to the data quality problems, the final goal of matching the point clouds was not fully achieved. While the synthetic point cloud data test was successful, the real world data obtained from the images captured could not be matched by any of the algorithms. A bad signal-to-noise ratio also prevented further systematic algorithm analysis. Results obtained during the work on field data here give a coarse overview of the procedure, rather than deliver hard facts. However, one can rate the approach and results as success, since higher quality sensors will allow more accurate data processing. Additional research is necessary to finally answer the question if point clouds produced by stereo cameras are sufficient for leaf area estimation and fruit detection in orchards. 4 Acknowledgements The authors would like to thank the ATB Potsdam, and especially Dr. Manuela Zude for the patience and understanding during the 3D-Mosaic project. We also want to thank Dr. Thomas Anken for almost a decade of cooperation and support in the matter of the application of visual perception in agricultural applications. 5 References Bradski, G. (2000). The OpenCV Library. Dr. Dobb s Journal of Software Tools. Bradski, G., & Kaehler, A. (2008). Learning OpenCV: Computer Vision with the OpenCV Library. O Reilly Media. Hirschmueller, H. (2001). Improvements in Real-Time Correlation-Based Stereo Vision. In SMBV01. Hirschmuller, H. (2008). Stereo Processing by Semiglobal Matching and Mutual Information. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 30(2), doi: /tpami Hirschmuller, H., Innocent, P. R., & Garibaldi, J. (2002). Real-Time Correlation-Based Stereo Vision with Reduced Border Errors. IJCV, 47(1-3), PointGREY. (2014). Point Grey - Stereo Vision Products - Triclops SDK. Retrieved June 2, 2014, from Tola, E., Lepetit, V., & Fua, P. (2010). DAISY: An Efficient Dense Descriptor Applied to Wide- Baseline Stereo. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 32(5), doi: /tpami Wu, C.-C., & Wang, Z.-F. (2006). Stereo Correspondence Using Stripe Adjacency Graph. In Pattern Recognition, ICPR th International Conference on (Vol. 1, pp ). doi: /icpr Zude, M. (2014). 3D-Mosaic. Retrieved April 2, 2014, from Proceedings International Conference of Agricultural Engineering, Zurich, /8