Plant and Canopy Reconstruction User Documentation. The University of Nottingham

Transcription

1 Plant and Canopy Reconstruction User Documentation The University of Nottingham 2014

2 Table of Contents Overview... 2 Introduction... 2 Program Input... 3 Point Clouds... 3 Image Sets and Camera Geometries... 5 Directory Structure... 7 Executing the Tool... 8 References Contact

3 Overview Introduction The Centre for Plant Integrative Biology s 3D plant reconstruction tool is a commandline based windows program that can be used to reconstruct surface information in plant canopies. The input of the tool is a 3D point cloud, combined with an input image set and camera geometries. The output is a completed plant mesh in the standard PLY format that can be exported into modelling or graphical applications. The tool is written in C#.NET, and makes extensive use of the.net libraries. For this reason a modern Windows machine is essential. A version of this software for other operating systems is under consideration, but is unlikely for the foreseeable future. Due to the complexity of the data structures used within our software, it is likely that the computer running the reconstruction will require at least 4Gb of available RAM. We would recommend at least 8Gb of available RAM for most reconstructions. 2

4 Program Input Out tool reconstructs plant mesh data from an existing point cloud. Point clouds can be obtained through a variety of means, such as with laser scanning, however in our experiments we have made use of the algorithms presented in [1] and the associated tool PMVS. Along with the point cloud, our software requires a set of images of the plant, or plants, to be reconstructed. The image set must be calibrated, in other words the 3D world position of each camera must be known, or have been calculated before reconstruction can begin. Details on this process can be found below. With all data available, files must be placed in a single location, using a consistent and specific directory structure, also described below. This approach avoids the requirement of manually specifying the location of each file before reconstruction can begin. Point Clouds A point cloud is simply a list of 3-dimensional points that comprise our input data. Various file formats exist to store point clouds, and three formats can be read by the reconstruction tool. Point cloud data can be read in either straight text format, the Stanford PLY format or the PMVS Patch format. The file format used will often depend on the means by which the point cloud was produced, however the choice of format will not affect the operation of our reconstruction tool. Figure 1 provides a sample of each of the three file formats that can be input into the reconstruction software. PLY and Patch files are identified by their header information, the text format should contain no header information. The text format should contain only the X, Y and Z co-ordinates of each point, with spaces used to separate each value. Each point should be output on a single line. The PLY and Patch formats are more verbose, with options to include additional information, such as the normal direction of a point, and colour. This information can be included, but will be ignored by our software as it is not used during reconstruction. During our experiments we have used the PMVS software to construct the initial point clouds. This software outputs clouds as both a PLY and Patch file, they are usually located in the models directory of the PMVS reconstruction. Either is suitable as a starting point for reconstruction using our tool. An optional file named clip.txt can be included inside the patches directory, and will be read if the --plane-filter option is set when the program is run. This file simply contains six numbers, X Y Z U V W, where X Y Z is a point on the clip plane, and U V W is the normal orientation. If the plane filter is applied, any points below the plane will be removed. 3

5 Text Format PLY Format ply format ascii 1.0 element vertex property float x property float y property float z end_header The recognised text format contains no header. Each line contains a single point, given as X Y Z separated by spaces A PLY file contains a header, found between the ply and end_header lines. After this a number of points are listed, one on each line. The number of points must equal the value given by element vertex for the file to be read successfully. Patch Format PATCHES PATCHS clip.txt The patch file is output by the PMVS point cloud reconstruction software. The file begins with a header PATCHES, and a value representing how many points are included in this file. The first line after each PATCHS sub-heading gives the position of each point, other information is included but is ignored The optional clip file gives the position and orientation of a plane that is used to filter points. This is useful for removing large areas of background, such as points that have been reconstructed on the floor below the plant. These points will likely be removed by the colour filter, but the plane filter is faster and more precise in some cases. Figure 1: Examples showing the three available file formats for point cloud input. 4

6 Image Sets and Camera Geometries Our tool requires images of the captured scene, taken from a variety of angles. The number of images required will depend on the scene, but in general the greater the number of images, the better the reconstruction is likely to be. This is because reconstruction of each small section of a leaf surface is best when at least one image has a good view of that leaf. To be specific, we define a good view as taken perpendicular to the leaf surface, and not obscured by other leaves or objects. In our experiments we use more than twenty images per set, however we have experimented with as many as 70. We would encourage users to test their image capture setup and ascertain the optimum number of images for their experiment. Note that the greater the number of images, the greater the memory and computational requirements of the tool. The majority of allocated memory is given to image and related data, thus doubling the number of input images is likely to double the memory requirements of the tool. In order to use the 2D images within the reconstruction process, our software must have access to the geometric information for each camera that was used to capture each image. This can be obtained through camera calibration, when used with a static image capture system, or through so-called structure from motion algorithms that will operate on arbitrarily placed images. If automated calibration is required, we recommend the use of the VisualSFM [2] system for automated camera calibration. This software also utilises PMVS for point cloud reconstruction, so can produce all necessary input files for the reconstruction tool. Users are encouraged to read the documentation supplied with these external tools for more information. Output of VisualSFM is provided in numerous files, our tool reads the detailed data file named cameras_v2.txt, which can be found in the root directory of each VisualSFM reconstruction. This file can be renamed as required, however the contents mut remain unchanged. Where VisualSFM is not the calibration tool used, our tool can read standard text files containing the camera projection matrix and the camera normal (direction of the view). Figure 2 shows examples of these file formats. Please note that our software requires camera data associated with each image, thus the number and order of cameras listed in cameras_v2.txt should match the number and order of images found in the image directory. Alternatively, the number of individual files in text format should match the number of images provided. 5

7 Text Format The directory should contain multiple numerically ordered files, one for each camera. Each file contains three lines for the 3x4 projection matrix for this camera. Details of the derivation of this matrix can be found in our paper, published alongside this tool. VisualSFM Format jpg C:\Directory\images\0004.jpg The VisualSFM camera format contains a number identifying the camera count, followed by a list of parameters for each camera. The parameters included contain all the information necessary to reconstruct the projection matrix. All camera geometry is contained in a single text file, the name is unimportant. Figure 2: Examples showing the two possible file formats for inputting camera geometry into our tool. 6

8 Directory Structure Our tool requires a specific directory structure in which input files are stored, with each folder stored in a working directory that is specified when the program executes. Figure 3 provides an overview of this directory structure. Figure 3: Example of the file structure expected by the tool, containing a working directory and three sub-directories. The working directory can have any name, and is specified when execution begins. Below this are three mandetory directories, named patches, images and cameras. The patches folder contains the point cloud data associated with this reconstruction. Usually this will be a single file, but multiple files can be read if the point cloud is stored in this way. The images folder contains each of the captured images. Specific names for each image are not required, but images are loaded in alphabetical or numerical order. Finally, the cameras folder contains the calibration data for each camera, either in a series of files, or a single VisualSFM file as discussed above. As with the image folder, each camera is loaded in order. This means that the ordering of both images and cameras must be consistent to match each image to the geometry. 7

9 Executing the Tool The tool can be executed from the command line, with additional parameters providing the information required to direct the reconstruction. Users may find it easier to store the entire command in a Windows batch file, if it is to be used or adjusted multiple times. To run the tool with all default settings, the program can be executed as follows: reconstructor.exe path-to-working-directory Most parameters used in the tool, such as the radius to use for point segmentation, provide default values that will work on many datasets. However, it is recommended that these values be customised for a given image capture setup to ensure optimum reconstruction. Options are given after the working directory, when the program is run. For example: reconstructor.exe C:\Reconstruction\ --min-cluster-size 30 Brief details of each optional parameter are given with the command: reconstructor.exe --help Each option can be given as a full command, preceded by a double dash, --. Some options that don t require values can be shortened to a single letter preceded by a dash. A detailed description of each option is provided in Table 1. The optional flags, also shown in table 1, do not require values, it is assumed that a specific flag is true if passed as a parameter. For example: reconstructor.exe C:\Reconstruction\ -fs This command includes both -f -s, and will instruct the software to output both a filtered cloud and a segmented cloud before resuming the reconstruction process. Where such options are chosen, output files will be saved in the working directory, under the sub-directory output. The final output of the reconstruction software will be saved as working-directory/output/triangulation.ply in the PLY file format. 8

10 Option Expected values Default Description --camera-type Either of: VSFM PR VSFM Indicates which of the two accepted formats will be used for the camera calibration data --segmentation-radius A positive real number 0.01 The distance between points below which they will be considered for the same cluster. This value will depend on the scale of the point cloud --min-cluster-size --max-cluster size Any integer greater than zero An integer greater than the minimum cluster size 10 The minimum number of points allowed in a single segmented cluster. Points in clusters smaller than this will be discarded 60 The maximum number of points allowed in a single cluster. Points above this value will be split into other clusters --alpha radius A positive real number 0.01 The alpha value used when creating alpha-shape surface estimates. This number should usually be similar to the segmentation radius, as both are representative of the expected distance between points on the same surface --level-set-iterations Any positive integer 200 The number of level set iterations to run --halting-percentage --zbuffer-resync-frequency Any real number zero or greater Any integer greater than zero 0.0 A value indicating when level sets should halt due to inactivity. If a level set changes size by less than the indicated percentage, it will stop iterating. 20 How many level set iterations to run between resynchronising the z buffer data structures --terminate-after One of: filtering segmentation surface-estimation all all Indicates that processing should stop before the full reconstruction is complete. This is useful for testing earlier stages of the reconstruction, such as segmentation. This should be used with the -f, -s, and -a flags to view output at the appropriate stage -p, --plane-filter - - Indicates whether to apply a planar clipping line to the points before other processing. The position and orientation of this plane must be supplied in clip.txt within the patches folder -c, --colour-filter - - Indicates whether to apply a greenbased colour filter to remove non plant points before surface reconstruction -f, --output-filtered-points - - Indicates whether to output the plane and colour filtered points. The points are saved in ply file format in the software output folder, as filtered.points.ply 9

11 -s, --output-segmented-points - - Indicates whether to output the segmented point cloud, with vertices coloured based on the cluster they are in. The points are saved in ply file format in the software output folder, as segmented.points.ply -a, --output-alpha-triangulation - - Indicates whether to output the initial surface reconstruction based on alpha shapes. The mesh is saved in ply file format in the software output folder, as alpha.triangulation.ply Table 1: Details of all optional parameters and flags that can be passed as command line arguments to reconstructor.exe. The following example shows the full command line execution of the software, using a number of the optional parameters. Those parameters that are left at the default values are not required. reconstructor.exe C:\Reconstructions\Test\ -pcfs --camera-type PR -- max-cluster-size segmentation-radius level-setiterations 100 These options indicate that both the plane and colour filters should be applied, and that the filtered and segmented point clouds should be output. The camera type is set to NP, that is, the normal and projection matrices stored in separate files for each camera. The maximum cluster size is increased to 140, and the segmentation radius is increased to Finally, the number of level set iterations is decreased from the default value of 200, to 100. As the working directory has been set as C:\Reconstructions\Test\, the program will look for the necessary files within: C:\Reconstructions\Test\patches\ C:\Reconstructions\Test\images\ C:\Reconstructions\Test\cameras\ All output files will be saved in: C:\Reconstructions\Test\output\ 10

12 Once execution begins, the console window will provide information on the progress of the reconstruction. Depending on the stage of reconstruction being processed, it will appear much like Figure 4. Reading camera parameters: Done Loading images: Done 40 Reading patches: Done Colour Filter: Done Constructing Search Tree: Done Clustering patches: Done Flattening Clusters: Done Triangulating Clusters: Done Calculating Cluster Visibility: Done Calculating Distance Maps: Done Converting RGB images to NG: Done Syncing Z Buffers: Done Analysing Cluster Histograms: Done Running Level Sets... Iteration 2 Figure 4: Expected output of the reconstruction tool References [1] Furukawa, Yasutaka and Ponce, Jean. Accurate, Dense, and Robust Multi-View Stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 32, Issue 8, Pages , PMVS software available at [2] Wu, Changchang. VisualSFM: A visual structure from motion system VisualSFM software available at Contact Details on the tool and its development can be found on the Centre for Plant Integrative Biology website at 11