PHOTOGRAMMETRIC METHOD:

Transcription

1 Before to start, we suggest you to read our article Chiari Y, Wang B, Rushmeier H, Caccone A. Using digital images to reconstruct three-dimensional biological forms: a new tool for morphological studies. Biological Journal of the Linnean Society (2008). To know how to obtain the pictures and to calibrate the camera to reconstruct the object in 3D read our paper and check the tutorial of the PhotoModeler software. Make sure to not change any camera parameters between taking the images of the object and of the calibration target. Make sure also that the object do not occupy the entire image in the picture. This could create problems due to distortion. Before to start (see below for each method additional explanations to see what you would need): The pipeline components run on Windows XP and also on Vista. In this latter case, choose to run Cygwin in compatibility for Windows XP to avoid any problem. Software to purchase: PhotoModeler (Eos System Inc., SRI Small Vision System (SVS) software (Videre Design Inc., Software to download for free: Cygwin ( Install Cygwin using the Dos option for ASCII files. Make sure that you install the whole package of Cygwin (check the different icons to be sure of that). When you install Cygwin, also ImageMagick is automatically installed. BS Contact Stereo (Bitmanagement, the demo version is enough for our purpose). This software is actually not mentioned below, because it is not a necessary step in the pipeline, but it can be used to view the models (either partial or final). Rapidform Basis (INUS Technology, Any image processing software (e.g., PaintShop, Photoshop, Corel Draw). The pipeline available on this website. Copy all files under Pipeline to the search path of Cygwin. PHOTOGRAMMETRIC METHOD: You just need to use PhotoModeler and follow the tutorial to process the images and obtain the 3D model. No need to purchase or download any other software. The technical support of PhotoModeler is of a great help in case of problems. STEREO VISION METHOD:

2 No need to purchase PhotoModeler and process the images with it. Just process them with the SRI Small Vision System to obtain the partial 3D models. Because to align the partial models you need to use component of the pipeline, the pipeline available on this website needs to be downloaded. The stereo vision method runs using Cygwin. To run the stereo vision method, follow the instructions as indicated for the hybrid method below. HYBRID METHOD: Before to start: If you want to process your images using this method, all your pictures need to be organized as sequence pairs (as you would obtain from a linked camera pair, see our article for additional information). No need to give a special name to the pictures, as long as you have a left and a right image for each picture pair (e.g., name_1_l and name_1_r, etc.). This has to be done for both, the pictures of the object you want to reconstruct as well as for the camera calibration pictures. Create a folder called FolderName (which means you can give it the name you want) and create under that a folder called Photomodeler. The pictures pairs of the object need to be organized in a folder called RawData under the Photomodeler folder, while the pictures pairs of the calibration will be organized in a folder called Calibration, also under the Photomodeler folder. You will then have -FolderName/Photomodeler/RawData -FolderName/Photomodeler/Calibration How to start: Every step of the pipeline is run on a Cygwin window that needs to be open to start the processing (and leave it open even while you are working with the SVS software, since you will need Cygwin at different stages). If you are familiar with the command line in Cygwin, you can skip the beginning of this section and move to the following steps directly. Otherwise, to access the data folder where the RawData and Calibration folders are type on the Cygwin window cd c: (in this way you go to the directory c ) dir (or ls) (in this way you can see all the files and sub-directories) and proceed typing cd or dir depending of the path you need to reach the directory where the folder with the data (images) is. When you reach the directory with the command dir, with cd FolderName you enter in the folder that contains the folder called Photomodeler where the RawData and Calibration folders are. You have to be in this FolderName folder to move to the next step.

3 First step- resizing: A restriction of the SRI Small Vision Software (SVS) is that it works with images of limited number of total pixels. Thus, all images need first to be down-sampled to a maximum of 800x600 pixel resolution. NOTE: reducing the pixel resolution, will decrease the resolution of the images, so we suggest you to not reduce them below the necessary size. The file SVPM_SuperScriptNoTexture.sh included in the pipeline folder with all the other components contains basic info relative to the following steps of the pipeline. You can open this file with a text editor. TYPE in the Cygwin window the codes written below to start the pipeline. NOTE: the codes are different depending if you run the hybrid or the stereo vision methods: FOR THE HYBRID METHOD SVPM_SuperScriptNoTexture.sh $1(name) $2(resize percentage) FOR THE STEREO VISION METHOD SV_SuperScriptNoTexture.sh Where $1name corresponds to your FolderName with the pictures inside organized in Photomodeler/RawData or Photomodeler/Calibration (as written above) and $2resize percentage is the percentage of which you want to resize your pictures to process them. For both, the hybrid and the stereo vision methods, you need to resize the pictures. Type 50 or any other parameters that would reduce the images size to maximum 800x600. E.g. SV_SuperScriptNoTexture.sh Turtle PhotoModeler processing (if you process your images using the stereo vision method, you can skip this step and go directly to the stereo vision processing part below): Open the PhotoModeler program and use it to calibrate the camera and create a Photomodeler calibration project first. In case you have the calibration file ready, you do not have to do it again (so, you can just process the object pictures). Make sure you use the RESIZED.jpg files to do the calibration. Once you are done with the calibration (or you have done it before), come back to the Cygwin window and press ENTER. Use PhotoModeler to reconstruct the model following the software instructions. Make sure to use ALL the RESIZED.jpg data images. Export the camera configuration file from PhotoModeler as below:

4 In PhotoModeler, click Project -> Table -> Photo Table and select 'Camera All'. Then press 'Export', and choose 'Tabs' instead of 'Commas' and 'Full precision'. Click 'Export' to export the configuration file. Keep the default file name 'TableExport.txt'." Camera All is a configuration that you have to build the first time you use the program. From Photo Table press Configure (call the new file Camera All) and put in this order: #Photo Description Center x Center y Center z Omega Phi Kappa Image width Image height Roll angle 35 min Focal length Format width Format height Principal point x Principal point y K1 K2 K3 P1 P2 Fov H Fov W Largest residual RMS residual In a similar way, from PhotoModeler export feature points file as below: Go to 'Project-> Table -> Points Table' and select 'Point'-> 'Export', and choose 'Tabs' instead of 'Commas', press 'Export' to export the points file. Keep the default file name 'PointExport.txt'.

5 As for the Camera All, it is necessary to build the Point. From Project -> Points Table -> Configure (call the new configuration Point ) and put in this order: ID X(m) Y(m) Z(m) When you are done with the reconstruction and exporting the configuration as explained above, come back to the Cygwin window and press ENTER. Press ENTER again to confirm that you have TableExport.txt and PointExport.txt files ready. Stereo Vision processing: the first step is converting from PhotoModeler configuration files to SVS calibration files. For each image pair taken (we faked a virtual camera rig for each pair of images taken), a group of calibration images for SVS will be created by the converting program. Those calibration images will be used in the SVS software to calibrate the virtual camera rigs. In the SVPM_SuperScriptNoTexture.sh file are indicated as $MIN to $MAX, which indicates that you need to run the calibration for each image pair. NOTE: $MIN and $MAX indicate how many pairs of object images you have. If, for example, you have 10 pairs of object images, $MIN = 1 and $MAX = 10. For each pair of object images, there will be 9 pairs of calibration images. In this method, you need to create one particular faked camera rig for each pair of object images, so you will then have 10 faked camera rigs in this case and they need to be calibrated one by one. For calibrating each of them, you then need 9 calibration image pairs. The calibration images will be called as $1-A-B-C, here A stands for the index of object image pairs, in this case from 1 to 10; B stands for the index of calibration image pairs, from 1 to 9; C stands for Left or Right. In case you are using the stereo vision method, then you can use the calibration images obtained for the stereo vision processing and resized in the first step of the pipeline. Hybrid and stereo vision methods at this step share the same code. This step will be automatically executed and smallvcal.exe started after the user presses ENTER at the end of the previous section. Run SV_smallvcal.exe (SVS) and press 'Calibrate' in the menu; then 'Load all' to load all calibration pairs and click on one of the image pairs. You need to repeat this process for each image pair. E.g. you need to click $1_1_cal1. Then you do the calibration as written below. When you are done with it, you need to redo the same process for $1_2_cal1 etc, until you have done it for each image pair ($MIN-$MAX, see paragraph above for the explanation of why this). Click 'Features' on the bottom of the SVS window to find all feature points on every image. Delete all image pairs in which features are not successfully found. Click 'Custom' [case sensitive here] and change

6 'Imager Width/Imager Height' (in pixel) to the *CORRECT* value (after the resize, to know what this correct value is, just check the size of a few pictures, they have to be all the same size). The values are just the width and height of the resized images. Select 'Fixed Aspect', 'Kappa2', 'Kappa3', 'Tau1', 'Tau2'; uncheck 'Zero Disparity'. If you find the rectification image is too undistorted, please uncheck the 'Kappa3. Then select 'custom' [case sensitive here] and change 'Sq Size' to 30, '#Sq X' to 9 and '#Sq Y' to 7." Finally click 'Calibrate' to iteratively solve the calibration model. If successful (it means that the calibration results are displayed in the Debugging window. The fx/fy value for the left and right camera should be similar to each other), save calibration results to $1-*.ini; otherwise, choose different subset calibration image pairs to try again. * refer to the photo number (between the MIN-MAX of SVPM_SuperScriptNoTexture.sh, so for image pair 1, *=1). Remember to repeat this step for each image pair. When you are done with the calibration, go to the Cygwin window and press ENTER and press ENTER again to confirm that you have the $1-$MIN.ini to $1-$MAX.ini files ready. Then, for each stereo rig from $MIN to $MAX, the next step involves the stereo matching (reconstructing the portion of the specimen that is visible in each image of the pair): Run SV_smallv640.exe. NOTE: if you start to run the pipeline typing in the Cygwin window the command SVPM_SuperScriptNoTexture.sh (or SV_SuperScriptNoTexture.sh for the stereo vision method) then the program will open automatically at this step, otherwise you just need to double click on it to open it. Under 'File', click 'load parameters' ($1-*.ini) (* is from $MIN to $MAX). Under 'File', click 'load images'. Click on one of the $1-*-L/R/C/Q.bmp to start. These are the object images and are in the FolderName ($1) made at the beginning. Then, press 'Freeze' and Select function ->'Stereo' and then 'Multi-scale' and change 'Conf', 'Unique', 'Disparity' and 'Window' until a continuous, smooth surface occurs in the right window. Playing with these parameters you will see the image becoming more or less definite, the texture more or less cloudy and you will be able to recover more or less surface of the object. Under 'File', click 'batch store', give a name as $1-* (where * is from $MIN to $MAX) and the following files will be saved automatically: $1-*-R-L/R/C/Q.bmp; $1-*-R-D.bmp; $1-*.3pi; $1-*.p.wrl; $1-*.cam. NOTE: for traditional stereo vision method, all the camera files ($1-*.cam files) are the same; for the hybrid method they are different. The camera files record the camera information in a format which is understandable by the following part of the pipeline. In stereo vision method, there is only one camera rig (one image pair generated by the pictures taken with the linked camera), so all the.cam files are the same; in hybrid method, there are multiple virtual camera rigs, and each of them has a corresponding.cam file. In

7 case that for some stereo rig (image pair), we cannot get good results as smooth surface after the proceeding described above, we can just discard that stereo rig. When you are done with the previous part, come back to the Cygwin window and press ENTER and press ENTER again to confirm that you have all the above files ready. The part using the SVS software is finished. You can close the SVS software. The pipeline will then automatically (without the need of the user typing anything) convert 8bit disparity images (these are temporary files used by the stereo vision pipeline and the hybrid pipeline.) to 24bit bmp file. You can see the few commands ran by the pipeline in the Cygwin window. Final 3D reconstruction: at this point you need to interactively clean up diag files (*-Diag.tif). This can be done with any image processor software. Cleaning the images means that you need to remove parts on the model that are not clear, the background in the images and remain at the end with a part of the object that you want to reconstruct that has high resolution and it is without missing part on its texture. Save the cleaned file as $1-*-R-Diag-Cleaned.tif to the FileName folder (the same one made at the beginning of the process, $1). When you are done with the cleanup work, go to the Cygwin window and press ENTER. Press ENTER again to confirm that you have all the -Diag-Cleaned.tif files. Each one of these files should have a partial model or a part of the object that you want to reconstruct (eventually, you can check these partial models using BS Contact Stereo). The pipeline will then run a few commands to generate texture mapped triangle meshes from 3pi files (these are temporary files recording the geometry of the partial models) and cleaned diag images. Registration: this step is to align the partials models. Note there is little difference in the hybrid pipeline and the stereo vision pipeline. In the hybrid pipeline, the SVPM_GlobalRegister.exe will then run automatically after the last step of the previous paragraph, and for each partial model, mark (click on) landmarks on the partial model. Open PhotoModeler to see the correspondences of these points. Basically to align the partial model, you need to combine them together and this is done marking the landmark and ordering them as they are in PhotoModeler. At least 6 points (6 landmarks) on each partial model need to be marked/identified to decrease the possibility of error in the 3D reconstruction. In the stereo vision pipeline, the SV_GlobalRegistration.exe will then run automatically after the last step of the previous paragraph. The processing is similar to the hybrid pipeline. The only difference is that the user needs to buildup a virtual model since there is not in this case the Photomodeler model on which to refer to click the correspondences of the point (see above-hybrid method part). A rough drawing made on a piece of paper will be fine (e.g. the one in Figure 1 in our BJLS paper). The feature points in the virtual model/drawing do not have to be the physical landmarks; they can be anything which can be easily identified in the partial

8 models. The drawing does not have to be precise in any means, it is only for indexing the landmarks used in the registration. For each partial model, user manually marks as many feature points as he can. NOTE: before to move to the next step and do the registration, if on some of the partial model files (.wrl) you cannot find enough landmarks (6 or more), then you can go to FolderName and delete the $1.globalregistration with not enough landmarks. After that, you need to re-do the process of marking/identifying the landmarks and then process to the next step, the Global registration. Register the models globally by clicking 'Register' in the popup menu. You can view the registration results by clicking 'Show registration results'. To export the $1.aln file, click 'Save registration results'. NOTE: at this stage DO NOT turn off the GlobalRegister.exe. The pipeline will automatically run a code to finish the global registration. Check the $1.GlobalRegistration files and make sure that each partial model has at least 6 landmark points available, then go to the Cygwin window and press ENTER to automatically run another code of the pipeline. When you are done with the global registration, go back to the Cygwin window and press ENTER. Press ENTER again to confirm that you have the $1.aln file ready. Because the user cannot clean the models perfectly, a few other commands will be run to further clean the 3D reconstruction. Colorization (this step add color and texture information to the final model; otherwise there will be only the 3D form): Use GlobalRegister.exe (the window is still open from before) to colorize the merged models, clicking 'Colorize Merged Model' in the popup menu. When you are done with GlobalRegister.exe, exit the Program and press ENTER in the Cygwin window to finish the 3D reconstruction. You are now done with all the process and the final 3D model has been saved as $1-withnotexture.wrl and $1-withnotexture.ply" Scaling: Every method presented here needs the scaling step, because the camera parameters computed are not absolute values. You can scale the object before to export it from PhotoModeler, following the software tutorial and using for example a formula as the one indicated in our article to calculate the scaling value. In the Cygwin window TYPE: SVPM_Scale.exe $1.wrl $1-scaled.wrl Scaling value Where $1 is again the FileName with the wrl extension (the final 3D model obtained with the pipeline). The $1-scaled.wrl is the name of the scaled 3D model and Scaling value is the scaling factor computed

9 with the formula in our paper (so instead of typing scaling value you just need to write there the corresponding scaling value number). NOTE: The above mentioned steps do not need to run in a sequence. User can decide to stop at different stage. If you stop for example at the end of the PhotoModeler processing of the hybrid methods, the next time that you want to continue to run the pipeline, you just need to open the Cygwin window, be in the FolderName and start the pipeline TYPING the command: SVPM_AfterPhotomodeler.sh NOTE: This webpage is kept updated and renewed. For any technical problem or general question about the methodologies, their possible applications and the content of our paper, please contact Ylenia Chiari (yle at yleniachiari dot it) and Bing Wang (bing dot wang at yale dot edu).