ZOOPROCESS / PLANKTON IDENTIFIER PROTOCOL For COMPUTER ASSISTED ZOOPLANKTON SORTING

Transcription

1 ZOOPROCESS / PLANKTON IDENTIFIER PROTOCOL For COMPUTER ASSISTED ZOOPLANKTON SORTING Picheral, Garcia-Comas, Elineau, Romagnan, Desnos 2015/01/28 More info available in: J. Plankton Res. Gorsky et al. 32 (3): 285. Content 1 INTRODUCTION DIAGRAMS Prediction and validation in same categories using an existing learning set (LS) Prediction and validation with additional detailed categories using an existing LS Creation of an initial Learning Set, Prediction and Validation Prediction and Validation, Learning Set Improvement RAPID DESCRIPTION OF MAIN STEPS GLOSSARY PREDICTION of IDENTIFICATION VALIDATION of IDENTIFICATIONS CREATE AN INITIAL LEARNING SET GENERAL PROCEDURE: vignettes are in sample or profile (>UVP5) folders PREVIOUS PROCEDURE: vignettes in sample/profile folders (ZOOSCAN only) 14 6 EVALUATION OF A LEARNING SET USE SUBSET TOOLS TO CREATE A LEARNING SET USE SUBSET TOOLS TO CHECK VALIDATION (Quality Control) EXTRACT AGAIN VIGNETTES FROM VALIDATED FILES

2 1 INTRODUCTION COMPUTER ASSISTED ZOOPLANKTON SORTING formally named semi-or automatic recognition is a mean to ease and accelerate the sorting of organisms. Computers and algorithms are only a help, and the final decision of sorting is made by humans. We describe below the general strategy to perform computer assisted sorting of plankton using the Zooprocess/PKId/XnView tools. This method is applicable for all sorts of images acquired with diverse instruments (ZooSCAN, FlowCAM, UVP5, ISIIS, microscope) and processed with Zooprocess. 2 DIAGRAMS We present below the main workflows that may be useful according to the diverse needs Prediction and validation in same categories using an existing learning set (LS) Figure 1: Daily routine computer assisted sorting of plankton diagram. This diagram depicts the simplest protocol to obtain sorted vignettes and identification files from images treated with Zooprocess. It assumes that the person performing computer assisted sorting of plankton uses an existing learning set (LS), and validate plankton vignettes in the predicted categories only. Letters A to K refer to the procedure main steps described in

3 2.1.2 Prediction and validation with additional detailed categories using an existing LS Figure 2: Daily routine computer assisted sorting of plankton diagram with expert validation. Same protocol as in Fig.1 and identification is made by a human. It assumes that computer assisted sorting of plankton is performed using an existing LS and validation of plankton vignettes in more detailed/numerous categories than the predicted ones. Letters A to K refer to the procedure main steps described in 2.2. This workflow is the one that most people routinely use Creation of an initial Learning Set, Prediction and Validation Figure 3: Computer assisted sorting of plankton diagram with creation of a new LS, and validation steps. Same protocol as in Fig.1 and Fig. 2 from the creation of the LS to the validation into detailed categories. It assumes 3

4 that computer assisted sorting of plankton is performed using a newly created LS. Validation steps are the same as described in Fig. 1 and 2. Letters A to K refer to the procedure main steps described in Prediction and Validation, Learning Set Improvement Figure 4: Computer assisted sorting of plankton diagram with all options in the same graph (creation of a new LS, validation, and quality control of validation using the subset tool). This diagram depicts the protocol to obtain sorted vignettes and identification files from images treated with zooprocess. It depicts the complete procedure from the creation of new LS to quality control of validations and the improvement of the LS subsequent to quality control and includes all options. It assumes that the technician or scientist performing computer assisted sorting of plankton do not use an existing LS, and create a new LS. Validation steps are the same as described in Fig. 1 and 2. Letters A to K refer to the procedure main steps described in RAPID DESCRIPTION OF MAIN STEPS A) First, check if you can use existing LS that applies on the samples you scanned. Coarse LS (or LS from another project/location) can be used to perform an initial prediction on a few samples. The resulting identifications (Ids) should then be validated and used to create a new, more accurate, LS. LS can be downloaded from the ZooSCAN website. 4

5 B) If you do not have any suitable LS, you need to create an initial simple LS to predict a few categories (< 10) containing ~50 or more objects per category. This operation should take less than half a working day including LS performances evaluation. C) Predict the Ids for a few scanned samples (< 5). D) Extract vignettes according to the prediction to visualize the prediction outputs. E) Add categories (detailed sorting) to the predicted categories, or not, to add detail to validate the prediction with more taxonomical resolution than the prediction. F) Validate the predicted Ids by sorting the vignettes using PKId or XNVIEW image browser, in the predicted categories only. G) Validate the predicted Ids by sorting the vignettes using XNVIEW image browser, in the predicted categories plus the additional categories. H) Print/update Ids from the validated vignettes in the data files (adding Ids to measurements in the rows of the PID file). I) According to the prediction real performances and the validation effort (% of Ids corrected and personal evaluation), you may consider the LS and predictions to be satisfactory or not. In the first case, go back to step C) to predict another set of samples. In the second case, carry on improving the LS using the set of validated vignettes. J) Improve the LS using the set of validated vignettes and create other LS from it using the dedicated Zooprocess tool ("automatic" LS). K) Improve the automatic LS by keeping only the better items (the ones that best correspond morphologically to the category, in each category). Perform some performance test (i.e. Cross Validation) to evaluate the newly improved LS. 2.3 GLOSSARY Accuracy: The proportion of the total number of classified objects that is correctly classified Category: A taxon or group of taxa used in the learning set and confusion matrix Classifier: A supervised learning algorithm applied to automated classification of objects. Classifiers are developed from a suite of characteristics extracted from each object. Confusion matrix (CM): A matrix illustrating both predicted (from the classifier) and true classifications of all object categories. Contamination: See false positive rate. dat1.txt file: The PID file completed with the predicted and validated categories, if a validation has been performed. 5

6 Error rate: Proportion of mispredicted organisms (to be manually corrected in order to obtain a fully validated data set). False positive rate: The proportion of objects that is incorrectly classified as belonging to a category of interest; also called contamination. Identification(s): name given to an object through prediction or validation; also referred to as ID, IDs, Ids Learning set: A set of vignettes of organisms sorted in categories by an expert and used in a supervised learning model; also called training set or LS. Log file: A text file containing details concerning the analyzed sample image.pid file: A data file resulting from image analysis by ZooProcess. Includes the LOG file above the data section. Each identified object occupies one row, with all the variables extracted from that object in columns. Plankton Identifier: Software for automatic recognition of plankton; also referred as PkID. Precision: The proportion of predicted positive objects that was correctly assigned. Prediction: automatic recognition of plankton using a supervised learning model, PkID, a LS and.pid files. Recall: see true positive rate True positive rate: The proportion of objects that is correctly classified as belonging to a category of interest; also called recall. Validation: Manual sorting of vignettes to the correct category, following initial automated classification of vignettes Variable: Attributes extracted from every detected object (see the list of extracted variables in Appendix 3). Vignette: An image of a single detected object; also called ROI (region of interest) ZooProcess: Software for image acquisition, treatment and analysis built for the ZooScan system 3 PREDICTION of IDENTIFICATION Predict identifications of planktonic objects in a set of samples using a LS. Do not forget to refer to Plankton Identifier users guide to learn to build a LS and a learning file. PLANKTON IDENTIFIER (Fig. 5): 1. Launch Plankton Identifier (PkID). 2. Click on the "Prediction" or "Data analysis" tab (middle) 6

7 3. Browse to the LS file folder in the top left window "select a learning file". Learning files should always be stored in: \Zooscan_project_name\PID_process\Learning_set. 4. Chose a learning file (Learn*.pid). 5. In the middle left window ("select sample file(s) ), browse to your unpredicted sample files (*.pid) folder. Note that processed sample files are automatically copied and stored by Zooprocess in: \Zooscan_ project_name\\pid_process\pid_results. 6. In the middle left window, select the samples PID files to be predicted (not more than PID files at a time to prevent computing memory problems). 7. In the "Original variables" window (second left), check the variables that will be used to classify the objects (un-tick position variables, see Zooprocess manual). 8. At this point, you can create customized variables in the second right window ("customized variables"). See PkID user guide. 9. In the "Identification Group" window (right) do NOT UNTICK ANY category (bug). The names represent all the categories present in your learning file, and they will all be predicted. 10. In the "Select a Method" tab (bottom left), select the "SpvLearning4 (Random forest)" method. 11. Enable save detailed results for each sample below the "Select a Method" tab (bottom left). 12. Start Analysis by clicking on "run" or "start analysis" tab. You MUST the results of the prediction in \Zooscan_project_name\PID_process\Prediction as *dat1.txt files. NOTE: One output file is saved for each sample (*dat1.txt). These files contain the following information: a header of processing info and sample metadata (identical as in the *.pid file), a data table consisting in the measurements derived from the image analysis (identical as in the *.pid file), with an extra column containing the prediction (identification) of each object. It is recommended to add an Analysis_YYYYMMDD_HHMM prefix to the analysis result files to keep track of the date of the prediction in the resulting filenames. 7

8 8

9 Figure 5: first caption: PkID prediction window with steps to follow to make a prediction from a Learning file and sample files ( 3). Second caption: PkID prediction window. Note that the "original variables" is scrolled down to show the variables which should be selected. 4 VALIDATION of IDENTIFICATIONS Each prediction should be checked and validated or corrected by a taxonomist to get data appropriate for a scientific purpose. ZOOPROCESS: Run the tool: EXTRACT vignettes in folders according to PREDICTION or VALIDATION for Zooscan or SORT vignettes in folders according to PREDICTION or VALIDATION for other instruments (Zooprocess main menu, first window). Select the option: "use NEW PREDICTION Files from Prediction folder". Keep the default options. If necessary, change the resolution to the scanning resolution. The contrast of the vignettes can be enhanced or reduced by adjusting the Gamma but 1.1 default value is standard for the Zooscan. Select the method to be used by Zooprocess to extract vignettes (random/range ) and the associated parameters. XnVIEW: Use XnView to sort vignettes in the appropriate categories (folder) if needed (validation/correction). At this stage, it is possible to: Sort vignettes in the predicted categories only (Fig. 1) Sort vignettes in additional categories (Fig. 2) ZOOPROCESS: Run the tool: LOAD identifications from sorted vignettes for Zooscan or Update Identifications in dat1 files after sorting in folders for other instruments (Zooprocess main menu, first window). Keep the default options except if you wish to get validations for many samples at a time. This operation is intended to print the validated/corrected identifications by the taxonomist in the result files. At the end of the update process, Zooprocess will tell you the percentage of vignettes that you resorted during validation (including misclassified vignettes and vignettes resorted in new finer groups defined by the expert). This information helps decide whether or not 9

10 improve the LS. A 20-30% error rate is common for diverse datasets (>25 predicted categories). NOTES: The vignettes of the predicted objects are extracted in the \Zooscan_ project_name \PID_process\sorted_vignettes\sample_name_YYYYMMDD_HHMM_to_validate folder and sorted in subfolders according to the prediction results. The prediction output files (sample_name_dat1.txt) are stored in \Zooscan_project_name\PID_process\Prediction as *dat1.txt files and automatically copied in \Zooscan_ project_name \PID_process\Pid_results\Dat1_extracted folder. By default, empty folders are also created using the ID text list from the config folder, in each sorted vignettes folder ("sample_name_yyyymmdd_hhmm_to_validate"). It is recommended to print new categories in these files if you want to complete (detailed sorting) the list of categories. Extracted sample_name_dat1.txt files in prediction folder are renamed to sample_name_dat1.ext to prevent multiple extraction of the same sample vignettes. For users of Zooprocess versions earlier than 7.15 : DO NOT FORGET TO REMOVE ALL EXTRACTED FILES FROM PREDICTION FOLDER before using version 7.15 and upper! To ease the validation/correction of samples, we recommend the use of XnView (free software) instead of Windows Explorer. XnView allow moving many vignettes in one selection without memory problems. Launch XnView and browse to the folder that you want to validate (i.e., sample_name_yyyymmdd_hhmm_to_validate). Go through each of the subfolders (category) and check the vignettes. If some vignettes are misclassified (errors due to automatic classification) or can be classified in a more accurate category, move them to the right folder. When the validation/correction is done for all the categories, you must change the sample folder name from *_to_validate into *_validated manually. The output, updated, sample_name_dat1.txt files are copied in \Zooscan_project_name \PID_process\Pid_results\Dat1_validated. These final tables have a new extra column which contains the expert taxonomist classification of each object. 10

11 5 CREATE AN INITIAL LEARNING SET NOTES: We strongly recommend to spend very short time (~half a day) on this task, and to preserve working time to validate all your samples after automatic identification (prediction). We recommend naming the new learning file as Learning_YYYYMMDD_HHMM to keep track of the creation date. Note that this file is a concatenation of all the variables from the PID files of vignettes used to build it, and that a last column has been added with the Id of each object (see PkID users guide) GENERAL PROCEDURE: vignettes are in sample or profile (>UVP5) folders For Zooscan operations: two alternatives are possible to create a LS from scratch. 1. Use the vignettes present in the sample folders in \Zooscan_project_name \Zooscan_scan\_Work 2. Use the Extract vignettes for Plankton Identifier (Zooprocess ADVANCED mode only) option. See below ZOOPROCESS Switch to ADVANCED mode to be able to use the necessary Zooprocess tool (Zooprocess main menu). WINDOWS EXPLORER Copy the instrument_liste_ident.txt file(s) from the \Zooscan_project_name \Zooscan_config\ folder into the program_files\plancton_identifier\liste\ folder (instrument refers to Zooscan, UVP5, FlowCam, Généric). Create a subfolder in the \Zooscan_project_name\PID_process\Learning_set folder named YYYYMMDD_HHMM_initial. PLANKTON IDENTIFIER 1. Launch PkID and click on the "Learning" tab for version > 1.3 or click on the "Learn" tab for version < 1.3. Browse to the empty folder just created using 11

12 Windows Explorer ("YYYYMMDD_HHMM_initial"). Select it and click the "ok" button in PkID window. 2. Click on the folders icon up on the right to create categories (i.e., categories as copepoda, cladocera) using the "instrument_liste_ident.txt". 4. In Unsorted thumbs (top left window) select a sample/profile folder from which you will find specimens for each category Figure 6: PkID Learning menu window 12

13 Figure 7: PkID Learning working window with list creation pop-up 5. In the working window, select vignettes of specimens that you drag into the category folders (top right window) see figure 8. You can select multiple vignettes but avoid getting more than Redo the operation using vignettes from another sample/profile. 7. Once you have finished sorting the vignettes, click on the create learning file button (bottom right). Save it in the folder created at the start of the procedure: \Zooscan_project_name\PID_process\Learning_set\YYYYMMDD_HHMM_ initial. 13

14 Figure 8: PkID Learning working window with specimen sorting into folders 8. When you get the popup JOB COMPLETED, continue sorting?,click NO to end PREVIOUS PROCEDURE: vignettes in sample/profile folders (ZOOSCAN only) In the previous Zooprocess versions (before 7.18), most people did not extract their vignettes during the process and had to get vignettes from their samples. We describe here the adapted procedure to get these vignettes. ZOOPROCESS Switch to ADVANCED mode to be able to use the necessary Zooprocess tool (Zooprocess main menu). WINDOWS EXPLORER Copy the instrument_liste_ident.txt file(s) from the \Zooscan_project_name \Zooscan_config\ folder into the program_files\plancton_identifier\liste\ folder (instrument refers to Zooscan, UVP5, FlowCam, Généric). Create a subfolder in the \Zooscan project_name\pid_process\learning_set folder named YYYYMMDD_HHMM_initial 14

15 Open \Zooscan_ project_name \PID_process\Pid_results\ and COPY the sample_name.pid files you want to use to build your Learning set (a small representative subset of your whole set) in \Zooscan_project_name \PID_process\Pid_results\Unsorted_vignettes_pid\. ZOOPROCESS Choose the menu Extract vignettes for Plankton Identifier (ADVANCED mode only). Keep default options. If necessary change the resolution to the scanning resolution. The contrast of the vignettes can be enhanced or reduced by adjusting the Gamma but 1.1 default value is standard. Select the method to be used by Zooprocess to extract vignettes (random/range ) and the associated parameters. PLANKTON IDENTIFIER See Fig.6-8 and steps 1-8 in EVALUATION OF A LEARNING SET PLANKTON IDENTIFIER To EVALUATE a LEARNING SET before using it to predict Ids of organisms a. Click on "Data Analysis" tab if you use PkID versions < 1.3. If you use PkID v1.3 and more, click on "Evaluation" tab. b. On the learning set box, you select as Learning file your LS (Fig. 5 step 3). c. Select the method Cross-validation4 (Rndm tree) (Fig. 5 step 10) (bottom left). The method consists in splitting randomly the learning file into 2 half. By default, one half is used as a learning set, and the other one as a known set of object, to test the learning set. By default, this operation is repeated 5 times to obtain robust statistical values. d. Click on Start Analysis or "run". Include in the name of the analysis the date to keep track. Save results in \Zooscan_project_name\PID_process\Prediction. e. After analysis is completed, quit data analysis. Click on "Show report" (PkID v < 1.3) to visualize the results of the Cross Validation. f. Select the Analysis_name.txt you have created. Your web browser will open. 15

16 g. Click on cross-validation. You can see true classification (rows) versus automatic classification (columns). Recall should be above 70% for all groups, and contamination (1-Precision) below 20% to keep a category. 7 USE SUBSET TOOLS TO CREATE A LEARNING SET If you consider that you will save time in future validation tasks on the project by creating a new LS, we describe below a simple procedure to accelerate this operation. This operation should not last more than two hours. ZOOPROCESS Switch to ADVANCED mode. Select the Zooprocess tool: CREATE subset of a Learning Set from identified vignettes (random). The tool will randomly subsample vignettes from validated samples (only the ones renamed validated by the user) and place them in a new subfolder of the LEARNING_SET directory named YYYYMMDD_HHMM_random_N (Naming is automatic and N is the chosen number/ratio of vignettes). Consider ending with no more than ~ 300 vignettes for each category in your final LS. Option of using fixed number will over select low abundances categories, percent is more representative of the dominant categories that should be kept in a LS. XNVIEW Open this LS in PkID to clean the new learning set : steps 1 and a. Remove/delete folders/categories containing very low numbers of organisms/vignettes. b. Remove/delete organisms/vignettes that are misidentified. c. Remove/delete organisms/vignettes that are not morphologically or taxonomically representative of what should be predicted. Try as much as possible to end up with equally represented categories (approximately the same number of objects/vignettes/organisms in each category/folder) PLANKTON IDENTIFIER Test the new LS using cross-validation as described above ( 6). 16

17 As you moved some vignettes and probably deleted others, your LS will no longer be fully balanced. If you want to homogenise this new LS with the same number of organisms in each category, just run again the same Zooprocess tool selecting now the random LS (option 1))that you just created instead of the default Sorted_vignettes folder and adjust the number of vignettes to be selected in each category. There is no real need to have fully balanced LS. Experienced users will find their ways to optimize the prediction and decide themselves the relative proportions of the categories while a balanced LS functioning is easier to understand. 8 USE SUBSET TOOLS TO CHECK VALIDATION (Quality Control) Once validation/correction is done for a subsequent part of a project, one may need to check the quality of validation, and estimate the post-validation identification error rate. As imaging tools generate large number of vignettes to check, there is a need for estimating identification errors on a subset of the total amount of vignettes. The tools and methods presented in this section are intended to ease the process of quality control on identifications. ZOOPROCESS Switch to ADVANCED mode. Select the Zooprocess tool: CREATE subset of a Learning Set from identified vignettes (random). Selection of extraction options a. ONGOING VALIDATION (part of the whole set of vignettes): Opt for selecting a fixed number of objects in each category, in each sample. This option actually over select categories with low abundances with respect to categories with high abundances. It randomly subsamples vignettes from validated samples (only the ones renamed validated by the user) and place them in a subfolder of the Learning_set directory: YYYYMMDD_HHMM_random_N (N is the chosen number of vignettes). b. FINAL EVALUATION (whole set of vignettes): Opt for selecting a percentage of object in each category, in each sample. This option allow the 17

18 extraction of a set of vignette which is representative of the whole set of vignettes. Also select option 4) to get random vignettes from all DAT1.txt files. It randomly subsamples vignettes from validated samples (only the ones renamed validated by the user) and place them in a subfolder of the Learning_set directory: YYYYMMDD_HHMM_random_N (N is the chosen % of vignettes). A Learning file is automatically created by Zooprocess in the subset folder. WINDOWS EXPLORER OPTION: Make a backup copy of the set of vignettes extracted in the same subset directory (or ZIP the subset folder). XNVIEW Check the identifications and validate/correct the possible misidentification by sorting the misidentified vignettes with XnView (see 4) in YYYYMMDD_HHMM _random_n. PLANTON IDENTIFIER Create another Learning file using PkID (see 5) from YYYYMMDD_HHMM _random_n_copy and name it "YYYYMMDD_HHMM_ random_n_corrected.pid". EXCEL Matlab R Compare the 2 learning file: in excel or any program able to open *.pid files and make operations on them, open the 2 files and compare the last columns (identifications) to get the % of differences between them. 9 EXTRACT AGAIN VIGNETTES FROM VALIDATED FILES This operation is recommended to homogenize a project when the validation is finished, checked and identifications printed in result files (*dat1.txt in Dat1_validated folder, see 4, NOTES). It is also useful when a project not completed has been left in stand by for a long time. In this case it enables the operator to clean the "Sorted_vignettes" folder and restart the prediction and validation process from a well-organized set of files and vignettes. ZOOPROCESS: 18

19 Switch to ADVANCED mode. Run the tool: LOAD identifications from sorted vignettes for Zooscan or Update Identifications in dat1 files after sorting in folders for other instruments (Zooprocess main menu, first window). Select the option Load Ids from all subfolders to get validations for all samples at a time. This operation is intended to print the validated/corrected identifications by the taxonomist in the result files. WINDOWS EXPLORER Erase the content of the Sorted vignettes folder. ZOOPROCESS: Run the tool: EXTRACT vignettes in folders according to PREDICTION or VALIDATION for Zooscan or SORT vignettes in folders according to PREDICTION or VALIDATION. Select the option: "use VALIDATED files from Dat1_validated folder". Keep default options. If necessary change the resolution to the scanning resolution. The contrast of the vignettes can be enhanced or reduced by adjusting the Gamma but 1.1 default value is standard. Select the method to be used by Zooprocess to extract vignettes (random/range ) and the associated parameters. 19