The BATCH Program. Andrew A. Maudsley, Ph.D. University of Miami 2005 A. A. Maudsley

Transcription

1 Contents The BATCH Program Andrew A. Maudsley, Ph.D. University of Miami 2005 A. A. Maudsley Introduction... 1 Operation... 1 The Processing Definition File... 3 Error Handling... 4 Reprocessing... 5 Command Line Mode... 6 Acknowledgements... 7 Introduction The BATCH program enables pipelined processing of multiple studies under the MIDAS project. This is written in IDL. Operation Step 1. The application is typically started from the MIDASTools button-bar: Step 2. Subject and Study selection: Click the Study Selection/Browse button to bring up the MIDAS browser (hitting the space bar causes the default action to be taken, which is set to the Browse button on start up). Selection can be made on a Project, Subject, or Study level, where: If selection is made at the Project level, then all Subjects+Studies within that project will be taken. This list is then displayed on the widget. If selection is made at the Subject level and there are multiple Studies, then all Studies will be included. The Browser can be repeatedly used to make multiple selections, and multiple Subjects/Studies can be selected at a single time (using the CTRL or SHIFT keys). Selections from multiple projects can be included; however, the same processing pipeline will be applied to all data (the program can accommodate up to 200 studies). To remove a Study from the list, the user can select that study from the displayed list, and a prompt will be issued asking if that study should be deleted. Step 3. Select the Pipeline Definition file. 1

2 Step 4. If needed, turn On/Off any processing steps. Step 5. Click Start. When the subject and processing file selection is done the widget will expand to list the set of processes. An example of this is shown below: In the list shown on the lower section of this widget are the following columns: Enable if cleared, that process will be skipped. Application the name of the procedure to run, either an IDL command or the program to be spawned. Processing File the name of the file containing any processing parameters. Series The label of the series to be processed. Status This is updated during processing as either Done, Error, or Skipped (if the enable flag is cleared). Additional buttons include: Delete results of previous processing Enabling this will cause the results of a previous processing run to be deleted. See the Reprocessing section below regarding this option. Start will cause the processing to be started. 2

3 Clear All and Set All - Individual steps can be removed or enabled by clicking on the enable radio button at each process, or these buttons can be used to set or clear all steps. Edit Enables editing of the Processing Definition File (see below) The Processing Definition File To start each application the BATCH program spawns a command-line call or issues the equivalent IDL procedure call. All parameters are optional, and the requirement of each call is given in documentation for each application. If multiple parameters are defined, the order of these parameters will always be as shown in the list above. The Processing Definition is a simple TAB delimited text file that can be edited using any text editor, or changed within the BATCH program by clicking the EDIT button. The widget used to edit the definition files is shown here: At the top right can be seen a text entry area, which can be used to add some documentation to the definition file. In the main section the columns are as follows: Application: This can be an IDL command or an OS command to be spawned. For IDL applications the procedure call and the save file must be the same. Currently, the BATCH application has a list of 3

4 known IDL procedures, so if a new IDL command is added then the name must be added into the BATCH program. Processing File: The Proc File for each application. The name can be typed in or selection made using the Browse button. Project, Subject and Study: A 1 in the field means that the corresponding parameter will be passed on the command line, i.e. the XX_Project.xml file; the Subject.xml file; and the Study_ID. Series: The definition file uses the label used to identify the data node in the subject.xml file. The program then finds the corresponding Series_ID, which is passed to the application. All Series must be present when the program is started. Additional String: This defines a single text string passed to the application. If multiple parameters are required these will all be passed as a single parameter that will need to be parsed by the receiving application. The Dependency column: This is used to indicate up to 3 previous processing steps which must have been completed in order for that step to be initiated. So for example, a -1,-2 would mean that the previous step and the one before that must have been successfully run. The Cross and Plus buttons: These delete an entry or add a new line into the pipeline. Error Handling An error in a processes is made known to the BATCH program via an error log file written to disk. This uses the filename convention xxx_errorlog.txt, where xxx is the application name, e.g. FDFT_errorlog.txt. The Batch program tests for the existence of this file and will only proceed on to the next dependent processing step if there were no errors logged. The error log file will list the error, and in some cases the error message will also be printed to the IDL log window. If the processing is repeated when a log file is present, that file will be renamed, e.g. xxx_errorlog_1.txt. Multiprocessing Processes can also be spawned such that the batch program doesn t wait for them to finish. This is done using the qualifier /NOWAIT after the application name, e.g.: 4

5 volumizer/nowait 0 1 none 1 MRI_FLAIR none 0,0,0 volumizer 0 1 none 1 MRI_T1 none 0,0,0 This will run two copies of the volumizer at the same time. There are important limitations of using this option: 1. Errors issued by a spawned application will be missed and will not be indicated on the BATCH program widget. 2. The dependency flag will be ignored if it points to a spawned process. The BATCH program always assumes these completed without error. 3. Multiprocessing can be counterproductive if two or more applications are competing for the same resources. For example, the volumizer processing for the SI and SI_Ref data nodes is almost all disk I/O and running both simultaneously will not result in any speed improvement. 4. Most MIDAS processing steps must be done in a serial manner, e.g. the EPSI and FDFT processing for the SI data both require that these steps were already run for the SI_Ref data node. 5. Issuing multiple NOWAIT processes in a row will likely result in a timeout for access to the subject.xml file. To reduce the impact of a spawned process on the next regularly-scheduled process in the pipeline a delay of 5 seconds is used before starting the spawned process. A recommended strategy for using the NOWAIT command is therefore to interleave these with regularly scheduled processes. Reprocessing When repeating processing steps, it is not always clear whether any previous version of the data and record of the processing in the XML should be removed or not. For example, the FDFT program will always remove any previous entry in the XML (but does not delete the data), since there can only be one SI result per study; however, segmentation will take no action, since the user may wish to compare results of segmentation algorithms. Therefore, when redoing processing, the user may need to delete previous results depending on what needs to be done, and the Delete MIDAS Data and Nodes utility under the Tools section of the toolbar provides a method for doing this. Deciding what to delete or not requires a good understanding of the system; therefore, if any reprocessing is required it is highly recommended to simply delete everything and redo the whole processing pipeline. This can be done in the BATCH program by checking the Delete results of previous processing checkbox, which runs a program at the start of the processing for each subject to delete any previous result. However, note that this function will also delete data that may have been imported using the alternative method from Analyze format files, so it cannot always be used. In this situation the Delete MIDAS Data and Nodes should be used which allows more selectivity. The above option deletes results of all MRI processing, but does not delete all previously processed SI results; it leaves the results for the SI and SI_Ref after the EPSI regridding step. 5

6 This is because the raw multichannel SI data is very large and the normal procedure is to remove the input version of the data at the Volumizer and EPSI2 steps to save disk space; i.e. the Volumizer deletes the raw SI (dicom or raw format files) and creates *.vol files, and the EPSI2 deletes the *.vol files and creates the *.epsi files. Settings in these programs also allow keeping the previous data. Note, for this reason the user should always archive all study data before processing. Processing Multiple SI Series MIDAS supports processing of multiple SI image series acquired in a single study, e.g. if an acquisition is acquired at 2 different TE values. For this situation unique Series labels must be given when the data is imported using the convention of SI/SI_Ref for the first, SI2/SI2_Ref for the second, etc. Up to 4 consecutive acquisitions are supported. For automated batch processing in this situation each of the processing steps needs to be defined for each SI Series, e.g.: Function Label Volumizer SI_Ref Volumizer SI Volumizer SI2_Ref Volumizer SI2 EPSI SI_Ref EPSI SI EPSI SI2_Ref EPSI SI2 etc. The Pipeline File Format The pipeline definition file is a simple text file, using TABS to separate each entry. It can be conveniently edited using any text editor. An example is shown here: ; Created 8/15/2012 ; For TE = 17.6 ms, with Metafit fitting ; volumizer 0 1 none 1 MRI_T1 none 0,0,0 volumizer 0 1 none 1 SI_Ref none 0,0,0 volumizer 0 1 none 1 SI none 0,0,0 SEG_FSL4_T1 0 1 none 1 none -b -3,0,0 epsi2 0 1 EPSI2.xml 1 SI_Ref none -3,0,0 epsi2 0 1 EPSI2.xml 1 SI none -1,-3,0 FDFT 0 1 FDFT_SI_Ref_Spatial.xml 1 SI_Ref none -2,0,0 FDFT 0 1 FDFT_SI_Spatial.xml 1 SI none -2,-1,0 At the top can be seen lines starting with a semicolon and space, which can be used to add documentation within the file. For the command lines, the order of the parameters is as described in the previous section. 6

7 Command Line Mode This program can also be run from the IDL command line mode, as: BATCH, PipelineFile, ProjectXML, SubjectXML, StudyID PipelineFile = the path to the processing definition file ProjectXML = Path to a ***_Project.xml file. SubjectXML = Path to a subject.xml file. StudyID = OPTIONAL StudyID. If not defined, then all studies done if multiple present. In this mode of operation the widget is still displayed, so that progress can be monitored, but the widget is inactive. Typical use for command line operation is as: 1. This can be called with the Subject and StudyID defined, e.g. to start processing after import. 2. To automatically run a set of pipelined procedures. Intended use is to develop an automated test procedure: BATCH PipelineFile1 Project1 Subject1 BATCH PipelineFile2 Project2 Subject2 Acknowledgements This program was written by Andrew Maudsley under the MIDAS project, grant R01EB