Basic User's Guide on how to set up Ostrich for Model Calibration and Optimization

Transcription

1 Basic User's Guide on how to set up Ostrich for Model Calibration and Optimization Muluneh A. Mekonnen (Global Institute for Water Security, University of Saskatchewan) February 2, Introduction OSTRICH (Optimization Software Toolkit for Research Involving Computational Heuristics) is developed by Prof L. Shawn Matott (previously assistant professor at the Department of Civil and Environmental Engineering, University of Waterloo and now computational scientist at the Center for Computational Research, University at Bualo). OSTRICH has been created as a model-independent multi-algorithm optimization and parameter estimation program that allows researchers and eld-practitioners to automate the processes of model calibration and design optimization. Being model-independent, OSTRICH can be congured to operate with any modelling program that utilizes text-based input and output le formats. This basic user's guide provides a working knowledge of using Ostrich to calibrate or evaluate a model that uses text-based input and output le formats such as the MESH modelling system. For further details of the Ostrich program and the available optimization algorithms readers can refer to the Ostrich website ( 2 Minimum requirements to setup model calibration (or model evaluation) job using Ostrich The following three tools should be available to set up model calibration or model evaluation job using Ostrich: ˆ Ostrich executable - the executable for Ostrich is available for dierent operating systems and can be downloaded from the Ostrich home page, ˆ Model executable - this is the model to be calibrated or evaluated, ˆ Batch les - these are les to automate and guide the model calibration process, and ˆ Executable to compute the value of the objective function to be optimized (optional) - this executable is required if the model executable doesn't have an internal objective function algorithm (to compute the value for the objective function) or if the user needs to use his own objective function algorithm. How does it work? Remember that Ostrich can be congured to operate with any modelling program that utilizes text-based input and output le formats. This means that every time the parameter value changes then the textbased input le through which the parameter value will be read by the model should be modied. So, there are two basic components in the Ostrich program: 1) algorithms for searching parameter values, and 2)a text search and replace algorithm that updates the input les based on the generated parameter values. Being familiar with the second component of Ostrich is sucient to set up model calibration as well as model evaluation tasks. So, this user's guide will mainly clarify the second component of Ostrich. 1

2 .1 Where does Ostrich get all the required information? For every model calibration (model evaluation) setup the Ostrich program gets all the required information from its own input le called 'ostin.txt'. Below is a sample 'ostin.txt' le for the purpose of demonstration. Note that lines that start with the pound character (#) are comment lines and are ignored by the Ostrich program. 1 # S e c t i o n 1 2 ProgramType DDS 4 # S e c t i o n 2 5 ModelExecutable. / model_executable 7 # S e c t i o n 8 B e g i n F i l e P a i r s 9 input1. t p l ; input1. txt 10 EndFilePairs # S e c t i o n 4 1 BeginParams 14 #name i n i t. lower upper t r a n s f o r m a t i o n s format 15 par1 random none none none F11. 1 par2 random none none none F EndParams # S e c t i o n 5 20 BeginInitParams EndInitParams 2 24 # S e c t i o n 25 BeginResponseVars 2 #name f i l e n a m e keyword l i n e c o l token 27 SAE function_ out. txt ; OST_NULL 0 1 ' ' 28 EndResponseVars 29 0 # S e c t i o n 7 1 BeginGCOP 2 CostFunction SAE PenaltyFunction APM 4 EndGCOP 5 # S e c t i o n 8 7 BeginDDSAlg 8 PerturbationValue MaxIterations UseInitialParamValues 41 EndDDSAlg A closer look at the above le provides a clear picture on how Ostrich works. Here follows the section by section explanation: ˆ Optimization algorithm: At the beginning of the le the 'ProgramType' keyword is used to specify the optimization algorithm. In this case DDS (Dynamically Dimensioned Search algorithm (Tolson and Shoemaker, 2007)) is selected. The DDS options such as maximum iterations are provided at the end of the le using appropriate keywords (refer to the Ostrich manual for the dierent keywords available for the Ostrich Program). ˆ Model: Then the 'ModelExecutable' keyword is used to specify the model to be calibrated. This can be the model executable itself or a batch le which will in turn run the model. In this case the executable (or the batch le) is assumed to exist in the same directory as the 'ostin.txt' le. If it resides in a separate directory then the complete path should be provided. ˆ Changing input les: The third section enclosed by 'BeginFilePairs' and 'EndFilePairs' where the model input le (s) whose parameter values are supposed to change (and their corresponding template les) will be specied. In this case parameter values in one of the model input les 'input1.txt' will be changing. The user needs to prepare a template le input1.tpl which is exactly the same as input1.txt except that keywords (to be discussed in the following section) are specied in the template le at places where parameter values are required to change. ˆ keywords for parameters: The fourth section enclosed by 'BeginParams' and 'EndParams' is where keywords are assigned for the parameters to be changed. In this section the parameter ranges 2

3 are specied as well as the format based on which the values are to be printed at the respective locations of the keywords. The parameter keywords specied in this section should exist in the template les. Below is a sample le for input1.tpl le that is consistent with the above 'ostin.txt' specication par par ˆ Initial parameter values: The fth section contains user provided initial values enclosed by 'BeginInitParams' and 'EndInitParams' keywords. If Ostrich uses these initial values then the rst content of 'input1.txt' le produced from the template le will appear as follows: ˆ Value for objective function: Sections through to 7 are such that the value of the objective function that will be minimized is going to be read from the le 'function_out.txt'. With the 'ostin.txt' le, model input, model executable and the Ostrich executable les in place then double clicking the Ostrich executable should start the model calibration or model evaluation process. However, for practical applications and to facilitate saving of dierent outputs the Ostrich run can be wrap up through a batch le (like Ostrich.bat as is being used for the MESH modelling system) and other extra batch les (like save_best.bat and OST-MESH.bat les which are commonly being used for the MESH modelling system). In this case double clicking the 'Ostrich.bat' batch le will start the calibration/evaluation process. 4 Ostrich setup for the MESH model - for Windows This section provides a quick and basic guidance to setup Ostrich for the MESH model calibration or evaluation. Users can download all the necessary les and a test case from the MESH wiki site (under the MESH users category and under 'Calibration using Ostrich' sub-category). The setup can easily be adapted to any specic needs. The les used here are taken from the test case. Note that this guidance and Ostrich setup for the MESH model assumes that the MESH executable, the MESH input les and the MESH forcing les together with the 'ost' sub-directory reside under the same directory. And also the 'ost' sub-directory is assumed to contain: ˆ the 'ostin.txt' le, ˆ the template les, and ˆ the various batch les and the Ostrich executables (Ostrich.exe and OstrichMonitor.exe).

4 4.1 Description of required les First we need to have a working MESH model setup, meaning that we need to have a directory where MESH input les, forcing les and the MESH executable. Make sure that the MESH model successfully runs in that directory. Then we need to have an additional 'ost' directory that will contain the main Ostrich input le 'ostin.txt' (refer to the description given earlier in this document), template les (to be explained in the following section), the Ostrich executable and various batch les (also to be explained below). 4.2 Template les Template les are exact copies of the input les where we want parameter values to change except we put keywords at places where we want the parameter values to change. Refer to the template les included in the test case that you downloaded from the MESH wiki site. We have two template les under the 'ost' directory: ˆ MESH_parameters_CLASS.tpl - the following keywords are used: 1 xslp1 2 wfci1 xslp2 4 wfci2 ˆ MESH_parameters_hydrology.tpl - the following keyword is used: 1 _WF_R2 The 'ostin.txt' le should contain and provide the limits and formats of the above keywords as described below. 4. ostin.txt le 1 ProgramType DDS 2 #ProgramType ModelEvaluation ObjectiveFunction GCOP 4 ModelExecutable OST MESH. bat 5 PreserveBestModel save_best. bat 7 B e g i n F i l e P a i r s 8 MESH_parameters_CLASS. t p l ; MESH_parameters_CLASS. i n i 9 MESH_parameters_hydrology. t p l ; MESH_parameters_hydrology. i n i 10 EndFilePairs BeginParams 1 #name i n i t. lower upper t r a n s f o r m a t i o n s format 14 xslp1 random none none none E wfci1 random none none none E xslp2 random none none none E wfci2 random none none none E _WF_R2 random none none none F. 19 EndParams BeginInitParams E E 4. 0E E EndInitParams BeginResponseVars 2 #name f i l e n a m e keyword l i n e c o l token 27 SAE function_ out. txt ; OST_NULL 0 1 ', ' 28 EndResponseVars 29 0 BeginGCOP 1 CostFunction SAE 2 PenaltyFunction APM EndGCOP 4 5 BeginDDSAlg PerturbationValue MaxIterations UseInitialParamValues 9 EndDDSAlg 4

5 As shown in the 'ostin.txt' le it is clear to see that: ˆ the DDS program will be used for calibration (top) with the specied DDS parameters (bottom), ˆ the objective function will be handled by the GCOP algorithm (refer to the Ostrich manual for further details) based on the SAE value to be read from 'function_out.txt' le, ˆ the OST-MESH.bat le will run the MESH model, ˆ the save_best.bat le will save user specied les for the best runs, ˆ two template les (MESH_parameters_CLASS.tpl and MESH_parameters_hydrology.tpl) will produce the corresponding 'ini' les by changing the indicated keywords, and ˆ parameter values will be constrained by the minimum and maximum bounds and the values will be written according to the specied format (Note that for users information the format of all the parameters in each le is shown at the end of every line). 4.4 Ostrich.bat le As shown below the 'Ostrich.bat' le creates an output directory, launches the Ostrich monitor (facilitates visualization of Ostrich runs) and the Ostrich executable and nally moves output les to the output directory. o f f Ostrich Optimizaton Software Toolkit 4 REM Create s u b d i r e c t o r y f o r Ostrich output 5 SET DIR_OUTPUT=OUTPUT IF NOT EXIST %DIR_OUTPUT% mkdir %DIR_OUTPUT% 7 8 REM Delete p r e v i o u s l y generated output f i l e s 9 IF EXIST OstDone. txt DEL /Q OstDone. txt 10 IF EXIST OstQuit. txt DEL /Q OstQuit. txt 11 IF EXIST OstErrors0. txt DEL /Q OstErrors0. txt 12 IF EXIST OstOutput0. txt DEL /Q OstOutput0. txt 1 IF EXIST OstModel0. txt DEL /Q OstModel0. txt 14 IF EXIST OstExeOut. txt DEL /Q OstExeOut. txt 15 IF EXIST OstStatus0. txt DEL /Q OstStatus0. txt 1 IF EXIST OstSaveOut. txt DEL /Q OstSaveOut. txt 17 IF EXIST OstGcopOut. txt DEL /Q OstGcopOut. txt REM Launch the Ostrich Monitoring program 20 START "OSTRICH Monitor " OstrichMonitor. exe REM Launch Ostrich, wait u n t i l i t completes 2 START /WAIT "OSTRICH" /MIN Ostrich. exe 24 echo Done! > OstDone. txt 25 2 REM Wait a few seconds to allow monitor to d e t e c t 27 REM completion and c l o s e down 28 ping n 2 w 000 > nul 29 0 REM Copy Ostrich output i n t o i t s own s u b d i r e c t o r y 1 IF EXIST OstDone. txt move OstDone. txt %DIR_OUTPUT%\OstDone. txt 2 IF EXIST OstQuit. txt move OstQuit. txt %DIR_OUTPUT%\OstQuit. txt IF EXIST OstErrors0. txt move OstErrors0. txt %DIR_OUTPUT%\OstErrors0. txt 4 IF EXIST OstOutput0. txt move OstOutput0. txt %DIR_OUTPUT%\OstOutput0. txt 5 IF EXIST OstModel0. txt move OstModel0. txt %DIR_OUTPUT%\OstModel0. txt IF EXIST OstExeOut. txt move OstExeOut. txt %DIR_OUTPUT%\OstExeOut. txt 7 IF EXIST OstStatus0. txt move OstStatus0. txt %DIR_OUTPUT%\OstStatus0. txt 8 IF EXIST OstSaveOut. txt move OstSaveOut. txt %DIR_OUTPUT%\OstSaveOut. txt 9 IF EXIST OstGcopOut. txt move OstGcopOut. txt %DIR_OUTPUT%\OstGcopOut. txt 4.5 OST-MESH.bat le The 'OST-MESH.bat' le changes the directory one level up, copies the modied 'ini' le, runs the MESH model, copies the 'function_out.txt' to the 'ost' directory and nally changes the directory back to the 'ost' directory. 5

6 o f f Driver program f o r MESH C a l i b r a t i o n 4 REM MESH f i l e s are up one d i r e c t o r y 5 cd.. 7 REM F i l e name f o r the MESH e x e c u t a b l e 8 s e t MeshExecutable=1 00. exe 9 10 REM Bad value f o r model crash case 11 s e t BAD=1.0 e REM Delete p r e v i o u s l y generated f i l e f o r o b j e c t i v e f u n c t i o n value 14 d e l /q function_ out. txt 15 1 REM Copy input f i l e s c r e a t e d by Ostrich 17 copy o s t \MESH_parameters_CLASS. i n i. \ MESH_parameters_CLASS. i n i 18 copy o s t \MESH_parameters_hydrology. i n i. \ MESH_parameters_hydrology. i n i REM Run MESH model 21.\% MeshExecutable% 22 2 REM Compute the o b j e c t i v e f u n c t i o n. In t h i s case the MESH output 24 REM ' function_ out. txt ' i s used. Assign a bad value to the 25 REM ' function_out. txt ' f i l e i f MESH c r a s h e s. 2 i f not e x i s t function_ out. txt echo %BAD% > function_ out. txt 27 copy function_ out. txt o s t \ function_ out. txt REM Return to o s t r i c h working d i r e c t o r y 0 cd.. \ o s t 4. save_best.bat le The 'save_best.bat' le saves les of user interest. o f f SAVE BEST SOLUTION 4 REM Create s u b d i r e c t o r y to save the best s o l u t i o n 5 i f not e x i s t best mkdir best 7 REM Copy the f i l e s f o r the best s o l u t i o n to the best 8 REM s u b d i r e c t o r y 9 copy.. \ MESH_parameters_CLASS. i n i best \MESH_parameters_CLASS. i n i 10 copy.. \ MESH_parameters_hydrology. i n i best \MESH_parameters_hydrology. i n i 11 copy.. \ BASINAVG1\MESH_output_echo_print. txt best \MESH_output_echo_print. txt 12 copy.. \ BASINAVG1\MESH_output_streamflow. csv best \MESH_output_streamflow. csv 1 echo done! 4.7 Ostrich outputs There are a number of Ostrich output les. The author of this manual has the experience of looking the 'OstExeOut.txt' le to make sure that the MESH model is being run properly by the Ostrich program. And most importantly if there are errors (besides checking the 'OstErrors0.txt' le) the then this le provides useful information as to where to look for and correct the errors. The other le is the 'OstModel0.txt' le where all the parameter values and the corresponding objective function values are saved. 4.8 Setting MESH model calibration using Ostrich The easiest way of setting the MESH model for calibration using Ostrich is to: ˆ download the 'Ostrich_setup_and_test_case.zip' folder from the above mentioned MESH wiki site, ˆ run the test case (by double clicking the 'Ostrich.bat' le under the 'ost' directory) as is and get yourself familiarized with the Ostrich outputs, ˆ replace the input and forcing les with your own les;

7 ˆ prepare template les of your interest and modify the 'ostin.txt' accordingly, and ˆ again double click the 'Ostrich.bat' le under the 'ost' directory (Note that if you want the value of the objective function to be calculated by MESH then you need to set the 'AUTOCALIBRA- TIONFLAG' in the 'MESH_input_run_options.ini' le to 1. This is the sum of absolute value of errors between observed and simulated stream ow), and ˆ if everything works as expected then setup the model calibration based on parameters of your interest. References Bryan A. Tolson and Christine A. Shoemaker. Dynamically dimensioned search algorithm for computationally ecient watershed model calibration. Water Resources Research, 4,