Multi-level hierarchic Markov processes

Transcription

1 Software Users Guide: Multi-level hierarchic Markov processes Anders Ringgaard Kristensen Dina Notat No. 84 September 1999 Fourth Edition, August 2010

2 .

3 Software Users Guide: Multi-level hierarchic Markov processes Anders Ringgaard Kristensen Dina Notat No. 84 September 1999 Fourth Edition, August 2010 This note is also available on www at URL: Department of Large Animal Sciences Faculty of Life Sciences, University of Copenhagen Grønnegårdsvej 2, DK-1870 Frederiksberg C

4 .

5 . Preface This fourth manual is an updated version of manuals previously published as Dina Notat No. 84. It describes version 2.0 of the MLHMP software. The most recent version of the software system may at any time be downloaded from the home page of MLHMP at KU LIFE, August 2010 Anders Ringgaard Kristensen

6 Dina Notat No. 84, Third Edition

7 Contents 1 General information Purpose and facilities Language and platforms Installation Microsoft Windows Other platforms Using the program Starting the program Windows The Process tree window The optimization log window The results window The Limids window Loading models The Open option The Import option Saving models The Save and Save as options The Export option Edit model properties Plug-ins What is a plug-in? The plug-in list Installing and removing plug-ins

8 Dina Notat No. 84, Third Edition Building models Optimization Criterion of optimality Optimization Value iteration The menu bar File New Open Close Save Save as Print log Print tree Import ASCII Export ASCII Exit Edit Model properties Installed plug-ins Run Optimization Probability check Value iteration Simulation Functions Convert distributions to arrays - keep deterministic Convert distributions to arrays (incl. deterministic) Redefine reward Redefine output Reset reward and output to standard Collapse this hierarchic model

9 Software Users Guide: Multi-level hierarchic Markov processes Write founder transition matrix to disk Window Process tree Optimization log Results Help About Menus installed by plug-ins The choice bar Criterion of optimality Edit (Labels only or Parameter values) Button bars Button bar of the Process tree window New button Open button Save button Print button Structure button Add button Remove button Add LIMID stages or ordinary stages button Build button Check button Run button Button bar of the Optimization log window New button Open button Save button Print button Check button Run button Button bar of the results window

10 Dina Notat No. 84, Third Edition The comma-save button Print button The present/relative value button The future profitability button (no icon) Current policy Check button Run button

11 Chapter 1 General information 1.1 Purpose and facilities The purpose of the MLHMP software is to demonstrate the general concepts, parameter needs and structure of multi-level hierarchic Markov processes. Furthermore it provides a general graphical framework for definition and optimization of such models. Since bi-level hierarchic processes as well as ordinary Markov decision processes are just special cases, the software also supports such models. The idea behind multi-level hierarchic Markov processes is partly to allow simultaneous optimization of decisions on multiple time scales and partly to circumvent the well known curse of dimensionality in Markov decision processes. The notion of a multi-level hierarchic Markov process is described by Kristensen and Jørgensen (2000). For an introduction to ordinary Markov decision processes and bi-level hierarchic Markov processes applied in herd management applications, reference is made to Kristensen (1993). For a general presentation of the software it is recommended to read the article by Kristensen (2003). 1.2 Language and platforms The MLHMP software has been programmed entirely in Java. It runs under any platform supporting the Java Runtime Environment (version 1.6 or higher). In order to run the software, either the Java Runtime Environment (JRE) or the Java Development Kit (JDK) must be installed. The appearance of the graphical user interface will differ slightly depending on the platform. In this manual, the appearance under Microsoft Windows is used in all illustrations. 1.3 Installation Microsoft Windows Under Microsoft Windows a script for the native windows installer is available. Just download and run the mlhmp.msi install file. It installs the software and all plug-ins automatically and creates a shortcut in the windows start menu. 11

12 Dina Notat No. 84, Third Edition Other platforms MLHMP is installed manually. You must make sure that you have Java installed on your system. If that is not the case you can download it from Sun s web site. At least the Java Runtime Environment (JRE) must be installed on your system. It is not necessary to install the Software Development Kit (SDK). Having verified the presence of Java on your system, the recommended install procedure is as follows: 1. Create a new directory. 2. Download the mlhmp.jar file and save it in the new directory just created. 3. Download the plugins.jar file and save it in the new directory just created. 4. Download the Limid.jar file (from esthauge.dk) and save it in the same directory. 5. Create a script for launching the MLHMP software. 12

13 Chapter 2 Using the program 2.1 Starting the program When properly installed under Microsoft Windows, the program has been added to the Start menu. Under other operating systems (and if desired under Windows) it is recommended to create a script to launch the program. 2.2 Windows When the program is launched it displays the following empty window. In order to do something you must either create a model from scratch, automatically generate a model using an installed plug-in or load an existing model from the disk (see later). As soon as a model is loaded or has been created, the Process tree window will appear. As soon as the model has been optimized, the Optimization log window and the Results window become available through the "Window menu of the menu bar or by clicking the 13

14 Dina Notat No. 84, Third Edition 2007 appropriate tab of the tab pane. Each window has its own button bar with functions that are often specific for that window. In the following all three windows are described in details The Process tree window Description The process window displays the structure of the model, and it is also used for editing of models. The structure is shown as a tree having the fonder process as root. The other nodes of the tree represent child processes, stages, states or actions. In general, a node is either a process, a stage, a state, or an action. A node may either be shown collapsed (C) or expanded (X) depending on whether or not it shows its children. The following table shows the symbols used for each kind of node. C X Object Children Process. The identity is shown in brackets Stage (ordinary) LIMID Stage (experimental feature) State - Action represented directly by parameters None - Action allowing the current process to terminate None Ation represented by child process Action sharing its child process - Action sharing the child of another action None arent action having directly defined probabilities One or more stages One or more states One or more states One or more action One and only one process One and only one process One and only one process So, in general a process is regarded as an array of stages; a stage is regarded as an array of states; a state is 14

15 Software Users Guide: Multi-level hierarchic Markov processes regarded as an array of actions. An action may either be defined directly by parameters (reward, output, transition probabilities etc.) or it may be represented by a separate child process. Editing of labels Optionally, a name or a description may be associated with each stage, state or action node as a label that displays next to he symbol representing the node. If the Edit choice of the choice bar is set to Labels only", the name of a node may be edited by double-clicking the node as shown in the following figure: Editing of parameters A process has no other characteristics than its identity (which is generated by the system) and the list of stages that it contains. A stage in turn is defined by the list of states it contains, and in addition it may be given a name that may be edited as described in the previous section. If the stage is a LIMID 1 stage, a fourth tab with the label Limids appears to the right of the Result table tab as shown in the figure below. Editing of the parameters of a LIMID stage is done in the Limids window which is briefly described in Section Experimental feature: Limited Memory Influence Diagram 15

16 Dina Notat No. 84, Third Edition 2007 A state and an action, contain additional information which may be displayed and - sometimes - edited. In order to display this additional information, the Edit choice of the choice bar must be set to Parameter values. When double-clicking a state node the following information displays in a separate window: The information only makes sense if the model has been optimized. It shows the label, the currently selected action (after an optimization it shows the optimal action), the present or relative value of the state (present value under the discounting criterion and relative value under the average criteria), the expected reward until the end of the current child level one process, and finally, the corresponding expected output (zero under the discounting criterion). The only available options are to close the window and to print it. When double clicking an action node, the following information displays in a separate window: In general, fields that may be edited are white, and those that are calculated by the system are gray. If it is an action, which is defined directly by parameter values, the basic parameters like reward, output, stage length and transition 16

17 Software Users Guide: Multi-level hierarchic Markov processes probabilities may be edited by double clicking the appropriate fields. The reward and output (and sometimes the stage length) are defined to the left whereas the transition probabilities (and sometimes the stage lengths) are defined in the middle. The right part of the window is never edited. It contains the value of the action (in the policy improvement part of the algorithm, the action having the highest value is selected for the new policy), the label of the next stage (in this case Parity 2 of a sow). Since the example shows an action defined directly by parameters, no child process is defined (indicated by null ). Otherwise the identity of the child process would display in the Child field. The field Terminator? is a boolean field indicating whether the action may terminate the current process before the natural end, and the Distribution field shows the way in which the transition probabilities are generated. In this case the method is Simple enumeration indicating that the probabilities are stored in a simple array where each element represents a probability that may be edited directly in the middle of the window. If an other representation of the probability distribution is wanted, the popup menu specifying the Distribution class is clicked. The desired distribution is then selected from the menu as shown in the figure below: Having selected the desired distribution, the following procedure depends on the selection. If a Poisson, Binomial or Normal distribution is selected, the user is prompted for entering of the relevant parameters in a separate popup window: The parameters depend on the distribution. Having entered the parameters, the probabilities are generated and displayed according to the selected distribution. It is not possible to edit individual probabilities if the distribution has been defined this way. Only the parameter values may be edited. Actually, the individual probabilities are not stored in the model. They are calculated each time the model needs them. In other words, these distributions require less memory, but, more time for computation. Unless the model has problems with memory, the distributions should afterwards be converted to simple enumerations in order to speed up the calculations. Such a general conversion may have a dramatic effect on the time needed for optimization. The normal distribution needs specification of mean and standard deviation. Both parameters assume the unit of the state numbering of next stage. In other words, the stage number is assumed to make sense as for instance a count of something. This is relevant, if for instance the state number is equal to the litter size of a sow. Most often, however, the state number is only a label without any real interpretation. In such cases, a normal distribution with parameters defined relative to state numbers is hardly relevant. In some cases, however, a proper transformation of the parameters may supply the desired distribution. Let j {0, 1, 2, 3, 4, 5, 6, 7, 8} be the state at next stage. Assume that j represents the milk yield Y of a dairy cow, and that j = 0 represents Y 4500 kg, j = 1 represents 4500 < Y 5500 kg, j = 2 represents 5500 < Y 6500 kg,..., j = 8 represents Y > kg. If we denote as y j the mid-point of the milk yield interval represented by state j, we observe that j = (y j 4000)/1000. Suppose Y is N (8000, ). We may then consider j to be N (4, 2 2 ), and by specifying a mean equal to 4 and 17

18 Dina Notat No. 84, Third Edition 2007 a standard deviation equal to 2, the desired distribution is created. The binomial distribution prompts for entering of the probability and the maximum sample size, which by default is set equal to the highest state number at next stage. A lower value will cause the probability of higher state numbers to become zero. A higher value will lead to a situation where the probabilities do not sum to unity. The poisson distribution has only one parameter, which is the mean (in a poisson distribution the variance is equal to the mean). Nevertheless the user is also prompted for entering of a maximum sample size. By default it is set equal to the highest state number at next stage. The additional parameter is necessary, because we only have a finite number of states. The distribution therefore has to be truncated in order to sum to unity. The simple enumeration distribution is an ordinary multinomial distribution having each probability defined directly and stored in memory. This is the default representation of a distribution. The values are entered directly in the table in the middle. The states of the next stage appear to the left in the middle section. The probability for transition to any of those states is shown in a column. In order to assist the editing of transition probabilities, the current Probability sum is shown in the top right of the window. This sum has to equal one as close as the precision of floating numbers allows. If an other distribution (e.g. normal, binomial, or poisson) has been specified, and Simple enumeration is afterwards selected, the specified distribution is converted to a simple enumeration which is far more efficient in optimizations (but on the other hand requires more memory). The deterministic distribution is used when only transition to one particular state at next stage is possible. In other words one probability has the value 1 while all others have zero values. If a deterministic distribution is selected, the user must select the target state (the state having unity probability) in a separate popup menu as shown below. If a transition is deterministic it should also be defined so even though it may also be represented as a simple enumeration. The deterministic distribution, however, is far more efficient in calculations. The sparse enumeration is a kind of compromise between simple enumerations and deterministic distributions. It is used when only a few probabilities are non-zero. As a rule-of-thumb it may be used if the number of nonzero probabilities is less than half of all. The way of creating a sparse enumeration is by first selecting a simple enumeration, entering the probabilities, and afterwards selecting a sparse enumeration. The simple enumeration is then automatically converted to a sparse enumeration. A sparse enumeration is not editable. It has to be converted to a simple enumeration in order to be edited. The dummy - all zero option is not a distribution. It just puts all probabilities equal to zero. Ii is used for clearing the probability table if the user wants to start from scratch. If the action is defined as an entire child process, no values are editable (except the transition probabilities if it is a parent action having directly defined probabilities - such actions are displayed with the specific -icon in the process tree). If the model has not been optimized most numerical values are zero. Otherwise they are the parameters calculated from the child process indicated in the Child field. The options available in the window are to cancel (without saving changes), to accept and keep the changes (the OK button), to print the window or to select a distribution. 18

19 Software Users Guide: Multi-level hierarchic Markov processes Editing model structure The process tree window is also used for creating of models from scratch and for editing the structure (adding or removing nodes) of existing models. When starting from scratch, a process with one stage, one state and one action is created by selecting the New: Empty model item from the menu bar: This is only possible if no model is currently loaded. If a model is already loaded it has to be closed first (before it closes the program will ask whether or not to save the model first). As a result of the New empty model selection, a dialog window will prompt you for basic model properties like model name, interest rate, time unit (basis for interest rate), default stage length, and precision (to be used for automatic probability check): The Model name will appear in the title bar of the program window. The Basis for interest rate, Interest rate, level 0 and Default stage length must mutually fit each others. If you want to specify the interest rate as 10% on an annual basis, and the default stage length to be 3 days, you may either use the values (basis, interest, length) = (365, 0.1, 3), or the values (1, 0.1, 3/365). When you later enter actual stage lengths of the model, you must use the same unit as used for the Default stage length. The list of Defined quantities is used for renaming reward and output to more model specific terms as for instance Net returns and Piglets. Furthermore additional quantities to be used in Markov chain simulations may be defined. The three buttons to the right are used as follows: Add quantity: If the button is pressed, a popup window for entering of the name of the new quantity is displayed. Later, when the parameter values are being entered, a field for entering of the value for each defined quantity 19

20 Dina Notat No. 84, Third Edition 2007 is available in the Edit parameters window. Remove quantity: If a quantity is selected, and the "Remove quantity" button is pressed, the selected quantity is deleted from the model. If you try to remove either the reward or the output, an error message is displayed. Rename quantity: If the button is pressed, a popup window for entering of a new name for the quantity is displayed. If reward or output is renamed, the new names will appear in all windows and menus instead of the standard words Reward and Output. As a result of the definition of basic model properties a new model is displayed automatically. It is the simplest possible model with one stage, one state and one action: In order to edit the structure of the created model (or any currently loaded model) you need to enable structural editing which you do by clicking the Edit model structure -button: 20

21 Software Users Guide: Multi-level hierarchic Markov processes As a result, the Add component, the Remove component and the Build model buttons are activated. In order to add components (processes, stages, states or actions), you select the parent and click the Add component button. A toggle button determines whether stages are added as normal stages or as LIMID stages. In order to remove a component, it is selected, and the Remove component button is pressed. Both operations may also be performed using the pop-up menu which is enabled by right-clicking a component. When structural editing has been finished, you press the Build model button, and the new or revised model is created from the process tree image. If LIMID stages have been added, the Limids tab will appear, and the LIMIDs must be edited afterwards in the Limids window (see Section 2.2.4) The optimization log window The log window is split in two where the upper half displays the iterations of the last optimization or simulation in table format. The lower part displays exactly the same information, but as a continuous scrollable plain text log. In addition, it is used for information about other functions performed. Thus the content of the upper part is replaced each time an optimization or simulation has been performed, whereas the new iterations are just written to the end of the log in the lower part. The lower part of the window serves as a general text browser. The log may be saved to a file in plain text format, and correspondingly any plain text file may be opened and displayed in the window. At any time the contents may be cleared by pressing the New button of the log window button bar. Furthermore the visible part of the log may be printed by pressing the Print button. 21

22 Dina Notat No. 84, Third Edition The results window Description The results window has two purposes: To display the results of an optimization in different ways and to serve as a template for editing of policies used for simulation. After an optimization, the optimal policy appears in a large table as shown below. Each row in the table represents an optimal action which is shown at the right-most non-empty column. The previous columns identifies the stage and state of the process in question as well as stage, state and action of all parent processes. By pressing appropriate buttons, the optimal policy may be shown in more numerical ways as described in the next section. 22

23 Software Users Guide: Multi-level hierarchic Markov processes Additional facilities No matter what the table contains, the contents may be saved to a file in comma separated format. Such a file may be imported for further analyses in most standard spreadsheets like Excel. Furthermore, the visible part of the table may be printed. By pressing the -button, the present or relative value of each possible action is displayed in the table. If the criterion of optimality is Discounting the value displayed in the right-most column is the present value. Under the other criteria, the value displayed is the relative value of the action. By pressing the label-less button, the present or relative value of each action is displayed as the deviation from the corresponding value of action 0. In other words, the value displayed for action 0 in a state is always zero. If, in a replacement model, action 0 is Keep, and action 1 is Replace, the value displayed for action 1 is identical to what is called the future profitability or the retention pay-off in animal replacement literature. Pressing the pink button displays the optimal policy again. 23

24 Dina Notat No. 84, Third Edition 2007 Editing of policies and simulation The MLHMP software contains a probabilistic simulation facility. The idea is that in addition to the optimal value of the object function (as defined by the criterion of optimality), a number of additional technical or economical results may be calculated under the optimal policy or under any other user-defined non-optimal policy. The additional results may be calculated as the average value over time of any of the defined quantities of the model (including the reward and output). Furthermore the infinite time ratio between two quantities may be calculated. In order to run a simulation, a policy must first be available. Therefore the first step is to perform an optimization (refer to Section 2.8). The optimal policy is then displayed in the results window. By double-clicking an optimal action, a pull-down menu showing the possible alternative actions will display as shown below. An action is selected from the menu, and the policy has been modified for the state in question. When all desired changes (if any) have been made, a simulation may be performed by selecting the Simulation submenu of the Run menu. What happens next depends on the kind of simulation chosen from the menu and the model type (models are characterized by their time horizon which may be finite or infinite). For infinite time models, the kind of simulation may be chosen as set to Average quantity over time. The user 24

25 Software Users Guide: Multi-level hierarchic Markov processes then has to select the desired quantity. The average value over time is then calculated under the defined policy and displayed in the optimization log window. Also for infinite time models, the kind of simulation may be chosen as Average ratio between two quantities. The user then has to select both the desired quantities. The average ratio between the two quantities is then calculated under the desired policy and displayed in the optimization log window. For finite time models, the kind of simulation may be chosen as Total value over finite time. The user then has to select the desired quantity. The total expected value over the time horizon is then calculated under the desired policy and displayed in the optimization log window. Both types of models are also able to calculate present values under the policy by selecting the Discounted value menu item. For both types of models, the criterion may (prior to simulation) be set to Discounting. The present values of the defined policy are then calculated and displayed in the optimization log window. 25

26 Dina Notat No. 84, Third Edition The Limids window If one or more LIMID stages have been defined in the model, the Limid window will be available for editing the LIMID representing the stage in question. The feature is experimental (and not yet fully implemented). LIMIDs were originally introduced by Lauritzen and Nilsson (2001), and the technique of combining a LIMID and a Markov decision process has been briefly described by Nilsson and Kristensen (2002). The LIMID facility used is provided by the Esthauge LIMID software system. The LIMIDs are built and edited as described in the documentation of the Esthauge system. Reference is made to 26

27 Software Users Guide: Multi-level hierarchic Markov processes 2.3 Loading models Existing models (previously saved from MLHMP, created by other computer programs or manually edited) may - depending on format - be loaded by the Open option or the Import option. In both cases the basic file format is a special XML format for multi-level hierarchical Markov processes. The only difference between Open and Import is that Open reads from uncompressed text files, whereas Import reads from ZIP-compressed files The Open option The Open option is used for models saved in the special MLHMP XML format. It is recommended that you use the file-extension.hmp for this kind of files. The Open option may be selected either from the File menu or by pressing the Open button of the button bar. In either case, a file dialog displays in a separate window. The exact layout of the dialog window depends on the platform The Import option The Import option is used for models saved as ZIP-compressed XML files. Such files either originate from models saved through the File Export menu item or from conventionally saved files which have afterwards been ZIP-compressed. It is recommended that you use the file-extension.zip for this kind of files. 2.4 Saving models The Save and Save as options The Save and Save as options are the counterparts of the Open option thus saving in the specially defined XML format. The Save option is only enabled if a model has been loaded through the Open option or if it has already been saved once before. The Save option may be selected either from the File menu or by pressing the Save button of the button bar. The model is saved without further notice using the file name appearing to the left in the status bar at the bottom of the MLHMP window. Thus the previous version of the model is overwritten. The Save as option must be selected from the File menu, and a file dialog displays in a separate window. The exact layout of the dialog window depends on the platform. It is recommended to use the.hmp extension for such files The Export option The Export option is the counterpart of the Import option thus saving in ZIP-compressed format. The option is selected from the File menu, and a file dialog displays in a separate window. The exact layout of the dialog window depends on the platform. It is recommended to use the.zip extension for such files. 27

28 Dina Notat No. 84, Third Edition Edit model properties This option is selected from the Edit menu. It is used to define a name and a number of quantities for the model and for entering basic information (page 19) like interest rates and precision. Refer to Section for further details. 2.6 Plug-ins What is a plug-in? The MLHMP software has a facility for installing and removing plug-ins, which are Java classes extending an abstract ModelProvider class. The purpose of a plug-in is to generate an entire model for a specific purpose. The New menu item of the File menu will display a list of properly installed plug-ins. Selecting a specific plug-in will generate the corresponding model. It is entirely up to the plug-in to decide in what way the model is created. It may either be as a batch job with everything decided in advance, or it may be through an interactive dialog with the user (for instance prompting for prices, correlations, etc.) The plug-in list When MLHMP is downloaded and installed it may already be equipped with some plug-ins. It may also be possible to download additional plug-ins from the MLHMP home page. In order to install or remove a plug-in, you select the Installed plugins menu item of the Edit menu. 28

29 Software Users Guide: Multi-level hierarchic Markov processes Having selected Installed plugins a list of currently installed plug-ins displays at the right of the process tree window. In the example below, a total of 9 plug-ins have been installed Installing and removing plug-ins The class path Before it is possible to install a plug-in, the class file has to be placed on a location included in the class path specified with the initial call to Java. When installed under Microsoft Windows the directory [MLHMP]>/UserPlugins 2 is predefined as being included in the class path. 2 [MLHMP] is a reference to the directory where MLHMP is installed. The exact location may vary, but typically the path is something like C:/Program Files/MLHMP - Multi-Level Hierarchical Markov Processes. 29

30 Dina Notat No. 84, Third Edition 2007 If the full name of your plug-in is, for instance, foo.bar.myplugin the class file MyPlugIn.class can be placed in the directory [MLHMP]>/UserPlugins/foo/bar. It will then be accessible from MLHMP. Another option is to place the class file in any directory, jar or zip file and manually edit the class path to include the location. The class path is specified in the file [MLHMP]>/MLHMP.ini (use for instance NodePad for editing). Having included the location in the class path, the plug-in is accessible. Installing a plug-in A new plug-in is installed by pressing the + button below the list of installed plug-ins. The user is then prompted for entering the name of the plug-in in a popup window. If the specified class file does not exist (or if it is not a plug-in) an error message is displayed. In order to remove a plug-in, it is selected, and the - button is pressed. When plug-in management is finished, the OK button is pressed, and the user is told to exit MLHMP and reload in order to activate the changes. 2.7 Building models In order to build a model from scratch, you first have to define the structure of the model as described in Section and afterwards you must enter the parameters as explained in Section It is not possible to enter or edit parameter values while the structure is being defined or edited. Both tasks are carried out from the Process tree window as described in a previous section. When the parameters have been entered the consistency of the transition probabilities may be checked through the Probability check option which is selected either from the Run menu or by pressing the Check probabilities button as shown below. As a result, it is checked whether all probability distributions sum to 1 with the precision defined in the basic model properties. 30

31 Software Users Guide: Multi-level hierarchic Markov processes The result of the probability check is displayed in a message box which either states that no errors were found or informs you that errors were indeed found. In the latter case you must refer to the optimization log window for details. It should be emphasized that a positive result of a probability check in no way guaranties that the probabilities entered make sense. On the other hand a negative results certainly implies that something is wrong (or that the precision is set at a wrong value). 31

32 Dina Notat No. 84, Third Edition Optimization When the structure of the model has been defined, and all parameters have been entered, you may start using the model. To use it primarily means to calculate optimal policies under different criteria of optimality. For infinite time models (models with only one recurrent stage in the founder process), the standard method for optimization is the hierarchic algorithm described by Kristensen and Jørgensen (2000). Accordingly the method used is policy iteration in the founder process and value iteration in the child processes. An alternative method available is to use value iteration in the founder process as well. If the model is a finite time model (models with several explicitly defined stages in the founder process) the standard method for optimization is value iteration at all levels. No matter which method is used, the consequences of modifying the resulting policy may be exploited by probabilistic Markov chain simulation Criterion of optimality Four (five) alternative criteria of optimality are available: Discounting (finite or infinite time): Maximizes the present value of an infinite process starting in state i. This criterion is available for finite as well as infinite time models. Average rewards over output (infinite time): Maximizes the long run average rewards/output ratio 3. This criterion is not implemented for value iteration. Average rewards over time (infinite time): Maximizes the long run average rewards/time ratio 4. Under value iteration, this criterion is interpreted as maximization of average rewards per founder state. (This is in fact a criterion which is not implemented for the standard optimization method.) Before the iteration is started a warning describing this is shown in a separate message box: Total Rewards (finite time): Maximizes the total expected rewards over the time horizon of the model. Before an optimization is performed, the desired criterion is selected from the pull down menu in the choice bar. Only the options that are relevant for the loaded model are available Optimization This option is selected as Optimization from the Run menu. Alternatively, the run button may be pressed. The resulting optimal policy may be inspected in the results window. The value of the objective functions for all iterations (including the final optimal value) may be found in the optimization log window. Present (or relative) values for all founder states are shown. 3 If rewards and outputs have been renamed as described in Section 2.2.1, the new names are used. 4 If rewards have been renamed as described in Section 2.2.1, the new name is used. 32

33 Software Users Guide: Multi-level hierarchic Markov processes Value iteration This option is selected as Value iteration from the Run menu. Since it is the standard method used for optimization under finite time horizon the (separate) menu item is disabled for finite time models. For infinite time models the algorithm is a non exact alternative to policy iteration. The idea is to iterate until convergence. The user must specify the number of iterations to perform in a popup window. The terminal value is zero. The resulting approximately optimal policy may be inspected in the results window. The value of the objective functions for all iterations may be found in the optimization log window. Under the discounting criterion the values shown are cumulative present values for all founder states. Under the average criterion, the values shown for each founder state are the increase in total value. This increase asymptotically converges towards the long run average reward per founder stage. The relative progress of the value iteration is shown in the progress bar at the bottom right of the window as shown below. 33

34 Dina Notat No. 84, Third Edition

35 Chapter 3 The menu bar The menu bar contains the menus File, Edit, Run, Functions, Window and Help. 3.1 File The File menu contain a submenu used for creation of new models from scratch and menu items for all operations related to file management and printing New The New option is a submenu used for creating of new models from scratch. New: Empty model The first menu item of the New submenu creates a new empty model, which is the simplest possible containing only a founder process with one stage, one state and one action. This menu item is the only one which is available in all versions of the software. 35

36 Dina Notat No. 84, Third Edition 2007 New: [plug-in] In some versions of the software several other options are available. Such menu items represent installed plug-ins, and they creates models for specific purposes. Refer to Page 28 for an illustration Open This option displays a file dialog used to select an existing model stored in the special XML format in a file (typically with the extension.hmp ). It is only enabled if no other model is currently loaded Close This option closes the existing model. It is only enabled when a model is currently loaded. Before the model is closed, you are asked whether you want to save the model to file before closed. If you answer Yes, a file dialog for entering the file name is displayed Save This option saves the currently loaded model in XML format using the existing file name (from which the model was loaded or where it was saved using the Save as option). If the model has been edited, the new version will overwrite the existing model. The item is only enabled when an internal format filename exists Save as This option saves the currently loaded model in XML format using the filename entered by the user in a file dialog window. It is recommended to use the file extension.hmp. The option is only enabled when a model is currently loaded Print log This option displays a print dialog for printing the visible part of the optimization log Print tree This option displays a print dialog for printing the visible part of the graphical process tree. 36

37 Software Users Guide: Multi-level hierarchic Markov processes Import ASCII This option displays a file dialog used to select an existing model stored in zip-compressed format in a file (typically with the extension.zip ). It is only enabled if no other model is currently loaded Export ASCII This option saves the currently loaded model in zip-compressed XML format using the filename entered by the user in a file dialog window. It is recommended to use the file extension.zip. The option is only enabled when a model is currently loaded Exit This option exits the software without warnings. The currently loaded model is lost. 3.2 Edit Model properties This option displays the model properties dialog window (Page 19). It is only enabled when a model is currently loaded Installed plug-ins This option displays the plug-in management dialog window (Page 29). 37

38 Dina Notat No. 84, Third Edition Run Optimization This option calculates an optimal policy using the hierarchic algorithm described by Kristensen and Jørgensen (2000) for infinite time models and value iteration for finite time models. Refer to Section for further information Probability check This option performs a probability check of the entire model and displays the result in a message window as described in Section Value iteration This option calculates an approximately optimal policy using value iteration all over. Refer to Section for details. 38

39 Software Users Guide: Multi-level hierarchic Markov processes Simulation This submenu is used to calculate the technical and/or economic consequences of a predefined policy as described in Section Discounted value Calculates the discounted (present) value of all founder states. Average ratio between two quantities Initiates a dialog allowing the user to choose two quantities of which the infinite time average ratio is then calculated. The option is only enabled for infinite stage models. Average quantity over time Initiates a dialog allowing the user to choose a quantity of which the infinite time average over time is then calculated. The option is only enabled for infinite stage models. Total value over finite time Initiates a dialog allowing the user to choose a quantity of which the expected total value over the duration of the process is then calculated. The option is only enabled for finite stage models. 3.4 Functions This menu is used to for operations on the currently loaded model. It is only active when a model is displayed in the process tree window. It is deactivated while optimizations and simulations are performed. 39

40 Dina Notat No. 84, Third Edition Convert distributions to arrays - keep deterministic This option converts all distributions (except deterministic) to simple enumerations represented as arrays. This requires more memory, but it is much faster in optimizations and simulations. This option does not convert deterministic distributions because they are very efficient concerning both speed and memory Convert distributions to arrays (incl. deterministic) This option has the same effect as the previous one except that it also converts deterministic distributions to simple enumeration Redefine reward This option redefines rewards to be an other of the defined quantities. The desired quantity is selected in a separate popup window. 40

41 Software Users Guide: Multi-level hierarchic Markov processes Redefine output This option redefines outputs to be an other of the defined quantities. The desired quantity is selected in a separate popup window Reset reward and output to standard This option resets reward and output to the originally defined quantities Collapse this hierarchic model This option converts the currently loaded (hierarchical) model to an ordinary Markov decision process (i.e. with only a founder level). Since the currently loaded model is closed, the user is asked whether or not the hierarchical model should be saved before the conversion (see the dialog on Page 36). After the conversion the resulting (one level) Markov decision process replaces the hierarchical model Write founder transition matrix to disk This option writes the transition matrix of the founder process to a tabulator separated text file. The name of the file is specified by the user in a usual file save dialog. The currently defined policy is used to generate the matrix. State names are given in first row and first column of the file. The file may afterwards be imported in e.g. a spreadsheet. 3.5 Window This menu is used to select the window to be displayed. For a description of the various windows reference is made to Section

42 Dina Notat No. 84, Third Edition Process tree This option displays the process tree window showing the basic model structure. Selecting the tab Process tree has the same effect Optimization log This option displays the optimization log window showing iterations of optimizations and simulations. Selecting the tab Optimization log has the same effect Results This option displays the process tree window showing the results of optimization and value iteration. Selecting the tab Result table has the same effect. 3.6 Help The only menu item of the Help menu is the About option. 42

43 Software Users Guide: Multi-level hierarchic Markov processes About Displays information about the software. 3.7 Menus installed by plug-ins A plug-in is allowed to install its own menu on the menu bar. In the example below, a plug-in has installed a menu called Sow model. The functionality is completely defined by the plug-in. 43

44 Dina Notat No. 84, Third Edition

45 Chapter 4 The choice bar The choice bar is used for selecting the criterion of optimality and edit mode for model elements. 4.1 Criterion of optimality This pull down menu selects the criterion of optimality as described in Section Edit (Labels only or Parameter values) This pull down menu selects the editor to be invoked by double clicking a model component in the process tree window. The editor may either edit labels (Section 2.2.1) or - if the component is a state or an action - parameter values (Section 2.2.1). 45