PARTICIPANT DATA REPLICATION BATCHER USER GUIDE

Transcription

1 PARTICIPANT DATA REPLICATION BATCHER USER GUIDE VERSION: 3.03 DOCUMENT REF: PREPARED BY: TDPD72 ITDS DATE: 17 May 2013 Draft For pdrbatcher version 7.3 and later Am,tolion l:neror 1\Jln,ketOpe rctor Ltd AeN 94 07' Wv'IW.oemo.oom. ou infofffoemo.corr.ou NEW sou- H WAI.ES QU:ENSLAND SOUTH AUSTRALIA VCTORIA.. usrrauan CAP1"-l Tl'RRITORY

2 Disclaimer In this document, references to the software are to the pdrbatcher, unless specified otherwise. References to associated materials include this document, PARTICIPANT DATA REPLICATION BATCHER USER GUIDE. 1. Purpose This software and the associated materials have been produced by the Australian Energy Market Operator Limited (AEMO) for supply to users free of charge in order to demonstrate what users can use in their interface to an AEMO system plus to provide information about pdrbatcher as at the date of publication. They may not necessarily be of production quality and may not be fully tested. They are provided as an example only. If the provided functionality is insufficient for Participant use, users have to independently develop or acquire software to meet their needs. 2. No substitute This software and the associated materials are not a substitute for, and should not be read in lieu of, the National Electricity Law (NEL), the National Electricity Rules (Rules) or any other relevant laws, codes, rules, procedures or policies. Further, the contents of this software and the associated materials do not constitute legal or business advice and should not be relied on as a substitute for obtaining detailed advice about the NEL, the Rules, or any other relevant laws, codes, rules, procedures or policies, or any aspect of the national electricity market or the electricity industry. 3. No Warranty While AEMO has used due care and skill in the production of this software and the associated materials, neither AEMO, nor any of its employees, agents and consultants make any representation or warranty as to the accuracy, reliability, completeness or suitability for particular purposes of the software and the associated materials. A user runs the software or uses the materials at its own risk. AEMO does not represent or warrant that the software and materials are complete, free from errors, reliable and fit for the purpose set out above. Nor does AEMO represent or warrant that the software is free from viruses and other programs which may affect other software or systems. The software and associated materials are provided on the basis that users proposing to use or rely on the software or associated materials undertake responsibility for independently verifying and checking the accuracy, completeness, reliability or suitability of the software or associated materials. 4. Limitation of liability - To the extent permitted by law, AEMO and its advisers, consultants and other contributors to this software and the associated materials (or their respective associated companies, businesses, partners, directors, officers or employees) shall not be liable for any errors, omissions, defects or misrepresentations in the information contained in this software and the associated materials, or for any loss or damage suffered by persons who use or rely on such information (including by reason of negligence, negligent misstatement or otherwise). If any law prohibits the exclusion of such liability, AEMO s liability is limited, at AEMO s option, to the re-supply of the information, provided that this limitation is permitted by law and is fair and reasonable. Copyright Copyright 2013 Australian Energy Market Operator Limited. All rights reserved. Trademark Notices Microsoft, Windows and SQL Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Oracle and Java are registered trademarks of Oracle Corporation and/or its affiliates. Distribution Commercial-in-confidence with NEM participants This Document Identification Title: PARTICIPANT DATA REPLICATION BATCHER USER GUIDE Version: 3.03 Document ID: TDPD72 Responsible Department: ITDS Notes: pdrbatcher requires Oracle s JRE7. Documents made obsolete: The release of this document changes only the version of the PARTICIPANT DATA REPLICATION BATCHER USER GUIDE. No documents are made obsolete by releasing this document. Doc Ref: TDPD72 v May 2013 Page ii

3 Contents 1 Introduction Purpose Audience Scope Related resources Conventions Context What the software is for How do you use the software? Who can use this software? Software requirements Setup Downloading the application Environment pre-requisites Working directories Remote connectivity (FTP or NetBIOS) Installation Application Directory structure created Upgrades Configuration considerations Architecture of directories File masks Encrypted parameters Minimal configuration of the properties file Running Maintaining Logging configuration Running as a Windows service Installation Uninstalling Batcher extendibility File event handler Configuration Properties Working directory locations Holding directory (batcher_holding_dir) Transfer log directory (file_log_dir) Data source definitions Host sources (batcher_data_sources) Doc Ref: TDPD72 v May 2013 Page iii

4 4.2.2 Host mode (batcher_{ds}_mode) Host (batcher_{ds}_host) Username (batcher_{ds}_username) Password (batcher_{ds}_password) Password encryption (batcher_{ds}_password_encrypted) Home directory (batcher_{ds}_dir) Home archive directory (batcher_{ds}_archive_dir) FTP connection mode (batcher_{ds}_ftp_mode) FTP connection time-out (batcher_{ds}_ftp_timeout) Cycle definitions Heartbeat file (batcher_heartbeat_file) Polling interval (batcher_polling_interval) Wait before reconnect (batcher_fail_interval) Batcher inactivity time-out (batcher_inactivity_timeout) Thread definitions Number of threads (batcher_threads) Number of threads (batcher_threads_active) Data action mode (batcher_thread_{n}_mode) Data destination (batcher_thread_{n}_dest) Source directory override (batcher_thread_{n}_source_dir) Destination directory (batcher_thread_{n}_dest_dir) Subdirectory process list (batcher_thread_{n}_sub_dir) Subdirectory recursion (batcher_thread_{n}_dir_recurse) Subdirectory create (batcher_thread_{n}_dir_create) Subdirectory destination (batcher_thread_{n}_sub_dir_dest) Allow overwrite flag (batcher_thread_{n}_allow_overwrite) Include file mask (batcher_thread_{n}_inc) Exclude file mask (batcher_thread_{n}_exc) Time-out override (batcher_thread_{n}_timeout) Polling override (batcher_thread_{n}_polling_interval) Pause (batcher_thread_{n}_pause_interval) Activity monitor (batcher_thread_{n}_activity_monitor) Files before repoll (batcher_thread_{n}_process_limit ) File processing sequence (batcher_thread_{n}_process_order ) File transform operation (batcher_thread_{n}_file_translator ) Bandwidth throttling (batcher_thread_{n}_throttle ) Retention time (batcher_thread_{n}_retention_time_mins ) Source archive directory (batcher_thread_{n}_source_archive_dir ) Retention time (batcher_thread_{n}_retention_time_mins ) File purging definitions File purge configurations (file_purge_configurations) File purge directory (file_purge_{c}_directory) File purge directory retention time (file_purge_{c}_retention_time_mins) File purge frequency (file_purge_{c}_purge_frequency_mins) Purge include file mask (file_purge_{c}_inc) Purge exclude file mask (file_purge_{c}_exc) Configuration files Log4j properties file (log4j) Batcher configuration properties file (config) Index of properties Possible Architectures using pdrbatcher FTP or NetBIOS Doc Ref: TDPD72 v May 2013 Page iv

5 6.2 Single source using a single thread Single source using multiple threads Single source to multiple local directories Multiple sources to multiple local directories Bidirectional communication Multiple instances Glossary Needing Help?...29 Doc Ref: TDPD72 v May 2013 Page v

6 1 Introduction 1.1 Purpose This release of the user guide (PARTICIPANT DATA REPLICATION BATCHER USER GUIDE version 3.03) matches the Participant Data Replication Batcher version Audience This document is intended for registered participants technical and software development staff who are responsible for systems implementation. 1.3 Scope This manual covers the setting up and use of the Participant Data Replication Batcher version 7.3. This document assumes you have knowledge of: The Java application environment. The operating system your organisation is using. How the NEM systems operate Related resources The Participant Data Replication Batcher software is available on the "EITS Publications" secured webpage ( The following resources may be useful in addition to the information contained in this manual: Oracle JRE 7 download: Configuring logging (log4j) EMMS Data Interchange Guide, available on the "EITS Publications" secured webpage ( Guide to Market Systems: Maintaining and Extending Access, available on the "EITS Publications" secured webpage ( Participant Data Replication Batcher User Guide available on the "EITS Publications" secured webpage ( Participant Data Replication Batcher GUI Installer is available on the "EITS Publications" secured webpage ( 1.4 Conventions Important Note: important information is in this style. Note: additional information is in this style. Doc Ref: TDPD72 v May 2013 Page 1

7 2 Context 2.1 What the software is for The Participant Data Replication Batcher application (pdrbatcher) is responsible for moving files between remote and local directory locations. The software is multi-threaded and allows each thread to be configured to move a subset of files between the remote and local directory locations. A prime use for the Participant Data Replication Batcher is a simple interface to MMS to collect.csv files published to a participant s directory on the participant file server. A companion product, the Participant Data Replication Loader (pdrloader), can parse those files into a local database instance implementing the MMS Data Model. Another prime use for the Participant Data Replication Batcher is to submit files for AEMO processing. 2.2 How do you use the software? Remote Server Local Server Remote Directories (batcher_xxx_remote_dir) Remote Transfers via FTP or NetBIOS Receive Threads Send Threads Batcher Receive Threads Send Threads Local Copy Directories (batcher_xxx_local_dir) Local Directories (batcher_thread_{n}_local_dir) Local Copy Directories (batcher_xxx_local_dir) Local Copy Directories (batcher_xxx_local_dir) The Participant Data Replication Batcher application transfers files between remote and local directories (for example, from the AEMO participant file server to a local directory, or vice versa). The batcher can transfer files using either an FTP or a NetBIOS interface, but only FTP is supported at the AEMO file server.. Each local copy directory can receive all or a selection of the files, and each copied file is removed from the source directory. Each thread can be tuned for optimal performance to suit particular business needs. The application design of the batcher is a state machine with the file locations controlling the state. This means that, on restarting after a stop for any reason, the application automatically recovers by using the stored state in the local directories. Each of the local directories contains the results of the file transfers, and a process in the application periodically moves files to the appropriate directory. The design of the Participant Data Replication Batcher application assumes the access to the local directories is highly reliable and efficient, that is with the local directories hosted on the machine running the application. If space is a problem, consider using an ancillary process to empty the various local subdirectories by move or delete. The Participant Data Replication Batcher software is a batch application and hence does not include a graphical user interface. All configuration of the Batcher is done via the properties file. The properties file is only read at application startup, so any changes require the application to be restarted. 2.3 Who can use this software? This software is for each participant in the National Electricity Market interfacing with the MMS system by.csv file delivery mechanism (also known as the flat file delivery mechanism). Doc Ref: TDPD72 v May 2013 Page 2

8 2.4 Software requirements As delivered, the software runs under Oracle s JRE 7. The Participant Data Replication Batcher can run on both Windows and Unix-like operating systems. In a Windows 64-bit environment, both the Java and wrapper for service must be 32-bit or 64-bit software (that is, rather than one of each). All provided scripts are just samples and are neither recommendations nor requirements. Doc Ref: TDPD72 v May 2013 Page 3

9 3 Setup 3.1 Downloading the application The latest version is Participant Data Replication Batcher, available on the "EITS Publications" secured webpage ( Decompress the ZIP file to a base directory, retaining the directory structure recorded in the distribution file. The decompression creates a \pdr\batcher directory with four subdirectories. 3.2 Environment pre-requisites The Participant Data Replication Batcher needs: A set of local working directories. Connectivity to a data source, either remote (FTP or NetBIOS) or local (a directory). Important Note: AEMO s file servers only support FTP. Instances of the Participant Data Replication Batcher connecting to AEMO s file servers must use passive FTP Working directories On the local machine, the Participant Data Replication Batcher requires working directories for the transfer process. A default set creates as part of the decompression of the distribution file. Additionally, a purging process needs to be put in place so the working directories do not consume the free disk space on the local machine Remote connectivity (FTP or NetBIOS) To read from an FTP server, the Participant Data Replication Batcher requires host, password and directory details, plus remote transfer time-out, polling interval and remote transfer reconnect interval. You can override the local directory (where the Participant Data Replication Batcher looks for, or sends, files) and set the FTP connection mode. To read from a NetBIOS service (which excludes AEMO s file servers), the Participant Data Replication Batcher requires directory details, plus remote transfer time-out, polling interval and remote transfer reconnect interval. You can override the local directory (where the Participant Data Replication Batcher looks for, or sends, files). Each thread has filters for inclusion and exclusion (with inclusion having precedence over exclusion), plus an override of the remote transfer time-out, an override of the local directory, an override of the remote directory, override of the polling interval, setting of inter-file delay, and setting of transfer direction (send or receive). For either remote transfer mechanism, the Participant Data Replication Batcher requires a reconnection wait time, primarily to avoid flooding the logging file on disconnection. You can set a maximum time for a file transfer, set a polling interval, and specify a heartbeat file. 3.3 Installation Application 1. Install JRE 7 (if not already available). Note: If you are unsure of the version of Java installed on your system, enter java version at the command prompt or Unix shell to display the version of Java. 2. Unzip the ZIP file into a directory. Doc Ref: TDPD72 v May 2013 Page 4

10 3. Remove the delete.me files. 4. Edit the pdr_key.properties file in the \lib subdirectory and replace the default value with a 16 digit hexadecimal string of your own choice. Note: Before starting to configure your installation, ensure you update the 16 digit hexadecimal key in the pdr_key.properties file. Changing the encryption key (hash) ensures any encrypted passwords in your properties file cannot be decrypted using the default and publicly available hash. 5. Edit the pdr_log4j.properties file in the \lib subdirectory and update the log4j.appender.fileout.file parameter to redirect the log output file to the desired location. Note: Windows style directories using the backslash operator need to be escaped to be read properly from the properties configuration file. An example of a Windows path for this parameter follows: log4j.appender.fileout.file=c:\\pdr\\batcher\\log\\pdr.log Unix paths using the forward slash operator, /, may be entered directly and do not require an escape sequence. An example of a Unix path for this parameter follows: log4j.appender.fileout.file=/home/pdr/batcher/log/pdr.log 6. Edit the pdrenvironment batch file to ensure that the PATH environment variable includes a reference to the directory where the Java executable is installed on your system. Note: The Participant Data Replication Batcher scripts assume JRE 7. If you have another version of Java, ensure you change this batch file to reference the appropriate installation directories to suit. Placing the directory of the Java install at the start of the PATH environment variable ensures the Participant Data Replication Batcher always runs against the correct version of Java. Scripts are samples and not a recommendation or requirement. The design and implementation of a deployment plan is the responsibility of each participant. For example, considering the pdrenvironment.sh wrapper script in a Unix environment, a practice might be to adapt the script into the Unix services layer (generally in \etc\rc.d\init.d\* but may be different depending on the flavour of Unix) and configure it to execute at an appropriate run level at startup Directory structure created Decompressing the distribution file creates a directory (\pdr\batcher) with four subdirectories, collectively called the local directories. One directory (\Lib) contains the application and the other three are working directories for the Participant Data Replication Batcher: Holding: where the batcher saves files being downloaded from the AEMO file server, then (when complete) the batcher moves each file to the local directory for subsequent access. Reports: where the batcher puts files after completion of download, or is the source directory for uploads. Log: contains log files associated with the transfer process. The application files in the \Lib directory include: six JAR files (AppPdrBatcher.jar, commons-codec-1.4.jar, commons-pool jar, commons-net jar, ftp.jar, xml-apis.jar, xercesimpl.jar and log4j jar, apache-log4j-extras-1.1.jar) and one DLL file (NTEventLogAppender.dll), being the application files. three properties files (pdrbatcher.properties, pdr_log4j.properties and pdr_key.properties), to provide parameters for the running instance. One BAT file (pdrbatcher.bat), to run the application in Windows. One SH file (pdrbatcher.sh), to run the application in Unix. one BAT file (pdrenvironment.bat) and one SH file (pdrenvironment.sh) containing the definitions of the environment variables required to run the application. Doc Ref: TDPD72 v May 2013 Page 5

11 three files (pdrserviceinstall.bat, pdrserviceuninstall.bat, WinRun4J.jar, WinRun4J.exe, WinRun4J64.exe and WinRun4J.ini) to support the running of the application as a Windows service. two files (pdrpasswordencrypt.bat and pdrpasswordencrypt.sh) to generate an encrypted password (see Encrypted parameters on page 7). The only file in the directories other than \Lib is a placeholder file to cause the creation of required directories during the unzipping process, named delete.me. Remove these placeholder files. Before attempting to run, edit the two properties files (pdrbatcher.properties and pdr_key.properties) to suit your situation (see 3.4 Configuration considerations below). Note: When setting up the application, the directory structure does not have to be identical to the arrangement described above. By modifying the.properties file, the installer can distribute the directories over different locations. For example, if setting up a pdrloader instance, make the batcher output directory be the loader input directory Upgrades For information about upgrading to the current version of the Participant Data Replication Batcher application, see the document Participant Data Replication Batcher Implementation Note. 3.4 Configuration considerations Architecture of directories The directories created as part of the installation and the properties pointing to those directories in the supplied pdrbatcher.properties file are suitable for standalone testing and demonstration purposes. For use as part of an overall data replication system, the directories need review to ensure all file flows connect, and match the intended architecture (for examples, see 6 Possible Architectures using pdrbatcher on page 26) File masks In order to handle file priority efficiently, the Participant Data Replication Batcher application has multiple threads. Each thread is responsible for processing certain files based on file masks specified by the user. The multi-threading combined with the file mask feature allows higher priority handling of selected file types as well as allowing concurrent transfers. For example, high priority threads (that is, short time-outs) are appropriate for AEMO Dispatch data and lower priority threads are suitable for AEMO MT PASA data. Initiating multiple threads with different file masks involves more setting up effort, but allows for more efficient processing. Therefore, properties configuration is the key to ensure the multiple threads of the Participant Data Replication Batcher application run in a business-efficient manner. Many factors affect end-to-end performance, so tuning the various properties can deliver improvements for selected types of files over other types of files. Note: Beware of over-specifying filters, since it is possible to make a seemingly minor adjustment and possibly leave a type of file with a reduced priority or even no possible thread. For each process, always having a catch-all thread (that is, no filters) can avoid the situation of missing out on a file (even if it is rare). This also allows the batcher to continue in the event of an unexpected death of a thread. Filters are available for each thread. Each thread has an include filter and an exclude filter. Each filter is a comma-delimited list of strings, including no strings. The filter is the logical OR of the strings. Each string excludes leading and trailing spaces and includes asterisks (*) and question marks (?) behaving as wildcards. Each question mark represents a single non-specific character. Each asterisk is a series of non-specific characters (that is, zero to many characters). Doc Ref: TDPD72 v May 2013 Page 6

12 Usually, the beginning and end of each string has an asterisk so the search mask means matching anywhere in the target Include Filter If the include filter is empty or not defined, the include filter is effectively for all files. The include filter, if specified, has priority over the exclude filter. This means a file matching a specific include filter cannot be ignored by using the exclude filter. For more details, see Include file mask (batcher_thread_{n}_inc) on page Exclude Filter If the exclude filter is empty or not defined, the exclude filter is effectively for no files (equivalent to allowing all files through). For more details, see Exclude file mask (batcher_thread_{n}_exc) on page Encrypted parameters Encryption of the password properties avoids a security risk from a casual inspection of the properties file. Use of encrypted passwords can be disabled (see Password encryption (batcher_{ds}_password_encrypted) on page 12). The encryption key is exactly 16 hexadecimal digits long and is in the pdr_key.properties file. Modify the encryption key in the distribution file before determining and setting any encrypted password strings. Ensure the encryption key only has hexadecimal digits (that is, 0-9 and A-F only) and is exactly 16 digits long. In order to determine the encrypted value of a password, use the following from the command line in the \Lib directory, substituting mypassword in the command line with the password appropriate to the purpose: pdrpasswordencrypt mypassword Expect to see results similar to the following: Passwd - encrypting - mypassword Returns - qrs/fu0yliphjbzogwgaaq== Decrypting mypassword Note: Before starting to configure your pdrbatcher.properties file, ensure you update the 16-digit hexadecimal key in the pdr_key.properties file. Changing the encryption key (hash) ensures any encrypted passwords in your properties file cannot be decrypted using the default and publicly available hash. Using a secure encryption key also allows the main configuration file to be sent to AEMO without divulging your password. 3.5 Minimal configuration of the properties file The Participant Data Replication Batcher has been pre-configured to suit a typical installation. In most circumstances, the default values within the configuration are suitable for normal operations, however the Participant Data Replication Batcher is a highly configurable product to ensure that it can meet the needs of multiple clients. All the parameters for each instance of the Participant Data Replication Batcher are in a properties file defining the parameters in a name-value pair format. The Participant Data Replication Batcher reads the properties file at the start of the application and caches the values, so changes to the properties file require a restart of the running instance to take effect. Note: In Java properties files, comments begin with either a hash (#) or an exclamation mark (!). The minimal configuration steps required for a batcher installation are, in the pdr.properties configuration file: Doc Ref: TDPD72 v May 2013 Page 7

13 1. Enter the batcher_host parameter, which is the server name or IP address of the FTP server. 2. Enter the batcher_remote_dir parameter, which is the directory from which to download files from the FTP server relative to the user s home directory. 3. Enter the batcher_username parameter, which is the username to connect to the FTP server. 4. Enter the batcher_password parameter, which is the password associated with the username to connect to the FTP server. Note: The batcher_password property is, by default, an encrypted parameter and the plain text password needs to be encoded according to the Encrypted Parameters section of this document. To use a plain text password, enter the batcher_password_encrypted parameter as false. For specific detail on all configurable properties, please refer to 4 Configuration Properties on page Running Select the appropriate batch file or shell script based on the target environment operating system: For running on a Windows environment, choose the pdrbatcher.bat file. For running on a *nix environment, choose the pdrbatcher.sh file Maintaining The following housekeeping tasks are needed to keep a Participant Data Replication Batcher installation running smoothly: 1. Purge files from the pdrbatcher log directory older than a specified date. 2. MarketNet passwords have a 90 day expiry, so you need to have a password change process in place to ensure your password does not expire. In the event that your password expires or becomes locked out, please contact your Participant Administrator (who may need to contact the AEMO Help Desk). 3. Implement new releases of pdrbatcher as they are made available. AEMO wholesale systems are updated on a 6-monthly release cycle which includes any changes to participant interfaces. These releases are typically scheduled for production in late May and mid-november. Plan to have resources available to implement the MMS releases around this time and keep your systems on a supported configuration. 3.7 Logging configuration The logging system in the Participant Data Replication Batcher utilises the log4j logging framework, an industry standard logging implementation offering a rich set of functionality configurable using a properties file. The properties file defining the log4j configuration is the pdr_log4j.properties file contained in the \lib directory. The log4j configuration shipping with the Participant Data Replication Batcher uses a DailyRollingFileAppender resulting in a log file being generated for each calendar day. There are many ways to customise logging behaviour and target through the log4j framework. The details of how to configure log4j are beyond the scope of this document. However, there are a number of useful references available at the URL Note: Unless specifically needed, use separate log files for the loader and batcher, particularly when using the DailyRollingFile appender in log4j. Doc Ref: TDPD72 v May 2013 Page 8

14 3.8 Running as a Windows service There are a variety of commercial and open source solutions allowing Java applications to run as a Windows service. The distribution includes a configuration for a popular open source solution as an example of how running the Participant Data Replication Batcher as a Windows service can be achieved. The example software is 32-bit, so requires 32-bit Java. Participants need to adapt this example to software of their own choice. The distribution includes three files relevant to running as a Windows service: pdrserviceinstall.bat installs a Java application as a Windows service. pdrserviceuninstall.bat removes a Java application running as a Windows service. WinRun4J.exe is a 32-bit Windows wrapper for the 32-bit JRE WinRun4J64.exe is a 64-bit Windows wrapper for the 64-bit JRE WinRun.ini is a configuration file that defines parameters for creating the windows service Installation The procedure for installing the Participant Data Replication Batcher as a Windows service is as follows: 1. Edit the pdrserviceinstall.bat file and update the HOME and JRE environment variables to suit your environment. Ensure the path to the JRE DLL is valid (for example, %JRE%\\bin\\server\\jvm.dll). Be careful with relative paths; many installations fail to operate as expected because a path is specified relatively rather than absolutely. 2. From the command line in the lib directory, run the pdrserviceinstall.bat file supplying the following input parameters: o Parameter 1: The name of the Windows service to identify this one The following parameter combinations are valid: pdrserviceinstall.bat pdrbatcher If your installation fails, the first check is to run the application interactively (using the bat file in the Lib directory) and ensure the application runs correctly. Assuming this is the case, the most likely problem is that you are using a 64 bit JRE with the 32 bit service wrapper. It is not possible to mix the 32 bit service wrapper with a 64 bit JVM. The solution is to either install a 32 bit JVM or to use the 64 bit service wrapper. There is no real issue with running the pdrbatcher on a 32 bit JVM as its memory footprint is quite small Uninstalling The procedure for uninstalling the Participant Data Replication Batcher as a Windows service is: 1. From the command line in the \lib directory, run the pdrserviceuninstall.bat file supplying the name of the Windows service as a parameter, as follows: pdrserviceuninstall.bat pdrbatcher 3.9 Batcher extendibility The Participant Data Replication Batcher provides facilities to extend the application for notifications. Doc Ref: TDPD72 v May 2013 Page 9

15 3.9.1 File event handler The file event handler is a hook enabling execution of custom code when file events occur. The process for enabling a file event handler is: 1. Create your own Java class extending the pdrfileeventint interface. 2. Compile your class and ensure placing it in a directory visible to the CLASSPATH environment variable in the pdrenvironment.bat file. 3. Add the definition of your class with the batcher_event_class property in the pdrbatcher.properties file. Potential uses of the handler facility include sending an event message on the successful transfer of data to or from the remote source. Be aware that a poorly-performing event handler can have a significant detrimental effect on overall performance of pdrbatcher. The definition of the Java interface for the file event handler is: package au.com.nemmco.pdr.batcher; /** generic interface for trapping file based events */ public interface pdrfileeventint { /** event method that is called for successful file download processing */ public void fileprocess(string thefilename); /** event method that is called for successful file download processing */ public void fileprocesstotarget(string thefilename, String thepath); } Doc Ref: TDPD72 v May 2013 Page 10

16 4 Configuration Properties The properties file, pdrbatcher.properties, has the configuration relevant for each instance as a set of properties. The following material describes each property in groups. You need to ensure the value for each property is set appropriately to achieve your intended outcome. For Unix systems, the Windows file paths need to be converted to Unix paths. 4.1 Working directory locations Sample entries for working directory locations are: batcher_holding_dir=c:\\pdr\\hold file_log_dir=c:\\pdr\\logs Any referenced directories need to exist Holding directory (batcher_holding_dir) Scope: Global. The batcher_holding_dir property specifies the local directory where the batcher can place files during transfer. During normal Participant Data Replication Batcher operation, the files in the holding directory can be incomplete. After successfully transferring a file from a remote source, a process in the Participant Data Replication Batcher moves the file from the holding directory into the transfer directory. The holding directory needs to be local to the Participant Data Replication Batcher instance for performance and reliability. batcher_holding_dir=c:\\pdr\\temp Transfer log directory (file_log_dir) Scope: Global. The file_log_dir property specifies the local directory where the Participant Data Replication Batcher records the activity of the FTP operations. These log files are generally only used for diagnostic operations. Users are responsible for file management of older files. 4.2 Data source definitions Host sources (batcher_data_sources) Scope: All processing threads. The batcher_data_sources property specifies a comma-delimited list of data sources that the batcher software can transfer files to/from. The data source names are labels that are used to associate other configuration properties with each data source batcher_data_sources=remote,local Host mode (batcher_{ds}_mode) Scope: All processing threads. Doc Ref: TDPD72 v May 2013 Page 11

17 The batcher_{ds}_mode property specifies the communication protocol that is used to transfer information to/from data source {DS}. Valid values are FTP and LOCAL FTP mode will communicate via the FTP protocol and LOCAL will perform local filesystem operations (including remotely mounted filesystems such as a mapped network drive) batcher_remote_mode=ftp Host (batcher_{ds}_host) Scope: All processing threads. The batcher_{ds}_host property specifies the hostname or IP address associated with data source {DS}. The server name can be in the format of a text hostname or an accessible IP address. From a participant perspective via the private MARKETNET network, typically AEMO is for pre-production and for production. batcher_remote_host= Username (batcher_{ds}_username) Scope: All processing threads. The batcher_{ds}_username property specifies the login name for the remote FTP or Windows service associated with data source {DS}. For Windows, if the login name is in multiple parts separated by a slash (/) or backslash (\), remember to use a double backslash (\\) instead. For a typical username of the form nemnet\account, the equivalent value of batcher_{ds}_username property is nemnet\\account. batcher_remote_username=nemnet\\<username> Password (batcher_{ds}_password) Scope: All processing threads. The batcher_{ds}_password property specifies the login password for the remote FTP or Windows service associated with data source {DS}. The value in this property can be encrypted or not, depending on the value of the batcher_{ds}_password_encrypted property. For non-encrypted passwords (that is, the batcher_{ds}_password_encrypted property has the value of false), the batcher_{ds}_password property has the value of the password. For encrypted passwords (that is, the batcher_{ds}_password_encrypted property has the value of true), the batcher_{ds}_password property has the value encrypted using the key in the hash property in the pdr_key.properties file. For generating an encrypted password and securing your system, please see the procedure for updating the 16 digit hexadecimal key in the pdr_key.properties file (in Encrypted parameters on page 7). batcher_remote_password=v9y2jrlhqigoucssl73eua Password encryption (batcher_{ds}_password_encrypted) Scope: All processing threads. Doc Ref: TDPD72 v May 2013 Page 12

18 The batcher_{ds}_password_encrypted property specifies whether the password is encrypted or not: the password being specified in the batcher_{ds}_password property. Valid values are true and false. The default is true. For generating an encrypted password and securing your system, please see the procedure for updating the 16-digit hexadecimal hash key in the pdr_key.properties file (in Encrypted parameters on page 7). batcher_remote_password_encrypted=true Home directory (batcher_{ds}_dir) Scope: Global. The batcher_{ds}_dir property specifies the path to the home directory for data source {DS}. The home directory can be overridden based on the thread number using batcher_thread_{n}_source_dir or batcher_thread_{n}_dest_dir properties. Use a value for the batcher_{ds}_dir property based on the primary intended use for the Participant Data Replication Batcher instance as chosen for running (see 3.6 Running on page 8). Effectively, the value is the default remote directory if not overridden. For FTP, the path is a relative path from the user s remote home directory. batcher_remote_dir=<participant>\\import\\reports\\csvreports For NetBIOS, the path is an absolute path based on fileshare mappings. batcher_remote_dir=n:\\import\\reports\\csvreports Home archive directory (batcher_{ds}_archive_dir) Scope: All processing threads. The batcher_{ds}_archive_dir property specifies an archive directory associated with data source {DS}. When this property is defined, files processed from batcher_{ds}_dir will be moved to this archive directory instead of being deleted. batcher_remote_archive_dir=<participant>\\import\\reports\\csvarchive FTP connection mode (batcher_{ds}_ftp_mode) Scope: All FTP threads. The batcher_{ds}_ftp_mode property specifies the type of connection that the FTP client establishes to a remote FTP server for data source {DS}. Valid values are active and passive. batcher_remote_ftp_mode=active For connecting to AEMO via MarketNet, run FTP in passive mode, as per the default configuration for pdrbatcher FTP connection time-out (batcher_{ds}_ftp_timeout) Scope: Global. The batcher_{ds}_ftp_timeout property specifies the duration in seconds for a transfer operation to finish for data source {DS}. If a transfer operation takes longer than the batcher_{ds}_ftp_timeout time, the thread is deemed to be hanging and terminates the connection with the remote server. Adjust the time-out value so as to not give any errors under normal conditions. A value of 300 seconds, equivalent to 5 minutes, is a good starting point. Doc Ref: TDPD72 v May 2013 Page 13

19 This property is overridden by any batcher_thread_{n}_timeout property for the thread number. batcher_remote_ftp_timeout= Cycle definitions Sample entries for definitions regarding the general cycles within the Participant Data Replication Batcher are: batcher_heartbeat_file=pdrbatcherheartbeat.txt batcher_polling_interval=15 batcher_fail_interval=120 batcher_timeout=900 batcher_inactivity_timeout= Heartbeat file (batcher_heartbeat_file) Scope: Global. The batcher_heartbeat_file property specifies the filename of the file touched with every successful file transfer between remote and local directories. The value is a filename only, so excludes any path. If no definition, or the filename is null, no file is touched. The batcher_heartbeat_file property is optional and, if included, results in the update of the timestamp of the file of this name in the logging directory with each successful download of data. If the specified heartbeat file does not exist, then the Participant Data Replication Batcher creates the file. There is no content to the file. The heartbeat file is a monitoring point only, suitable for an external process to determine if the Participant Data Replication Batcher has ongoing activity. batcher_heartbeat_file=pdrbathb.txt Polling interval (batcher_polling_interval) Scope: Global. The batcher_polling_interval property specifies the duration in seconds between successive searches for new files (when the thread is not otherwise occupied). The default value is 5 (five seconds). This property is overridden by any batcher_thread_{n}_polling_interval property for the thread number. For control of the time between individual files, use the batcher_thread_{n}_pause_interval property. batcher_polling_interval= Wait before reconnect (batcher_fail_interval) Scope: Global. The batcher_fail_interval property specifies the duration in seconds to wait before attempting to reconnect after a remote transfer (FTP or NetBIOS) failure. The default value is 60 (sixty seconds). Doc Ref: TDPD72 v May 2013 Page 14

20 batcher_fail_interval= Batcher inactivity time-out (batcher_inactivity_timeout) Scope: Global. The batcher_inactivity_timeout property specifies the number of seconds after which the pdrbatcher logs an error message if no files have been processed within the time-out period. batcher_inactivity_timeout= Thread definitions A sample of the entries for thread management and definition, assuming data sources of remote and local are defined, is: batcher_threads=4 # thread 1 is high priority for dispatch data batcher_thread_1_local_dir=c:\\pdr\\import\\reports\\csvreports\\dispatch batcher_thread_1_inc=*_dispatch* batcher_thread_1_timeout=30 batcher_thread_1_activity_monitor=y # thread 2 is lower priority for everything except dispatch data batcher_thread_2_local_dir=c:\\pdr\\import\\reports\\csvreports batcher_thread_2_exc=*_dispatch* # extend default timeout to allow for bigger files batcher_thread_2_timeout=120 batcher_thread_2_activity_monitor=y # thread 3 is for financial reports batcher_thread_3_local_dir=c:\\pdr\\import\\financial batcher_thread_3_remote_dir=abcdef\\import\\reports\\pdfnonfinalss batcher_thread_3_mode=ftp # check every half-hour, and delay between each file batcher_thread_3_polling_interval=1800 batcher_thread_3_pause_interval=30 # thread 4 is for submission of bids, with frequent polling batcher_thread_4_local_dir=c:\\pdr\\export\\bids batcher_thread_4_remote_dir=n:\\export\\bids batcher_thread_4_polling_interval=10 batcher_thread_4_timeout=20 batcher_thread_4_direction=send Note: Beware of over-specifying filters, since it is possible to make a seemingly minor adjustment and possibly leave a type of file with a reduced priority. For each process, a useful practice is to always have a catchall thread (that is, no filters) to avoid the situation of missing out on a file (even if it is rare). This also allows the batcher to continue in the event of an unexpected death of a thread Number of threads (batcher_threads) Scope: Global. The batcher_threads property specifies the number of transfer threads to initiate. Doc Ref: TDPD72 v May 2013 Page 15

21 batcher_threads= Number of threads (batcher_threads_active) Scope: Global. The batcher_threads_active property specifies the comma delimited list of threads to instantiate. This property takes precedence over batcher_threads if it is defined the number of transfer threads to initiate. batcher_threads_active=1,3, Data action mode (batcher_thread_{n}_mode) The batcher_thread_{n}_mode property specifies the processing action for this thread. In the property name, the {N} is the thread number being from 1 (one) to the number in the batcher_threads property. Valid values are: o o o CONSUMER files are downloaded from the source, copied to the destination and then deleted from the source. SYNCHRONISE files are copied from the source to the destination. Delete operations at the source are not propagated to the destination. PURGE purge files from source older than a specified retention time. For thread number 1 to perform a consume operation, an example is: batcher_thread_1_mode=consumer Data destination (batcher_thread_{n}_dest) The batcher_thread_{n}_dest property specifies the data source to use as the destination to put data for this processing thread. In the property name, the {N} is the thread number being from 1 (one) to the number in the batcher_threads property. Valid values are defined in the property batcher_data_sources. For thread number 1 to copy information to a defined data destination named local, an example is: batcher_thread_1_dest=local Source directory override (batcher_thread_{n}_source_dir) The batcher_thread_{n}_source_dir property specifies the path to the home directory where the batcher looks for files. In the property name, the {N} is the thread number being from 1 (one) to the number in the batcher_threads property. This property overrides the batcher_{ds}_dir property. For thread number 1 for a path to a thread-specific directory, an example is: Doc Ref: TDPD72 v May 2013 Page 16

22 batcher_thread_1_source_dir=c:\\pdr\\thread1\\import\\reports\\csvreports Destination directory (batcher_thread_{n}_dest_dir) The batcher_thread_{n}_dest_dir property specifies the path to the home directory where the batcher copies information to. In the property name, the {N} is the thread number being from 1 (one) to the number in the batcher_threads property. This property overrides the batcher_{ds}_dir property. For thread number 1 for a path to a thread-specific directory, an example is: batcher_thread_1_remote_dir=<participant>\\import\\reports\\csvreports Subdirectory process list (batcher_thread_{n}_sub_dir) The batcher_thread_{n}_sub_dir property specifies a comma-delimited list of subdirectories to process when transferring files. The subdirectory names are relative to the remote and local directories and must be named consistently at both the local and remote ends. The {N} is the thread number. For thread number 1 for a relative path from the user s FTP home directory, an example is: batcher_thread_1_sub_dir=ss,sr,rsr So, using this example and assuming the batcher_thread_1_local_dir=\\nemrpt and the batcher_thread_1_remote_dir=<participant>\\import\\reports, the result is thread 1 downloads all files from the <PARTICIPANT>\import\reports\ss, <PARTICIPANT>\import\reports\sr and <PARTICIPANT>\import\reports\rsr directories into the \NEMrpt\ss, \NEMrpt\sr and \NEMrpt\rsr directories in a single download operation. Participants can use this facility to reduce the number of threads in the pdrbatcher.properties file Subdirectory recursion (batcher_thread_{n}_dir_recurse) The batcher_thread_{n}_dir_recurse property specifies whether the batcher should recursively process all subdirectories contained under the source directory and replicate those files to the destination directory. The default is N For thread number 1 to recursively process subdirectories, an example is: batcher_thread_1_dir_recurse=y Subdirectory create (batcher_thread_{n}_dir_create) The batcher_thread_{n}_dir_create property specifies whether the batcher should automatically create subdirectories on the destination that exist in the source but not in the destination. The default is N For thread number 1 to automatically create subdirectories as required, an example is: batcher_thread_1_dir_create=y Use this property in conjunction with either batcher_thread_1_sub_dir or batcher_thread_1_dir_recurse. Doc Ref: TDPD72 v May 2013 Page 17

23 Subdirectory destination (batcher_thread_{n}_sub_dir_dest) The batcher_thread_{n}_sub_dir_dest property specifies whether the subdirectory structure existing on the source location should be preserved when files are replicated to the destination location. The default value is Y, meaning that the directory structure will be preserved. The {N} is the thread number. For thread number 1 to collapse source subdirectories into a single directory on the destination, an example is: batcher_thread_1_sub_dir_dest=n Use this property in conjunction with either batcher_thread_1_sub_dir or batcher_thread_1_dir_recurse Allow overwrite flag (batcher_thread_{n}_allow_overwrite) The batcher_thread_{n}_allow_overwrite property specifies the permission for the thread to overwrite a file that already exists in the target directory. The default value ( N ) disallows overwriting. The batcher_thread_{n}_allow_overwrite property can be appropriate where there is a requirement to download a file whose name does not change with each successive publication of new data (for example, the PUBLIC_DISPATCH.zip file). The remote files are still deleted after they have been copied to the local target. An example to just allow thread number 1 to overwrite local files is: batcher_thread_1_allow_overwrite=y Include file mask (batcher_thread_{n}_inc) The batcher_thread_{n}_inc property specifies a list of strings to use as include masks. In the property name, the {N} is the thread number being from 1 (one) to the number in the batcher_threads property. The separator between strings is a comma (,). The filter is the string excluding leading and trailing spaces and including asterisks (*) and question marks (?). The thread identified by the number {N} handles files with any of the strings in the list, even if specified in the exclude file mask. Each question mark represents a single non-specific character. Each asterisk is a series of non-specific characters (that is, zero to many characters). Usually, the beginning and end of each string has an asterisk so the search mask means matching anywhere in the target. If the list is empty (zero length string) or the property not defined, the include mask is all files (*). An example to just receive files with _DISPATCH in the filename, for thread number 1, is: batcher_thread_1_inc=*_dispatch* Exclude file mask (batcher_thread_{n}_exc) The batcher_thread_{n}_exc property specifies a list of strings to use as exclude masks. In the property name, the {N} is the thread number being from 1 (one) to the number in the batcher_threads property. The separator between strings is a comma (,).The filter is the string excluding leading and trailing spaces and including asterisks (*) and question marks (?). The thread identified by the number {N} excludes files with any of the strings in the list, unless specifically identified in the include filter mask. Each question mark represents a single nonspecific character. Each asterisk is a series of non-specific characters (that is, zero to many Doc Ref: TDPD72 v May 2013 Page 18

24 characters). Usually, the beginning and end of each string has an asterisk so the search mask means matching anywhere in the target. If the list is empty (zero length string) or the property not defined, the exclude mask is for no files to be excluded. An example to receive files except those with _DISPATCH anywhere in the filename, for thread number 2, is: batcher_thread_2_exc=*_dispatch* Time-out override (batcher_thread_{n}_timeout) The batcher_thread_{n}_timeout property specifies the duration in seconds for a file to transfer. In the property name, the {N} is the thread number being from 1 (one) to the number in the batcher_threads property. The value for this thread overrides the value for the batcher_timeout property. If a transfer of a file takes longer than the batcher_thread_{n}_timeout time, the thread {N} aborts the transfer. An example to shorten time-out for more urgent and smaller files, for thread number 2, is: batcher_thread_2_timeout=30 Note: For a high priority thread, for example DISPATCH, set this time-out to something much less than the general time-out. The value of the thread time-out needs to be commensurate with the business cycle associated with the file types that this thread manages Polling override (batcher_thread_{n}_polling_interval) The batcher_thread_{n}_polling_interval property specifies the duration in seconds between successive searches for new files (when the thread is not otherwise occupied). In the property name, the {N} is the thread number being from 1 (one) to the number in the batcher_threads property. The default value is the value for the batcher_polling_interval property. The value for this thread overrides the value for the batcher_polling_interval property. All files selected in a poll are processed sequentially before the directory is polled again. To control the file transfer rate, use the batcher_thread_{n}_pause_interval property to set a wait between files and the batcher_thread_{n}_polling_interval property for the wait between polling the directory. An example to lessen the search load (for example, for less urgent files), for thread number 6, is: batcher_thread_6_polling_interval= Pause (batcher_thread_{n}_pause_interval) The batcher_thread_{n}_pause_interval property specifies the delay between each file processed, in seconds. In the property name, the {N} is the thread number being from 1 (one) to the number in the batcher_threads property. The default value of batcher_thread_{n}_pause_interval property is 0 (zero). The batcher_thread_{n}_pause_interval property is useful for throttling the file transfer rate when there are many files in the source directory (as when catching up from an interruption) and the files have a lower priority than another thread. The batcher_thread_{n}_pause_interval property is designed for use in conjunction with the batcher_thread_{n}_throttle property. An example to space out individual file transfers, for thread number 3, is: Doc Ref: TDPD72 v May 2013 Page 19

25 batcher_thread_3_pause_interval= Activity monitor (batcher_thread_{n}_activity_monitor) The batcher_thread_{n}_activity_monitor property specifies whether the Participant Data Replication Batcher thread uses this thread to monitor overall activity. In the property name, the {N} is the thread number being from 1 (one) to the number in the batcher_threads property. Valid values are Y and N. The default value is N, meaning that this thread is not monitored. For configuring to alert if connectivity to AEMO systems is lost, set only threads with a remote directory onto the AEMO file server to Y. batcher_thread_2_activity_monitor=y Files before repoll (batcher_thread_{n}_process_limit ) The batcher_thread_{n}_process_limit property specifies the number of files to process before the Participant Data Replication Batcher thread rereads the source directory for the list of files. In the property name, the {N} is the thread number being from 1 (one) to the number in the batcher_threads property. Valid values of the batcher_thread_{n}_process_limit property are any integer, with zero or less meaning off. If the property is omitted, the default value is off, meaning that rereading the list of files from the source directory happens when the current list is exhausted. The batcher_thread_{n}_process_limit property is designed for use in conjunction with the batcher_thread_{n}_process_order property. batcher_thread_2_process_limit= File processing sequence (batcher_thread_{n}_process_order ) The batcher_thread_{n}_process_order property specifies the processing sequence of files in a source directory. In the property name, the {N} is the thread number being from 1 (one) to the number in the batcher_threads property. Valid values of the batcher_thread_{n}_process_order property are: NAME processing in alphabetic order of filename. NEWEST processing in descending date order; most recent file first. OLDEST processing in ascending date order; oldest file first. The default value is NAME. The batcher_thread_{n}_process_order property is designed for use in conjunction with the batcher_thread_{n}_process_limit property. batcher_thread_1_process_order=oldest File transform operation (batcher_thread_{n}_file_translator ) The batcher_thread_{n}_file_translator property specifies the file transform action to be performed as a given file is being transferred. In the property name, the {N} is the thread number Doc Ref: TDPD72 v May 2013 Page 20

26 being from 1 (one) to the number in the batcher_threads property. Valid values of the batcher_thread_{n}_file_translator property are: NONE pass through, default setting ZIP Zip files from source directory, send compressed file to target UNZIP Unzip files from source directory, send an uncompressed file to target The default value is NONE. batcher_thread_1_file_translator=zip Bandwidth throttling (batcher_thread_{n}_throttle ) The batcher_thread_{n}_throttle property specifies the maximum speed of FTP file transfers in bytes per second. In the property name, the {N} is the thread number being from 1 (one) to the number in the batcher_threads property. Valid values of the batcher_thread_{n}_throttle property are any integer, with zero or less meaning off. If the property is omitted, the default value is off so no bandwidth throttling occurs for this thread. The batcher_thread_{n}_throttle property is designed for use in conjunction with the batcher_thread_{n}_pause_interval property. batcher_thread_3_throttle= Retention time (batcher_thread_{n}_retention_time_mins ) The batcher_thread_{n}_retention_time_mins property specifies the file retention time in minutes associated with a PURGE mode. In the property name, the {N} is the thread number being from 1 (one) to the number in the batcher_threads property. An example to remove all files older than 30 minutes is: batcher_thread_{n}_retention_time_mins= Source archive directory (batcher_thread_{n}_source_archive_dir ) The batcher_thread_{n}_source_archive_dir property specifies the archive directory for the data source. The default is defined by the batcher_{ds}_archive_dir property. Where this value is defined, a CONSUMER processing thread moves processed files to this source archive instead of deleting those files. Where this property is not defined, sources files are deleted once processed. batcher_thread_{n}_source_archive_dir=<participant>\\import\\reports\\csvarchive Retention time (batcher_thread_{n}_retention_time_mins ) The batcher_thread_{n}_retention_time_mins property specifies the file retention time in minutes associated with a PURGE mode. In the property name, the {N} is the thread number being from 1 (one) to the number in the batcher_threads property. An example to remove all files older than 30 minutes is: Doc Ref: TDPD72 v May 2013 Page 21

27 batcher_thread_{n}_retention_time_mins= File purging definitions A sample of the entries for file purging configuration is: # Turn on automated filesystem purging file_purge_configurations=holding -- remove any files in the holding directory more than 1 hour old, run task every 30 minutes file_purge_holding_directory=c:\\pdr\\batcher\\holding file_purge_holding_retention_time_mins=60 file_purge_holding_purge_frequency_mins= File purge configurations (file_purge_configurations) Scope: Global. The file_purge_configurations property specifies a comma-delimited list of configurations to expect within the properties configuration file. The default value is NULL, meaning there are no file system purging configurations. file_purge_configurations=holding,archive meaning that the application expects two configurations, holding and archive File purge directory (file_purge_{c}_directory) Scope: Specific purging configuration. The file_purge_{c}_directory property specifies the local directory to purge for configuration {C} (see File purge configurations (file_purge_configurations) above). file_purge_holding_directory=c:\\pdr\\batcher\\holding file_purge_log_directory=c:\\pdr\\batcher\\log meaning that the holding configuration purges c:\pdr\batcher\holding and the log configuration purges c:\pdr\batcher\log File purge directory retention time (file_purge_{c}_retention_time_mins) Scope: Specific purging configuration. The file_purge_{c}_retention_time_mins property specifies the maximum age of files to be retained in the directory. Any files created before this offset from the current system time are purged. file_purge_holding_retention_time_mins=60 meaning that the holding configuration purges any files created more than 60 minutes before the current system time File purge frequency (file_purge_{c}_purge_frequency_mins) Scope: Specific purging configuration. The file_purge_{c}_purge_frequency_mins property specifies how often to run the purge operation for the specified configuration. The default is daily, once every 24 hours. Doc Ref: TDPD72 v May 2013 Page 22

28 file_purge_holding_purge_frequency_mins=30 meaning the holding configuration runs every 30 minutes Purge include file mask (file_purge_{c}_inc) Scope: Specific purging configuration. The file_purge_{c}_inc property specifies a list of strings to use as include masks. In the property name, the {C} is the purge configuration name as defined in the file_purge_configurations property. The separator between strings is a comma (,). The filter is the string excluding leading and trailing spaces and including asterisks (*) and question marks (?). The configuration identified by the name {C} handles files with any of the strings in the list, even if specified in the exclude file mask. Each question mark represents a single non-specific character. Each asterisk is a series of non-specific characters (that is, zero to many characters). Usually, the beginning and end of each string has an asterisk so the search mask means matching anywhere in the target. If the list is empty (zero length string) or the property not defined, the include mask is all files (*). An example to just purge files with _DISPATCHIS_ in the filename, for configuration holding, is: file_purge_holding_inc=*_dispatchis_* Purge exclude file mask (file_purge_{c}_exc) Scope: Specific purging configuration. The file_purge_{c}_exc property specifies a list of strings to use as exclude masks. In the property name, the {C} is the purge configuration name as defined in the file_purge_configurations property. The separator between strings is a comma (,).The filter is the string excluding leading and trailing spaces and including asterisks (*) and question marks (?).The configuration identified by the name {C} excludes files with any of the strings in the list, unless specifically identified in the include filter mask. Each question mark represents a single nonspecific character. Each asterisk is a series of non-specific characters (that is, zero to many characters). Usually, the beginning and end of each string has an asterisk so the search mask means matching anywhere in the target. If the list is empty (zero length string) or the property not defined, the exclude mask is for no files to be excluded. An example to purge files except those with _DISPATCHIS_ anywhere in the filename, for configuration holding, is: file_purge_holding_exc=*_dispatch_is* 4.6 Configuration files The Participant Data Replication Batcher uses two files to control the configuration, with the defaults being the pdr_log4j.properties file and the pdrbatcher.properties file. For separate installations, the two configuration files are in different directories so are distinct. To configure the Participant Data Replication Batcher for multiple instances from a single installation, both of the configuration files need an alternative for each instance other than the first. Each file is set using a system property. However, the setting cannot be in the files themselves and must be set on the command line initiating the instance (see the pdrbatcher.bat or pdrbatcher.sh files). It is probably best to create separate initiation scripts (for example, pdrbatcher2.bat or pdrbatcher2.sh files). For example (for Windows): Doc Ref: TDPD72 v May 2013 Page 23

29 java -cp "%CLASSPATH%" Dlog4j=pdr_log4j2.properties Dconfig=pdrBatcher2.properties au.com.nemmco.pdr.batcher.apppdrbatcher Log4j properties file (log4j) The log4j property specifies the name of the file to use for the log4j properties file. The default, if not specified, is the pdr_log4j.properties file. The directory for the file is always the same as the Participant Data Replication Batcher application. The log4j property must be specified in the instance initiation (that is, in the java line). For more information on the purpose of this property, see 6.7 Multiple instances on page 27 and 4.6 Configuration files on page 23. For example (for Windows): java -cp "%CLASSPATH%" -Dlog4j=pdr_log4j2.properties Batcher configuration properties file (config) The config property specifies the name of the file to use for the pdrbatcher configuration properties file. The default, if not specified, is the pdrbatcher.properties file. The directory for the file is always the same as the Participant Data Replication Batcher application. The config property must be specified in the instance initiation (that is, in the java line). For more information on the purpose of this property, see 4.6 Configuration files on page 23. For example (for Windows): java -cp "%CLASSPATH%" Dconfig=pdrBatcher2.properties 4.7 Index of properties Doc Ref: TDPD72 v May 2013 Page 24

30 5 batcher_{ds}_archive_dir, 13 batcher_{ds}_dir, 13 batcher_{ds}_ftp_mode, 13 batcher_{ds}_ftp_timeout, 13 batcher_{ds}_host, 12 batcher_{ds}_mode, 11 batcher_{ds}_password, 12 batcher_{ds}_password_encrypted, 7, 12 batcher_{ds}_username, 12 batcher_data_sources, 11 batcher_fail_interval, 14 batcher_heartbeat_file, 14 batcher_holding_dir, 11 batcher_inactivity_timeout, 15 batcher_polling_interval, 14 batcher_thread_{n}_activity_monitor, 20 batcher_thread_{n}_allow_overwrite, 18 batcher_thread_{n}_dest, 16 batcher_thread_{n}_dest_dir, 17 batcher_thread_{n}_dir_create, 17 batcher_thread_{n}_dir_recurse, 17 batcher_thread_{n}_exc, 7, 18 batcher_thread_{n}_file_translator, 20 batcher_thread_{n}_inc, 7, 18 batcher_thread_{n}_mode, 16 batcher_thread_{n}_pause_interval, 19 batcher_thread_{n}_polling_interval, 19 batcher_thread_{n}_process_limit, 20 batcher_thread_{n}_process_order, 20 batcher_thread_{n}_retention_time_mins, 21 batcher_thread_{n}_source_archive_dir, 21 batcher_thread_{n}_source_dir, 16 batcher_thread_{n}_sub_dir, 17 batcher_thread_{n}_sub_dir_dest, 18 batcher_thread_{n}_throttle, 21 batcher_thread_{n}_timeout, 19 batcher_threads, 15 batcher_threads_active, 16 config, 24 Data source batcher_data_sources, 11 batcher_thread_{n}_dest, 16 batcher_thread_{n}_mode, 16 Destination directory batcher_{ds}_dir, 13 batcher_thread_{n}_dest_dir, 17 file_log_dir, 11 file_purge_{c}_directory, 22 file_purge_{c}_exc, 23 file_purge_{c}_inc, 23 file_purge_{c}_purge_frequency_mins, 22 file_purge_{c}_retention_time_mins, 22 file_purge_configurations, 22 log4j, 24 Logging. See file_log_dir Overwrite batcher_thread_{n}_allow_overwrite, 18 Password batcher_{ds}_password, 12 batcher_{ds}_password_encrypted, 7, 12 batcher_{ds}_username, 12 Polling interval batcher_polling_interval, 14 batcher_thread_{n}_polling_interval, 19 Source directory batcher_thread_{n}_source_dir, 16 Time-out batcher_{ds}_ftp_timeout, 13 batcher_inactivity_timeout, 15 batcher_thread_{n}_timeout, 19 Doc Ref: TDPD72 v May 2013 Page 25

31 6 Possible Architectures using pdrbatcher Refer to the MMS Data Interchange Guide for more details on architectures for using Participant Data Replication Batcher for communicating data to and from AEMO. 6.1 FTP or NetBIOS A parameter on the command line initiating the Participant Data Replication Batcher determines the default protocol for the Participant Data Replication Batcher instance. The \lib directory of the distribution has a batch file for each of the possible protocols. For connecting to AEMO s MarketNet, use FTP. The Participant Data Replication Batcher can transfer from one local directory to another. NetBIOS makes the most sense for this type of use. 6.2 Single source using a single thread The simplest configuration of the Participant Data Replication Batcher application has just one thread transferring all files from a source directory to a local directory. 6.3 Single source using multiple threads It is possible to set up multiple threads in the Participant Data Replication Batcher to handle specific file subsets from a single directory, allowing for adjustment of priority for each subset. It is wise to consider having one thread without filtering just to be sure of transferring all files, such as when maintenance might create non-contiguous filter sets. 6.4 Single source to multiple local directories The Participant Data Replication Batcher can copy files from a single source directory to multiple local directories, by specifying multiple local directories for a thread (using a comma-separated list). This facility is useful for arranging for parallel loading of data (say, for two systems requiring the same data feed). 6.5 Multiple sources to multiple local directories The Participant Data Replication Batcher application handles one source directory for each thread. Each thread is independent of other threads, provided the directories used by each thread are distinct. Handling multiple sources is thus having more of the simplest configuration of the Participant Data Replication Batcher application (just one thread transferring all files from a source directory to a local directory). 6.6 Bidirectional communication Receiving and sending data files in the same instance of Participant Data Replication Batcher application is practical. For example, such a set up is appropriate for receiving Dispatch data from AEMO, and submitting bid files to AEMO. This arrangement is a variation of the multiple sources to multiple local directories (above), with the twist that at least one thread is sending and one thread is receiving. To add prioritising of files, either direction (or both) can have multiple threads, as outlined above for a single source using multiple threads. Doc Ref: TDPD72 v May 2013 Page 26

32 6.7 Multiple instances It is possible to run multiple instances of the Participant Data Replication Batcher to service multiple business purposes. Take great care with any possible overlaps to prevent the risk of any instance becoming confused (or all of them)! Note: Only run one batch file at a time for any set of working directories. The files in the working directories store the current state for a Participant Data Replication Batcher instance. If the multiple instances are run on separate machines, each machine has its own installation. It is also possible to have multiple installations on a single machine, providing all working directories are different. If the multiple instances are run on a single machine, you may choose to run from a single installation even though the configurations need to be different. To configure the Participant Data Replication Batcher for multiple instances from a single installation, be aware of the following considerations. The Participant Data Replication Batcher uses two files to control the configuration, being the pdr_log4j.properties file and the pdrbatcher.properties file. Each instance of the Participant Data Replication Batcher must be independent of the other (since the state of each instance is maintained in the working directories). To run another instance means alternative properties configuration files must exist and the instance initiation needs to reference the alternative files (using system properties - see 4.6 Configuration files on page 23). Note that the instances share the pdr_key.properties file. One reason to configure the Participant Data Replication Batcher for multiple instances from a single installation can be to set up a disaster recovery (DR) pdrbatcher which has a main configuration to point to the production pdrbatcher to extract report files for the DR environment. In this case, you can set up a separate invocation script pre-configured to launch the DR pdrbatcher against the AEMO production system to bring up the DR site for production duties quickly. Doc Ref: TDPD72 v May 2013 Page 27

33 7 Glossary Term CLASSPATH CSV DI FTP hexadecimal digits IP address Java log4j MarketNet MMS NER NetBIOS Rules ZIP Explanation The CLASSPATH is an environment variable telling the Java compiler (javac.exe) where to look for class files to import or telling the Java interpreter (java.exe) where to find class files to interpret. Comma-separated values. Data interchange. File transfer protocol: a very common method of moving files between two Internet sites. characters 0-9 and A-F only Internet protocol address is a unique string of numbers identifying a computer on the Internet. These numbers are usually shown in four groups separated by periods (for example, ). Oracle's trademark for a set of technologies to create and safely run software programs in both stand-alone and networked environments. A Java logging facility, enabling logging at runtime without modifying the application binary. a private computer network containing participants and AEMO. Market Management System; software, hardware, network and related processes to implement the National Electricity Market (NEM). National Electricity Rules; also often just called the Rules. Network Basic Input Output System, an API that augments the DOS BIOS by adding special functions for local area networks (LANs). Almost all LANs for PCs are based on the NetBIOS protocol. AEMO s file servers exclude support for NetBIOS. National Electricity Rules (NER). A compressed file, containing one or more files. Doc Ref: TDPD72 v May 2013 Page 28

34 8 Needing Help? Getting Java errors? Check release of pdrbatcher is the latest. Check Java is Oracle s. Confirm Java version is currently supported by Oracle. Check the CLASSPATH environment variable. Not transferring files, or timing out? Check network configuration and performance, including firewalls. Try disconnecting and reconnecting. Modify properties to suit available network conditions. Try trickle. Check with local IT support. Not connecting or locked out? Check credentials (passwords do expire). Check successful connection using an FTP client from the same environment as where pdrbatcher runs. For Data Interchange with AEMO, ensure using passive FTP, since active FTP is unreliable. Observing abnormal behaviour? Check release of pdrbatcher is the latest. Check logs. Check with local IT support. Check reproducible in a test environment. Attempt a restart. Some sample errors are: Network dropout, so check all above help steps. ERROR 02 Feb 10:32:25 [CONSUMER_2] au.com.nemmco.pdr.batcher.processor.pdrbatcherhandlerint - Detected problem processing files from \wherever\import\reports\csvreports in thread ID 2: Unable to borrow FTP connection from object pool: Connection timed out: connect When requesting support for the supplied software, provide at least (after checking each thoroughly): Version of pdrbatcher. Logs showing the abnormal behaviour (compressed in.zip format). pdrbatcher.properties file. Anything relevant for related products (for example, pdrloader, Replication Manager). Diagram of architecture. Watch out for market notices and bulletins about MarketNet changes potentially affecting access, but unplanned events do occur. In preparation for handling network and system failures, plan and test alternate configurations (including failovers). Doc Ref: TDPD72 v May 2013 Page 29

35 To suggest corrections to this document or to request AEMO support, please contact AEMO s Helpdesk Telephone: (option 2), helpdesk@aemo.com.au. Doc Ref: TDPD72 v May 2013 Page 30