PARTICIPANT DATA REPLICATION LOADER USER GUIDE

Transcription

1 PARTICIPANT DATA REPLICATION LOADER USER GUIDE PREPARED BY: IMT Documentation Team Version No: 3.00 Date: 3 September 2012 Draft last updated 22 March 2013 Australian Energy Market Operator Ltd ABN inlo@aemo.com.au NEW SOUTH WALES QUEENSLAND SOUTH AUSTRALIA VICTORIA AUSTRALIAN CAPITAL TERRITORY TASMANIA

2 Disclaimer In this document, references to the software are to the Participant Data Replication Loader, unless specified otherwise. References to associated materials include this document, Participant Data Replication Loader User Guide. This document is made available to you on the following basis: 1. Purpose This Participant Data Replication Loader User Guide (Guide) has been produced by the Australian Energy Market Operator Limited (AEMO) to provide information about the Participant Data Replication Loader, as at the date of publication. 2. No substitute This Guide is not a substitute for, and should not be read in lieu of, the National Electricity Law (NEL), the National Electricity Rules (Rules) or any other relevant laws, codes, rules, procedures or policies. Further, the contents of this Guide do not constitute legal or business advice and should not be relied on as a substitute for obtaining detailed advice about the NEL, the Rules, or any other relevant laws, codes, rules, procedures or policies, or any aspect of the national electricity market or the electricity industry. 3. No Warranty While AEMO has used due care and skill in the production of this Guide, neither AEMO, nor any of its employees, agents and consultants make any representation or warranty as to the accuracy, reliability, completeness or suitability for particular purposes of the information in this Guide. 4. Limitation of liability - To the extent permitted by law, AEMO and its advisers, consultants and other contributors to this Guide or the Participant Batcher software (or their respective associated companies, businesses, partners, directors, officers or employees) shall not be liable for any errors, omissions, defects or misrepresentations in the information contained in this Guide or the Participant Batcher software, or for any loss or damage suffered by persons who use or rely on such information (including by reason of negligence, negligent misstatement or otherwise). If any law prohibits the exclusion of such liability, AEMO s liability is limited, at AEMO s option, to the re-supply of the information, provided that this limitation is permitted by law and is fair and reasonable. Note References to software are to the Participant Data Replication Loader (software distribution file name), unless otherwise specified. References to associated materials include this Guide. Your use of the software is also subject to the following: (a) General - The software may not necessarily be of production quality and may not be fully tested. It is provided as an example only. If the provided functionality is insufficient for the use of a user, users have to independently develop or acquire software to meet their needs. (b)no Warranty A user runs the software or uses the materials at its own risk. AEMO does not represent or warrant that the software and materials are complete, free from errors, reliable and fit for particular purposes. Nor does AEMO represent or warrant that the software is free from viruses and other programs which may affect other software or Australian Energy Market Operator Ltd ABN inlo@aemo.com.au NEW SOUTH WALES QUEENSLAND SOUTH AUSTRALIA VICTORIA AUSTRALIAN CAPITAL TERRITORY TASMANIA

3 systems. The software and associated materials are provided on the basis that users proposing to use or rely on the software or associated materials undertake responsibility for independently verifying and checking the accuracy, completeness, reliability or suitability of the software or associated materials. Copyright Trademark Notices Copyright 2013 Australian Energy Market Operator Limited. All rights reserved. Microsoft, Windows and SQL Server are either registered trademarks or trademarks o Oracle and Java are registered trademarks of Oracle and/or its affiliates. Distribution Available to the public. Prepared by IMT Documentation Team Last update Thursday, 25 July :13 PM Notes Added section Needing Help?, updated references to AEMO Information and Support Hub, and freshened the layout and document organisation. Updated for v7.3 release Documents made obsolete The release of this document changes only the version of Participant Data Replication Loader User Guide. Further Information For further information, please visit AEMO s website or contact: AEMO Information and Support Hub Phone: 1300 AEMO 00 ( ) and follow the prompts. supporthub@aemo.com.au PAGE 3

4 Table of Contents 1 INTRODUCTION Overview Who is this software for? What this user manual covers Assumed knowledge Requirements Resources Outline of functions 8 2 SETUP File subscriptions Batcher Downloading the application Environment prerequisites Choosing an architecture 12 3 INSTALLATION Database Application Configuration considerations Minimal configuration of the.properties file Running Logging configuration Running as a Windows service Using NT database authentication 28 PAGE 4

5 3.9 Loader configuration Loader extendability 31 4 CONFIGURATION PROPERTIES Database Folder locations File definitions Automated recovery definitions Thread definitions File purging definitions Configuration files Index of properties 52 5 DATABASE TABLES Entity-relationship diagram Table descriptions Examples 62 6 NEEDING HELP? Java messages File transfer PDR control tables MMS Data Model tables Other Requesting AEMO assistance 68 PAGE 5

6 1 Introduction 1.1 Overview The Participant Data Replication Loader application (pdrloader) application parses CSV files to load AEMO-supplied data into a local database instance implementing the MMS Data Model. Participant Data Replication Loader is part of the set of changes implementing Change Notice 833, collectively called Data Interchange. The Participant Data Replication Loader (plus other components in the communication path between AEMO and Participants) allows recovery from various failures, including network failure (causing missing data), hardware failure (requiring failover to a possibly out-of-synchronisation loader and database), and system failure (causing build-up of data at various points). Recovery requires a message facility from the loader back to the data source (at AEMO), implemented entirely with CSV files. Given the asynchronous nature of the messages and responses, the Participant Data Replication Loader records status data in DBMS tables. 1.2 Who is this software for? This software is for each participant in the National Electricity Market interfacing with the EMMS system by the CSV file delivery mechanism in Data Interchange. 1.3 What this user manual covers This manual covers the setting up and use of the Participant Data Replication Loader, version Assumed knowledge This document assumes you have knowledge of: The Java application environment. The operating system you are using. How the NEM systems operate Glossary TERM CLASSPATH CSV DI EXPLANATION The CLASSPATH is an environment variable telling the Java compiler (javac.exe) where to look for class files to import or telling the Java interpreter (java.exe) where to find class files to interpret Comma separated values Data Interchange 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 6

7 TERM FTP IP address Java JDBC log4j EMMS MarketNET NER Rules EXPLANATION File Transfer Protocol: a very common method of moving files between two Internet sites Internet Protocol address is a unique string of numbers identifying a computer on the Internet. These numbers are usually shown in four groups separated by periods (for example, ). Oracle s trademark for a set of technologies to create and safely run software programs in both stand-alone and networked environments Java DataBase Connectivity: an industry standard for database-independent connectivity between the Java platform and a wide range of databases. The JDBC interface provides a calllevel API for SQL-based database access A Java logging facility enabling logging at runtime without modifying the application binary. Market Management System; software, hardware, network and related processes to implement the National Electricity Market (NEM); a AEMO department responsible for maintaining the system a private computer network containing participants and AEMO National Electricity Rules; also often just called the Rules National Electricity Rules (NER) ZIP Table 1: Glossary of terms 1.5 Requirements A compressed file, containing one or more files The software runs under the JRE 7 runtime engine (with the possible addition of the JDK s jre\bin\server folder) and requires a JDBC driver and handler suitable for the target database. The distribution file from AEMO contains JDBC drivers for Oracle and SQL Server for 32-bit environments. The Participant Data Replication Loader can run on both Windows and unix-like operating systems. 1.6 Resources The Participant Data Replication Loader software is available on the "Data Interchange" web page ( Registration/Understanding-Energy-Market-Information-Systems/Data-Interchange). If not running Participant Data Replication Loader as a Windows service, install JRE 7 (if it not available already). For running Participant Data Replication Loader as a Windows service and if not already available, you can install JDK 7, or JRE 7 plus the jre\bin\server from JDK 7. If you only install JRE 7, the VM specified in the ServiceInstall script is not installed by default (%JRE%\bin\server\jvm.dll). JDK 7 does include the Java HotSpot Server VM. The under heading Redistributable JDK(TM) Files and subheading jre\bin\server\, says On Microsoft 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 7

8 Windows platforms, the JDK includes both the Java HotSpot(TM) Server VM and Java HotSpot Client VM. However, the Java SE Runtime Environment for Microsoft Windows platforms includes only the Java HotSpot Client VM. Those wishing to use the Java HotSpot Server VM with the Java SE Runtime Environment may copy the JDK's jre\bin\server folder to a bin\server folder in the Java SE Runtime Environment. Software vendors may redistribute the Java HotSpot Server VM with their redistributions of the Java SE Runtime Environment. (Viewed 15 March 2013.) Other relevant documents on the "Data Interchange" web page ( Energy-Market-Information-Systems/Data-Interchange) include: EMMS Data Interchange Guide. Documents under the headings PDR Loader and PDR Batcher. The following resources may be useful in addition to the information contained in this manual: Java download Oracle JDBC Driver FAQ Microsoft JDBC Driver for SQL Server Overview of the JDBC Driver for SQL Server Configuring logging (log4j) Outline of functions The Participant Data Replication Loader application parses AEMO-supplied CSV files into a local participant database. The interface to the database is SQL, so the database can be any accepting a standard SQL interface (for example, Oracle and SQL Server). The Participant Data Replication Loader application polls a local folder and processes data from files into the local database conforming to the MMS Data Model. If the recovery features are enabled, periodically the Participant Data Replication Loader application sends a manifest request file to AEMO regarding the state of the Participant Data Replication Loader. AEMO sends a manifest report in response to the request. Participant Data Replication Loader compares the manifest report with the local database to identify any missing data, and, if necessary, sends a request to AEMO for any missing files. The catching-up process is at a lower priority than normal file handling. The application design of the loader is a state machine with the file locations controlling the state. This means that, on restarting after a stop for any reason, the application automatically recovers by using the stored state in the local directories. Each of the local 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 8

9 directories contains the results of the file processing, and a process in the application periodically moves files to the appropriate folder. The design of the Participant Data Replication Loader application assumes the access to the local directories is highly reliable and efficient. Hence, the recommendation is to set up the structure of the application with the local directories hosted on the machine running the application. If space is a problem, the recommendation is to use an ancillary process to empty the various local subdirectories by move or delete. The application design of the loader uses several tables in the database for control, mapping and audit. Each CSV record updates the database according to the entries in the set of tables starting with PDR. 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 9

10 2 Setup 2.1 File subscriptions Before installing the Participant Data Replication Loader application, ensure you have subscribed to the required files from AEMO to supply the data you require for the MMS Data Model. Ensure you have at least two days of files available, so you can be sure you have not missed any data. To establish file subscriptions, use the Data Subscription application, part of EMMS web portal (EMMS > Data Interchange >Data Subscription). Ensure your subscriptions are only for the MMS Data Model database, otherwise you need to filter out the files from the data flow into the Participant Data Replication Loader application. 2.2 Batcher The Participant Data Replication Loader requires a data feed and, for automatic recovery, a data delivery channel to AEMO. AEMO supplies Participant Data Replication Batcher application for this purpose. For more details, refer to the EMMS Data Interchange Guide on the "Data Interchange" web page ( Market-Registration/Understanding-Energy-Market-Information-Systems/Data-Interchange). 2.3 Downloading the application For a new standardised installation, refer to the titles beginning with Participant Data Replication Loader GUI Installer on the "Data Interchange" web page ( Energy-Market-Information-Systems/Data-Interchange) as an easy-to-install alternative Participant Data Replication Loader The latest version of the Participant Data Replication Loader application is available on "Data Interchange" web page ( Registration/Understanding-Energy-Market-Information-Systems/Data-Interchange) under the heading PDR Loader. Choose the file appropriate for your database supplier from the titles starting with Participant Data Replication Loader for. Decompress the ZIP file to a base folder (C:\ recommended), retaining the folder structure recorded in the distribution file. The decompression creates a \pdr\loader\ folder with eight subdirectories MMS Data Model configuration The Participant Data Replication Loader requires configuration to suit your installed version of the MMS Data Model and the EMMS environment. With default values for properties, any missing configuration is automatically retrieved from EMMS. 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 10

11 2.3.3 Replication Manager A companion product, Replication Manager, is also very useful to control the operation of the Participant Data Replication Loader (see Replication Manager on web page "Data Interchange" ( Registration/Understanding-Energy-Market-Information-Systems/Data-Interchange). 2.4 Environment prerequisites The Participant Data Replication Loader needs: Access to a local database (read and write) in which to install the PDR* tables for controlling the loader process, plus access to all the target tables (MMS Data Model). A set of local working directories Local database To interact with the local database, the Participant Data Replication Loader requires the following items: Database connectivity details server, username and password. A JDBC driver with user and/or technical documentation sufficient to configure the loader for connectivity and optimal performance. Each target database requires a unique parameter set for best performance. Please consult the documentation supplied with the JDBC driver being used to determine what parameters may be required Working directories On the local machine, the Participant Data Replication Loader requires working directories for the loader processes. A default set creates as part of the decompression of the distribution file. Check each folder and delete each file with the name delete.me. Additionally, a purging process needs to be put in place such that the Loader working directories do not consume the free disk space on the server Configuring loader The Participant Data Replication Loader needs information about the data in the files it receives for processing, including what columns in what tables in the MMS Data Model the data is to be placed. The configuration information is part of the MMS Data Model. With default properties, the configuration information downloads automatically when needed. 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 11

12 2.5 Choosing an architecture Source The Participant Data Replication Loader application handles two source directories for each running instance. The primary source folder is for transactions and has priority over the secondary folder. The secondary source folder is for trickle input, occurring only when there is no input in the primary folder. Each of these sources is a single folder. Other than the priority, both source directories are equivalent and form a common input source to the Participant Data Replication Loader. (Many of the diagrams in this document show a single source for the Participant Data Replication Loader, reflecting the common processing from both directories.) The following diagram shows the two file flows from AEMO to the Participant Data Replication Loader, where they effectively become one input with different priorities. Remote Server Local Server NEM Participant File Share (data) Batcher Local Directory Local Directory AEMO CSV files Source Directory (file_reports_dir) Source Directory (file_trickle_reports_dir) Loader Participant Production Database NEM Participant File Share (trickle) AEMO CSV files The purpose of the two directories is to ensure the processing of catch-up transaction files does not hold up processing of current data Multiple instances It is possible to run multiple instances of the Participant Data Replication Loader to service multiple participant databases. In general, only one instance of the Participant Data Replication Loader should be run against a single target database. If the multiple instances are run on separate machines, each machine has its own installation. It is also possible to have multiple installations on a single machine, providing all working directories are different. If the multiple instances are run on a single machine, you may choose to run from a single installation even though the configurations need to be different. To configure the Participant Data Replication Loader for multiple instances from a single installation, be aware of the following considerations. The Participant Data Replication Loader uses two files to control the configuration, being the pdr_log4j.properties file and the pdrloader.properties file. Each instance of the Participant Data Replication Loader must be independent of the other (since the state of each instance is maintained in the working 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 12

13 directories and the database). To run another instance means alternative properties configuration files must exist and the instance initiation needs to reference the alternative properties configuration (using system properties see File purging definitions on page 48). Note that the instances share the pdr_key.properties file. Refer to MMS Data Interchange Guide for more details on setting up of multiple instances. 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 13

14 3 Installation 3.1 Database The target database to which the Participant Data Replication Loader is to load data needs to be configured with ten (10) controlling tables to provide the mapping of the CSV data to database tables, plus logging of transfers. The table names are: PDR_REPORT_AUDIT. PDR_REPORT_AUDIT_SUMMARY. PDR_REPORT_RECORDS. PDR_REPORT_RECORD_MAPPING. PDR_REPORT_TYPE_CONFIG. PDR_REPORT_TRANSFORMS. PDR_REPORT_ROW_FILTERS. PDR_CONFIG. PDR_MANIFEST_LOG. PDR_REQUEST_LOG. For more details about the contents of these tables, see Database Tables on page 53. The steps to install the controlling tables for each of the supported database platforms follow Oracle database 1. Log into database as schema owner. 2. Run SQL scripts contained in the lib folder in the following order: i. oracle_create_pdr.sql, to create the Participant Data Replication Loader configuration tables. ii. populate_parser_config_metadata.sql, to insert configuration data required by the application and to allow delivery of future updates by files. 3. Commit database transactions SQL Server 1. Log into database as schema owner. 2. Run SQL scripts contained in the lib folder in the following order. i. sqlserver_create_pdr.sql, to create the Participant Data Replication Loader configuration tables. 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 14

15 ii. iii. sqlserver_create_pdr_sp_msql.sql, to create a stored procedure. populate_parser_config_metadata.sql, to insert configuration data required by the application and to allow delivery of future updates by files. 3. Commit database transactions. 3.2 Application 1. If not already available, install JRE 7. For more details, see Resources on page 7. TIP If you are unsure of the version of Java installed on your system, enter java version at the command prompt or unix-like shell to display the version of Java. 2. Unzip the.zip file (as relevant to your choice of database) into a folder (C:\ recommended). 3. Remove the delete.me files. 4. Edit the pdr_key.properties file in the \lib sub-folder and replace the default value with a 16 digit hexadecimal string of your own choice. TIP Before starting to configure your installation, ensure you update the 16 digit hexadecimal key in the pdr_key.properties file. Changing the encryption key (hash) ensures any encrypted passwords in your properties file cannot be decrypted using the default and publicly available hash. 5. Edit the pdr_log4j.properties file in the \lib sub-folder and update the log4j.appender.fileout.file parameter to redirect the log output file to the desired location. TIP Windows style directories using the backslash operator need to be escaped to be read properly from the properties configuration file. An example of a Windows path for this parameter follows: log4j.appender.fileout.file=c:\\pdr\\log\\pdr.log Unix-like paths using the forward slash operator, /, may be entered directly and do not require an escape sequence. An example of a unix-like path for this parameter follows: log4j.appender.fileout.file=/home/pdr/log/pdr.log 6. Edit the pdrenvironment file to ensure that the PATH environment variable includes a reference to the folder where the Java executable is installed on your system. Ensure 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 15

16 the changes are made to the relevant file for your operating system (.BAT file for Windows and.sh file for unix-like systems). TIP The Participant Data Replication Loader uses JRE 7. If you have earlier versions, ensure you change this batch file to reference the appropriate directories for Java. Placing the folder of the Java install at the start of the PATH environment variable ensures the loader always runs against the correct version of Java. 7. For unix-like systems, the following two steps are necessary due to the specific environment in the overall build process for the Participant Data Replication Loader. o In the shell scripts (.SH files), each line is terminated with a carriage return and line feed. If your system cannot accept such line termination, remove the carriage returns (for example, as follows, in the \Lib folder): cat pdrloader.sh tr d \r > pdrloader.sh2 mv pdrloader.sh2 pdrloader.sh cat pdrenvironment.sh tr d \r > pdrenvironment.sh2 mv pdrenvironment.sh2 pdrenvironment.sh cat pdrpasswordencrypt.sh tr d \r > pdrpasswordencrypt.sh2 mv pdrpasswordencrypt.sh2 pdrpasswordencrypt.sh o The shell scripts (.SH files) do not have execute permission for the unix-like operating system. From within the \Lib folder, allow execution of the shell scripts as follows: chmod 750 *.sh TIP If the distribution file is unzipped to the C:\ sub-folder on a Windows system accessing an Oracle database, the sample application starts by executing pdrloader.bat with just the editing of hash in pdr_key.properties file and db_user, db_password and db_url, in the pdrloader.properties file. To observe a file being loaded, copy an existing.csv file (zipped or unzipped) to the Reports folder. Exercise care to ensure no effect on production operations by running this application Folder structure created Decompressing the distribution file creates a folder (\pdr\loader) with eight sub-directories. One folder (\Lib) contains the application and the other seven are working directories for the Participant Data Replication Loader: Archive: where the loader places successfully processed files. 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 16

17 Error: where the loader places files it cannot load into the database. Log: contains log files associated with the loader processing. Reports: where the loader searches for files to load into the local database. Reports/Queue: where the loader places files that are queued for processing. ReportsTrickle: where the loader searches for trickled files to load into the database when there are no files in the \Reports folder. Request: where the loader places request files to send to AEMO. Temp: where the loader can place temporary work files. The application files in the \Lib folder include: The application files, being five.jar files: AppPdrLoader.jar, commons-codec-1.4.jar, commons-pool jar, apache-log4j-extras-1.1.jar and log4j jar. For the JDBC driver of choice, additional.jar files (for example, for Oracle, ojdbc6.jar). To provide parameters for the running instance, three.properties files (pdrloader.properties, pdr_log4j.properties and pdr_key.properties). To run the application, one.bat file (pdrloader.bat) for Windows and one.sh file (pdrloader.sh) for unix-like systems. The definitions of the environment variables required to run the application, being one.bat file (pdrenvironment.bat) for Windows and one.sh file (pdrenvironment.sh) for unix-like systems. To generate a support dump file, one.bat file (pdrloadersupportdump.bat) for Windows and one.sh file (pdrloadersupportdump.sh) for unix-like systems. Five files (pdrserviceinstall.bat, pdrserviceuninstall.bat, WinRun4J.exe, WinRun4J64.exe, and WinRun4J.ini) to support the running of the application as a Windows service. SQL files to set up the database: o For Oracle, being oracle_create_pdr.sql and populate_parser_config_metadata.sql. o For SQL Server, being sqlserver_create_pdr.sql, sqlserver_create_pdr_sp_msql.sql and populate_parser_config_metadata.sql. SQL files to update an existing database: o For Oracle, being oracle_alter_pdr.sql and populate_parser_config_metadata_delta.sql. 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 17

18 o For SQL Server, being sqlserver_alter_pdr.sql and populate_parser_config_metadata_delta.sql. Each folder, other than \Lib folder, has a placeholder file to cause the creation of each folder during the unzipping process. The placeholder file, named delete.me, needs to be removed from each folder. Before attempting to run, edit the two.properties files (pdrloader.properties and pdr_key.properties) to suit your situation (see Configuration considerations below) and ensure the database has the control and target tables configured. TIP When setting up the application, the folder structure does not have to be identical to the arrangement detailed in the above table. By modifying the.properties file, the installer can distribute the directories over different locations. 3.3 Configuration considerations File masks In order to handle file priority efficiently, the Participant Data Replication Loader application has multiple threads. Each thread is responsible for processing certain files based on file masks specified by the user. The multi-threading combined with the file mask feature allows higher priority handling of selected file types as well as allowing concurrent transfers and loads. For example, high priority threads (that is, with short timeouts) are appropriate for AEMO dispatch data and lower priority threads are suitable for AEMO MT PASA data. Initiating multiple threads with different file masks involves more setting up effort, but allows for more efficient processing. Therefore, properties configuration is the key to ensure the multiple threads of the Participant Data Replication Loader application run in a business-efficient manner. Many factors affect end-to-end performance, so tuning the various properties can deliver benefits.improvements for selected types of files over other types of files. TIP Beware of over-specifying filters, since it is possible to make a seemingly minor adjustment and possibly leave a type of file with a reduced priority or even no possible loader thread. For each process, a highly recommended practice is to always have a catch-all thread (that is, no filters) to avoid the situation of missing out on a file (even if it is rare). This also allows the loader to continue in the event of an unexpected death of a thread. 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 18

19 Filters are available for each thread. Each thread has an include filter and an exclude filter. Each filter is a comma-delimited list of strings, including no strings. The filter is the logical OR of the strings, so that any match satisfies the filter. Each string excludes leading and trailing spaces and includes asterisks (*) and question marks (?) behaving as wild-cards. Each question mark represents a single non-specific character. Each asterisk is a series of non-specific characters (that is, zero to many characters). Usually, the beginning and end of each string has an asterisk so the search mask means matching anywhere in the target Include filter If the include filter is empty or not defined, the include filter is effectively for all files. The include filter, if specified, has priority over the exclude filter. This means a file matching a specific include filter cannot be ignored by using the exclude filter. For details, see Include file mask (file_load_thread_{n}_inc) on page Exclude filter If the exclude filter is empty or not defined, the exclude filter is effectively for no files (equivalent to allowing all files through). For details, see Exclude file mask (file_load_thread_{n}_exc) on page Encrypted parameters Encryption of the password properties avoids a security risk from a casual inspection of the properties file. The encryption key is exactly 16 hexadecimal digits long and is in the pdr_key.properties file. Using encryption is optional (see Password encryption (db_password_encrypted) on page 34). The default is for encryption to be used. To use encryption, modify the encryption key in the distributed pdr_key.properties file before determining and setting any encrypted password strings. Ensure the encryption key only has hexadecimal digits (that is, 0-9 and A-F only) and must be exactly 16 digits long. To generate the encrypted key requires the environment is properly set up (see the pdrenvironment file). TIP Before starting to configure your pdrloader.properties file, ensure you update the 16 digit hexadecimal key in the pdr_key.properties file. Changing the encryption key (hash) ensures any encrypted passwords in your properties file cannot be decrypted using the default and publicly available hash. 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 19

20 In order to determine the encrypted value of a password, use the following from the command line in the \Lib folder, substituting mypassword in the command line with the password of your choice: pdrpasswordencrypt mypassword Expect to see results similar to the following: Passwd - encrypting - mypassword Returns - qrs/fu0yliphjbzogwgaaq== Decrypting mypassword The returned value is the value for the password in the pdrloader.properties file (see Target database password (db_password) on page 34). TIP If your password has any special characters with special meaning to the operating system you are using, escape those special characters appropriately. For example, if your password is!password and you are using a unix-like system, the exclamation mark needs a preceding backslash (\) to be treated as a normal single character (pdrpasswordencrypt \!Password) Automated recovery Automatic recovery involves communication back to AEMO (that is, requesting a manifest of files sent by AEMO to the participant ID). AEMO s response is a manifest report. The protocol is asynchronous and file-based, with state information recorded in the local DBMS. By default, automated recovery is disabled (for historical compatibility reasons). To enable automated recovery, set the file_load_automated_recovery property to true (see Activate automated recovery (file_load_automated_recovery) on page 44). The supplied sample scripts have automated recovery as enabled to provide the significant benefits for which the Participant Data Replication Loader is designed. NOTE No information about the state of any participant environment is stored within AEMO s databases. AEMO systems simply react to requests from participant systems to send copies of files already sent at least once. Privacy is assured in the same way as for participants sharing access to a common file server. 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 20

21 Automated recovery detail The manifest report itself is a list of all files that AEMO has a record of delivering to the participant folder forward of the request date-time. The returned manifest does contain an overlap to ensure there are no boundary conditions around the time of the request, and includes the requesting instance identifier. This data is then loaded into the PDR_MANIFEST_LOG table and set to a "RECONCILE" status. (For details about control tables and columns, refer to Database Tables on page 53). Additionally, the MANIFEST_END_DATETIME column in the PDR_REQUEST_LOG associated with the request is updated so the system has a record of the timestamp that the AEMO systems have provided data for. Note that this end time becomes the request date-time of the next manifest request. The initial RECONCILE state of data in PDR_MANIFEST_LOG indicates that when the reconciliation process runs, it must check all these rows to see if there is a matching entry in the PDR_REPORT_AUDIT table with a SUCCESS status that indicates that the file has been loaded. Files are identified by the EVENT_QUEUE_ID, FILE_ID, and CONFIDENTIALITY columns. All rows with matching records in PDR_REPORT_AUDIT are then deleted from this table and, under normal circumstances, no rows remain. If there are any rows left, these rows are set to a MISSING status, which then makes each of them a candidate for re-requesting. When run, the re-request process extracts all rows (up to a configurable limit of 100) with a MISSING status and then generates an archive request. An archive request is a list of files that the participant system is requesting the AEMO systems to resupply. All resupplied files use the trickle feed. A response to an archive request is on a per row basis (equivalent to a per file basis) and AEMO does apply a threshold to how many re-requests are accepted in any one single archive request. The archive response file is also a standard AEMO.CSV format file and updates the rows in the PDR_MANIFEST_LOG to indicate a STATUS that they have been re-requested and the archive_request_id under which this occurs. When the files eventually get loaded, the rows in the PDR_MANIFEST_LOG get deleted the next time that the reconciliation process runs, and so the gaps are gradually cleared. Independently of this process, a timeout thread is used to manage the scenario where a request has been acknowledged by the AEMO systems but no corresponding response has in fact been received. This is to ensure that in the event that the AEMO systems do not respond appropriately, that the participant system can in fact return the rows to a state where it can repeat the request to ensure that all data gaps have been cleared. 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 21

22 3.4 Minimal configuration of the.properties file The Participant Data Replication Loader has been pre-configured to suit a typical installation. In most circumstances, the default values within the configuration are suitable for normal operations. However, the Participant Data Replication Loader is a highly configurable product to ensure that it can meet a wide variety of needs. All the parameters for each instance of the Participant Data Replication Loader are in a properties file defining the parameters in a name-value pair format. The Participant Data Replication Loader reads the properties file at the start of the application and caches the values, so changes to the properties file require a restart of the running instance to take effect. TIP In Java properties files, comments begin with either a hash (#) or an exclamation mark (!). The minimal configuration steps to make to the pdrloader.properties configuration file are: 1. Enter the db_url property, which is the connection string required for your database s JDBC driver. 2. Enter the db_user property, which is the database user with which to connect to the target database. 3. Enter the db_password property, which is the password associated with the database user used to connect to the target database. TIP The db_password property is an encrypted parameter by default and the plain text password needs to be encoded according to the Encrypted Parameters section of this document. To use a plain text password, also enter the db_password_encrypted property as false. For specific detail on all configurable properties, please refer to the Configuration Properties on page 34 of this document. 3.5 Running Run the pdrloader.bat batch file or pdrloader.sh shell script Control loading of data to specific tables It is possible to turn off data being loaded to a specific table by: 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 22

23 Using Replication Manager to switch off the loading of any chosen table. For more details, refer to the Replication Manager Installation Guide. Editing the data in the IS_ACTIVE column of the relevant record of the PDR_REPORT_RECORDS table (which is what Replication Manager actually does). For details about the table, refer to PDR_REPORT_RECORDS on page 57. When a given report is set to active, the Participant Data Replication Loader populates the MMS Data Model tables by matching the data in the file with its target location in the database. When a report is set to inactive, the Participant Data Replication Loader processes the file but loads no data into the database. When a report is reset to active, any loads missed cause a data gap. By changing the local configuration (of which the AEMO systems have no knowledge), you continue to receive the data in the.csv file that is generated to your folder on AEMO s participant file server. If you keep these files, you can use these files later to fill the data gap in the RDBMS table. AEMO also keeps copies of all files sent to participants for a limited time, so providing a source of data if you lose your files (that is, primarily for participant disaster recovery scenarios). Files can be re-requested manually using Replication Manager Maintaining The following housekeeping tasks are needed to keep a Participant Data Replication Loader installation running smoothly: 1. Purge files from the pdrloader /Log folder older than a specified date. 2. Purge (and long-term archive?) files from the pdrloader archive folder older than a specified date. 3. For an Oracle environment, ensure that database statistics are kept up to date. Please refer to your Oracle DBA. 4. Optionally, you can purge rows from the PDR_REPORT_AUDIT table if the table reaches a size that causes issues, but we recommend keeping at least 6 months online to match the EMMS file archive. 5. Ensure all other components in Data Interchange are also running smoothly (see the EMMS Data Interchange Guide) Troubleshooting Data is not loading to a table If data is not loading to a table, check the following: 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 23

24 Ensure the MMS Data Model pdrloader configuration has been loaded, by checking for a row in PDR_REPORT_RECORDS associated with the DESTINATION_TABLE. If the row does not exist, check the file is part of the data intended for the MMS Data Model (such as checking in the MMS Data Model Table to File to Report Relationships workbook). o If the file is not intended for the MMS Data Model, either unsubscribe to the file or filter out the file into an alternative processing stream, so pdrloader does see the file. o If the file is intended for the MMS Data Model, there is a mismatch somewhere between the incoming data, the (cached) configuration of the pdrloader and the MMS Data Model. The mis-match is most likely to be related to the version, so ensure you have the latest files coming in, the database conforms to the latest MMS Data Model version and restart the pdrloader. Ensure the relevant report is set ACTIVE for loading, either by using Replication Manager or viewing the PDR_REPORT_TYPE_CONFIG table directly. Ensure you are subscribed to the relevant reports, by using the EMMS Data Subscription web product to view the report subscriptions for your participant ID. Ensure your batcher has connectivity to the EMMS participant file server, including checking the log file for inactivity timeout messages. Ensure your MarketNET account has not been locked out, by establishing an FTP session from a command prompt Repeatedly re-requesting a file If pdrloader fails to load a file that was subscribed to, the loader continues to re-request that file until it is successfully loaded. The recovery process is based on subscriptions as at the time the file was generated, not on current time. The way out of this is one of the following: Correct the configuration error that is causing the file not to load, since a successful file load stops this file being re-requested. Use Replication Manager to abandon the file from the request cycle. Update the database directly, by changing the STATUS in PDR_MANIFEST_LOG to ABANDONED for the file. When a file is in the ABANDONED state, Participant Data Replication Loader no longer requests it. The row can be left in the PDR_MANIFEST_LOG table to record the fact that there was an issue at some stage. You have the option to manually delete this row from the database if you do not wish to retain a permanent record. 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 24

25 Primary key violation The pdrloader does not normally experience primary key violations as it checks for the existence of data before inserting new rows. The most common reason that a primary key violation occurs is that you have subscribed to the same data twice (such as P5MIN and P5MIN_LEGACY, or BID and BID_AS files). Two separate loader threads are trying to load the same data at the same time from separate files and one fails on key violation. Under these circumstances, the pdrloader re-requests the file that had the primary key violation and (eventually) reloads it, so there is no overall failure. The resolution to this problem is to use the Data Subscription EMMS web interface to unsubscribe from the duplicate data. A further cause of primary key violations is attempting to use the standard Oracle database handler against an Oracle 11g database instance. There is a defect in Oracle 11g that requires a workaround in a targeted database handler Error message in pdrloader relating to report record properties This typically occurs if a report is being processed that does not have a matching configuration loaded in the PDR_REPORT_RECORDS table. In this case, pdrloader is unable to process the file and generates an error message. The resolution to this issue is either of the following: Load the required pdrloader Configuration into PDR_REPORT_RECORDS. Unsubscribe the file using the EMMS Data Subscription web product Data is not being populated in a column The pdrloader only loads data from the.csv file that has a matching column name to a column name in the target table. In the case where the.csv file contains additional data that does not directly map to a database column, this unmapped data is ignored. The data for the primary key columns of any target table must be included in any.csv file. The resolution for this issue is both the following steps, in sequence: 1. Ensure you are subscribed to the latest MMS Data Model files. 2. Restart the pdrloader. The pdrloader caches information about the target database structure for performance reasons, and restarting pdrloader refreshes the cache pdrloader is running but has not requested a manifest The PDRLoader only requests a manifest when there are no.csv files waiting to be loaded. If you have a significant data backlog, no manifest operations occur until all the pending data is loaded. Since the manifest is part of the automated recovery feature, also ensure the 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 25

26 file_load_automated_recovery is set to true in the pdrloader.properties file (see Database commit after table load (file_load_table_commit) on page 42) Requested manifest reports not being received If pdrloader is requesting manifest reports but they are not being received, check the following items: 1. Ensure your pdrbatcher is properly configured for the return path to the folder on the EMMS participant file server. 2. Ensure you are subscribed to the manifest file using the EMMS Data Subscription web product. 3.6 Logging configuration The logging system in Participant Data Replication Loader utilises the log4j logging framework, an industry standard logging implementation offering a rich set of functionality configurable using a properties file. The properties file defining the log4j configuration is the pdr_log4j.properties file contained in the \lib folder. The log4j configuration shipping with the Participant Data Replication Loader product uses a DailyRollingFileAppender resulting in a log file being generated for each calendar day. There are many ways to customise logging behaviour and target through the log4j framework. The details of how to configure log4j are beyond the scope of this document. However, there are a number of useful references available at the URL TIP The loader and batcher need to use separate log files, particularly when using the DailyRollingFileAppender in log4j. 3.7 Running as a Windows service There are a variety of commercial and open source solutions allowing Java applications to run as a Windows service. The distribution includes a configuration for a popular open source solution (WinRun4J) as an example of how running the Participant Data Replication Loader as a Windows service can be achieved. Participants can adapt this example to software of their own choice. For running Participant Data Replication Loader as a Windows service and if not already available, you can install JDK 7, or JRE 7 plus the jre\bin\server from JDK 7. For more details, refer to Resources on page 7. 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 26

27 The distribution includes files relevant to running as a Windows service to use an Oracle database: pdrserviceinstall.bat installs a Java application as a Windows service. pdrserviceuninstall.bat removes a Java application running as a Windows service. WinRun4J.exe is a Windows wrapper for the Java run time environment required to run the Participant Data Replication Loader under a 32-bit JRE environment. WinRun4J64.exe is a Windows wrapper for the Java run time environment required to run the Participant Data Replication Loader under a 64-bit JRE environment. WinRun4J.ini is a file containing the configuration data that defines the setup of the Windows service Installation The procedure for installing the Participant Data Replication Loader as a Windows service is as follows (assuming Java is installed appropriately refer to Resources on page 7): 1. Edit the pdrserviceinstall.bat file to update the HOME and JRE environment variables to suit your environment, ensuring the path to the JRE.DLL is valid (such as, %JRE%\bin\server\jvm.dll ). 2. Configure the HOME environment variable in the pdrserviceuninstall.bat file 3. From the command line in the /lib folder, run the pdrserviceinstall.bat file supplying the following input parameters: o Parameter 1: The name of the Windows service to identify this instance The following parameter combinations are valid: pdrserviceinstall.bat pdrloader Installation errors During installation, error messages indicate oversights, with actions as follows for the more common circumstances: MESSAGE The LoadLibrary function failed for the following reason: The specified module could not be found. Could not load the Java Virtual Machine. ACTIONS Check that the install of the JVM has put the jvm.dll as specified in the JRE environment variable in the pdrserviceinstall.bat file. Check that the install of the JVM has put the jvm.dll as specified in the JRE environment variable in the pdrserviceinstall.bat file. 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 27

28 3.7.3 Uninstalling The procedure for uninstalling the Participant Data Replication Loader as a Windows service is: From the command line in the lib folder, run the pdrserviceuninstall.bat file supplying the name of the Windows service as a parameter. The name of the Windows service is identified by the name that was supplied in the install procedure. The following parameter combinations are valid: pdrserviceuninstall.bat pdrloader 3.8 Using NT database authentication From a security perspective, it may be desirable to allow the loader application s connection to the target database to use NT authentication. The procedure for this varies across JDBC driver implementations and requires a type 2 driver. This document only covers the requirements for the Oracle database platform. The procedure for creation of an externally identified user in Oracle is beyond the scope of this document. Configuring the Participant Data Replication Loader to use NT database authentication requires the following steps: Log into the Windows account under which the Participant Data Replication Loader is to be run and validate the database connectivity by typing the following at the MS-DOS command prompt: C:\pdr\Lib>sqlplus /@your_service_name_or_sid SQL*Plus: Release Production on Fri May 20 15:02: Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved. Connected to: Oracle9i Enterprise Edition Release Production With the Partitioning, OLAP and Oracle Data Mining options JServer Release Production SQL> In the pdrloader.properties file, configure the db_url property to use the OCI style URL. An example is: db_url=jdbc:oracle:oci8:@your_service_name_or_sid You need to use the Oracle JDBC driver compatible with your Oracle client or server install. This is generally found in the $oracle_home\jdbc\lib folder, and has a name like classes12.zip or classes12.jar. Please consult your Oracle documentation for further information. 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 28

29 Remove ojdbc6.jar from the Participant Data Replication Loader s lib folder and, in the pdrenvironment.bat file, replace the entry of ojdbc6.jar in the CLASSPATH variable with the full path to your Oracle JDBC driver (with the name like classes12.zip or classes12.jar). A typical example of the altered entry in the pdrenvironment.bat file is: set CLASSPATH=AppPdrLoader.jar;C:\oracle\ora92\jdbc\lib\classes12.jar; ftp.jar;commons-codec-1.2.jar;commons-pool-1.3.jar;cryptix32.jar;log4j jar;. Remove the db_user and db_password properties from the pdrloader.properties file. Alternatively, simply comment out the properties by putting a hash (#) at the start of the line. 3.9 Loader configuration The mapping of file to database is in a suite of configuration tables in the local database. The configuration is a managed dataset supplied by AEMO as source files loadable by the Participant Data Replication Loader. It is possible to customise the mapping configuration to suit the requirements of a particular participant. However, retaining a standard configuration install ensures the data model remains up-to-date with delivery of configuration upgrades. For more details, refer to the MMS_Data_Model_pdrLoader_Configuration_Configuration_vx.x.x.zip file from the appropriate subfolder of \MMS Data Model\ for your installed version Input file anatomy File name Input files have a file name pattern similar to the following format: <VISIBILITY>_<REPORT_TYPE>_<CREATION_DATETIME><_LEGACY>.ZIP. The Participant Data Replication Loader uses the <REPORT_TYPE> parameter of the file name to determine whether this report is subject to sequential processing criteria. The Participant Data Replication Loader processes files in file name order, so the loader relies on the file names order to be the same as the file generation order. Files generated by AEMO comply with this sequencing convention Record types The input files contain three record types: C: A comment record. I: An information record defining the column names of data to follow. D: A data record containing data values matching the column names in the preceding I record. 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 29

30 The key data obtained from the first C record includes: AEMO Source Database File Creation Date File Creation Time 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 30

31 The key data obtained from the I and D records is: Report Name Report Sub Type Version Data Column Names The loader matches the Report Name, Report Sub Type and Version to records in the PDR_REPORT_RECORDS configuration table. The loader matches the Report Name, Report Sub Type, Version and Column name to records in the PDR_REPORT_RECORD_MAPPING configuration table Loader extendability The Participant Data Replication Loader provides facilities to extend the application for notification and customised transforms File event handler The file event handler is a hook enabling execution of custom code when file events occur. The process for enabling a file event handler is: 1. Create your own Java class extending the pdrfileeventint interface. 2. Compile your class and ensure placing it in a folder visible to the CLASSPATH environment variable in the pdrenvironment.bat file. 3 September 2012 Draft Version No: 3.00 Participant Data Replication Loader User Guide PAGE 31