USER FILE DESIGN SPECIFICATION

Transcription

1 USER FILE DESIGN SPECIFICATION COUNT: Page 1 of 45

2 DOCUMENT APPROVAL ROLE NAME SIGNATURE DATE AUTHOR Ben Haworth 20/03/2014 EDITOR Nicole Williamson-Ashi 12/11/2015 REVIEWER Mark Pearce 20/11/2015 DOCUMENT HISTORY ISSUE DATE ISSUED REASON FOR ISSUE VERSION /07/2009 General update as part of overall Data Transfer Handbook review. VERSION /02/2015 Inclusion symbol as an allowable character in CHAR data items. WERSION /12/2015 Document and branding refresh COUNT: Page 2 of 45

3 CONTENTS Document Approval 2 Document History 2 Introduction 5 Scope and purpose of the document 5 References 7 Change Control 7 User file context 7 Data Transfer Catalogue Relationship 9 Design Principles 10 Overall Principles 10 Overall Structure 10 Design Statements 12 File Characteristics 12 File Header and Trailer 12 Groups 16 User File Generation Rules 17 Generic Rules 17 Encoding Rules - General 17 Encoding Rules Domain Formats 18 Encoding Rules Data Items 19 Encoding Rules Additional Characters 20 Fixed Length Variant 20 Variable Length Variant 20 Pool Transfer File Format 21 File Identifier and Optional Elements 21 Checksum requirements 21 Pool Transfer File Format Header and Footer 21 COUNT: Page 3 of 45

4 Example User File Format 24 File Structure 24 Fixed Length Variant 28 Delimited Variant 30 Understanding the examples 30 COUNT: Page 4 of 45

5 INTRODUCTION 1. This document is intended for data analysts, system designers and implementers in organisations which intend to connect to the Data Transfer Service managed by ElectraLink. 2. It describes how users should structure the files which are to be transferred between Users applications and the Data Transfer Network (DTN) Gateway. When used in conjunction with the current version of the Data Transfer Catalogue (DTC), users should be able to design and implement User Files corresponding to any data flow in the DTC. 3. User Files are those files which industry participants will send to and receive from the DTN via their Gateways. 4. The aim of this User File Design Specification is to provide an unambiguous structure for data which will be transferred between a User s applications and the User s DTN Gateway. The format of User Files is intended to be as simple as possible to implement but, at the same time, be capable of supporting the complexity of the data flows contained in the DTC. Scope and purpose of the document 5. This document describes the principles, structure and syntactic rules for constructing User Files. The document is generic, i.e. not specific to any particular data flow and is intended to apply to all users of the DTS. The document should be used in conjunction with the DTC (versions 3 and beyond), where the logical content of individual data flows is specified. 6. User Files must be in a form that is unambiguous to the user applications and the DTN Gateway. User Files are converted into Network Files by the DTN: in order for this process to be successful it is necessary that all User Files adhere strictly to the rules in this document for structuring the file and encoding of data items. 7. The User File Format defined in this document represents the default format with which DTN Gateways will be pre-configured. The default format specifies the order and content of information in the User File. 8. Two variants of the User File Format are defined: a fixed length variant and variable length delimited variant. DTN Gateways will be pre-configured to cope with both variants. Users may choose which of the two User File Format variants their particular DTN Gateway will use when delivering flows to them, based on the flow in question, and the Market Role associated with that flow. This document discusses the specification and use of File Format variants in some depth and provides details of the available options within the DTN. The technical means by which the DTN Gateway identifies which of the variants to use, is not relevant to the design of User Files and is not included. 9. In addition to the User File Format specification described in this document, files may be produced in Pool Transfer File Format. This document identifies differences between the User File Format and the Pool Transfer File Format and offers guidance about how to achieve interoperability. COUNT: Page 5 of 45

6 Term Data Item Data Flow Description Refers to atomic (i.e. non-divisible) data defined in the Data Transfer Catalogue, e.g. effective noon temperature. Refers to data flows as defined in the Data Transfer Catalogue e.g. D Daily Profile Data report. Domain Format Refers to domain formats as defined in the Data Transfer Catalogue e.g. DATE, TIME, INT(n). File Header File Trailer Group Network File Record User User File User File Format Pool Transfer File Format NULL The reference data at the start of a User File preceding the data flow information. The reference data at the end of a User File following the data flow information. A Group of data items is defined as a contiguous set of data items which are defined in the DTC to be at the same level (see DTC for definition of levels). Equivalent to a record in Pool terminology. The physical file of data exchanged between DTN Gateways. Equivalent to a Group (see above) An organisation making use of the DTN. The physical file of data exchanged between a user system or application and the DTN Gateway. The format of the User File exchanged between a user system or application and the DTN Gateway. There are two variants: fixed and variable. The format of the User File exchanged between a user system or application and the DTN Gateway. This was previously referred to as Pool File Format, however this format now incorporates a number of new fields in the header record. This new terminology will reduce the confusion between the original Pool File Format and the new Pool Transfer File Format. A status of data items in Pool Transfer File Format, equivalent to an optional item in User File Format. Table 1: Glossary of key terms COUNT: Page 6 of 45

7 References 10. References are made to the following documents: Data Transfer Catalogue DTC Electronic File Transfer Specification S03P3 Gateway Interface Specification S03P2 Change Control 11. This document is subject to formal change control procedures which are published separately. USER FILE CONTEXT 12. The User File Design Specification described in this document establishes the content and structure of the data which will be transferred between a User s DTN gateway and their local systems and applications. The transfer files are referred to as User Files in contrast to the Network files which are used to transfer data between Gateways. The relationship between these file types is illustrated in Figure 1. Sender s systems and network User Files Gateway Network Files DTN DTS Boundary Figure 1: Relationship between User files and the DTN Network Files Gateway Reciever s systems and network User Files COUNT: Page 7 of 45

8 13. This document contains all necessary specifications, structures and encoding rules to ensure that User Files generated by user applications will be unambiguous and syntactically correct when presented to the DTN Gateway. 14. In order to be accepted by the default configuration of a DTN Gateway, the physical structure of the User Files must adhere to the rules set out in this document. 15. The logical structure of a data flow within a User File is defined in the DTC; the following section defines the relationship between the DTC and the structure of User Files. 16. A User File will be generated initially by the sender of a message and passed to their local Gateway where it will be converted into a Network File by the Gateway. After transiting the Network, the Network file will be received by a recipient s Gateway and converted into a User file suitable for the recipient. 17. The DTN Gateways are being produced to give Users a choice of variants of User File Format: a) Fixed Variant; b) Variable length delimited Variant; and c) XML1 18. Variants of the User File Format will be pre-configured into the DTN Gateways for each flow prescribed in the Data Transfer Catalogue. The variant to be received by each User will be specified within the User s Gateway according to the Data Flow, Flow Version Number, Market Participant Role and the Test Data Flag. This will allow a single Gateway to support a variety of applications and systems for each User. [N.B. Users may use the Gateway to convert between Fixed and Variable Variants of a Data Flow between different Market Participant Roles within a Market Participant but traffic through the Gateway will be charged for]. 1 The XML variant is described in detail in a separate DTS User File Design Specification document COUNT: Page 8 of 45

9 DATA TRANSFER CATALOGUE RELATIONSHIP 19. This section describes how User File Formats relate to the data flows defined in the DTC and explains the flow-specific information which is contained in the DTC. Data flow contents defined in Data Transfer Catalogue Header Flow data Flow data Flow data Trailer Start End Requirements for the complete file specified In User File Design specification (this document) Figure 2: Relationship between the User File Design and the Data Transfer Catalogue 20. This document must be used in conjunction with the Data Transfer Catalogue. Although they are separate documents, the User File Design Specification and Data Transfer Catalogue will be kept aligned. 21. This document defines the generic structure of the User File, including the principles for constructing User Files, file design statements, the structure of the file header and trailer and the physical encoding rules for all the domain formats, for example DATE, TIME, INT defined in the DTC. This generic structure is common to both User File variants. 22. The DTC contains the flow specific information, incorporating the logical definitions of all the flows and their constituents, which are: a) definition and classification of Groups; b) definition and classification of data items; c) definition of domain formats; and d) definition of data flows, defining the structure of each flow, the order and the allowed repetitions of Groups, and data flow items within that flow. 23. All data items which are specific to the physical format of the User File are also included in the Data Transfer Catalogue. COUNT: Page 9 of 45

10 24. At present, there is no documented industry-wide requirement for file sequencing, and there are no facilities in the network to act on sequencing information while files are in transit. Consequently, file sequencing information is not included in the User File Format header. If a business requirement to convey sequencing information for a particular data flow or flows is defined in future, this will be subject to formal ElectraLink change control procedures. In that case it would be included as a data item in the data flow or flows where sequencing is required and will be specified for the relevant flows in the DTC. 25. Acknowledgement files will be the same as the normal User File Format structure, and will contain a single acknowledgement record. The definition of acknowledgements is contained in the document Electronic File Specification [Ref 1] which can be found elsewhere in the Data Transfer Handbook. 26. The DTC has been aligned with ELEXON s documentation with the result that non-unique Group Identifiers have been introduced; these may have an impact on implementers of the User File Format. ELEXON specifies Group Identifiers which are not unique across flows but only within flows. This affects flows defined and owned by ELEXON helpdesk which are listed in the DTC s Data Flow Index using (p) to denote if they are defined by ELEXON. Group Identifiers for non-elexon flows will be kept unique. DESIGN PRINCIPLES Overall Principles 27. There are a number of key principles which are adopted in the design of the file format: a) The structure of the User File Format is designed to be as simple as possible such that it can be produced and read by a variety of applications on different hardware/operating system platforms. At the same time it has been designed to support all data flows within the Data Transfer Catalogue; b) The file format is generic for all data flows and independent of operating system or hardware specific renditions; and c) The design of the User File Format aims to minimise as far as possible the effort and time required for Users to develop the capability to generate User Files. 28. The scope of this section covers design issues for the User File Format only, not the Pool Transfer File Format, which is discussed in Section 7. Overall Structure 29. The User File is structured to accommodate the requirement to carry more than one instance of a data flow, and to accommodate the complexity of the flows themselves. Error! Reference source not found. d epicts the structure of a User File. COUNT: Page 10 of 45

11 30. The top level consists of the following items: a) A File header; b) Data flow instances (one or more instances of the same data flow); c) A File trailer. 31. The File header contains reference information which may be required in transit, for example for onward routing, prioritisation or audit purposes. The structure of the File header is independent of data flows. For each file there is only one physical header and physical trailer as shown in Figure 3 andd and the content of the header will not be repeated. 32. Following the File header information are one or more instances of the same data flow. Wherever the Data Transfer Catalogue specifies that the first Group within a data flow has a range of one, (e.g. D0028 Standing Profile Data Report ), there will only ever be one instance of a data flow in a file. In the case of a data flow in which the range of the first Group is given as 1-* the file may contain multiple data flow instances - for example five sets of half hour meter reading information would constitute five instances of flow type D0003 Half-Hourly Advances. 33. Beneath the top level, is the Group level, which corresponds to the DTC Groups already discussed in this document. The items in the top level (File header, Data Flow, File trailer) comprise a set of Groups. In the case of the File header and File trailer, the Group level is very simple because there is only one Group. In the case of individual data flows, the complexity of the Group level depends on the flow in question. The structure of each flow in terms of its constituent Groups is defined in the DTC. 34. The lowest level is the data item level, which corresponds to the data items defined in the DTC. A group is comprised of data items; data items do not appear in a User File without being part of a Group. An instance of any data item belongs to one, and only one, Group. The set of data items belonging to a particular Group are defined in the DTC for the data flow in question. The structure of data items within the File header and File trailer Groups are defined in this document. Figure 3: Overall Structure of a User File File Header Data flow (instance 1) Data flow (instance 1) Trailer Group Group 1 Group 2 Group 3 Group 1 Only Only Data Item Data Item Data Item Data Item Data Item Data Item Data Item COUNT: Page 11 of 45

12 Design Statements 35. This section specifies: a) the design which forms the basis of the file creation procedures; and b) Rules which must be adhered to in the production of User Files. File Characteristics 36. There is no DTS restriction on the file size or the number of instances. The size of data files and the number of instances of a data flow is an end to end application issue. 37. Each User File shall contain only one type of data flow and one version of that flow. Thus, a User File may not contain half hourly advance data (flow D0003) together with the cumulative information provided in the Meter Readings flow (flow D0010). 38. Each User File may only contain data flows for one destination as determined by the market participant, market participant role, Flow Version Number and Test Data Flag. 39. All data items are defined in terms of 7-bit ISO 646 printable characters to achieve hardware and operating system independence. The set of valid characters to be used in User Files (including special characters and delimiters within data items) is defined in Appendix B. Only characters within this character set may be used. Group and data item delimiters ( <LF> and ) must not be used within data items. File Header and Trailer 40. The file header is defined to contain the necessary addressing and timestamping information to ensure the integrity and routability of the file as a whole. This information will be used as part of the acknowledgement messages. 41. The file header has the Group names ZHV for variable format files and ZHF for fixed format files (ZHD for Pool Transfer File Format files) and the following structure (where status M denotes mandatory, O denotes optional, C denotes Conditional): Data Item Status Format Comments File Identifier M CHAR(10) File identifier - unique within Market Participant Data flow and Version Number M CHAR(8) Dxxxxnnn Consists of 5 char data flow reference followed by 3 character flow version number - where n has a range COUNT: Page 12 of 45

13 Data Item Status Format Comments of 0-9 e.g. 001, From Market Participant Role Code M CHAR(1) e.g. data aggregator value A From Market Participant Id M CHAR(4) Market participant To Market Participant Role Code M CHAR(1) e.g. data aggregator value A To Market Participant Id M CHAR(4) Market participant File creation timestamp M CHAR(14) DATETIME (GMT) Sending Application Id O CHAR (5) Application identifier. For possible future use Receiving Application Id O CHAR (5) Application identifier. For possible future use Broadcast O CHAR (1) For possible future use. Test data flag O CHAR (4) Indicates whether or not this file contains test data. Table 2: ZHV/ZHF - File Header Structure 42. Notes a) File identifier is an alphanumeric string which must be unique for all files generated by a given Market Participant. The file identifier is owned by the Market Participant and its format and use is the responsibility of the Market Participant generating it. File identifiers need to be named to allow the Users to uniquely identify that file within their organisation. It is used as a token for acknowledgements, audits and queries between Users but is not used directly by the network. Users should not infer any sequencing information from values of this identifier. The combination of the From Market Participant Id and the File Identifier uniquely identify the file among all DTN users. If a file needs to be resent from within the user s network and the originating application is not involved in the resending process, the same file identifier shall be used for the file that is resent. If, however, the resending of the file involves the regeneration of data by an originating application, a new file identifier must be used. This procedure safeguards against possible changes of the data content in a file regenerated by an application. COUNT: Page 13 of 45

14 b) Data Flow and Flow version number: This is a concatenation of the Data Flow Reference and Flow Version Number which is used to identify the version of the flow contained in the User File. The Flow Version Number will be used to differentiate the data flow version used in the User File Format. c) Sending and receiving application identifiers: are optional elements in the file header, which are reserved for future use in case of requirements for routing which cannot be met by the current criteria of Receiving Market Participant Identifier, Receiving Market Participant Role, Flow Type, Flow Version Number and Test Data Flag. At the present time, this identifier is a placeholder and not intended for operational use. The default values are space characters, for fixed format and an empty delimiter for variable format. d) File creation timestamp: contains the time (GMT) at which the User File Header starts to be generated. The main purpose of this timestamp is for audit and query purposes between two Users. It will not be used by the network and is not intended to be used for sequencing purposes. If it is required to regenerate the file for any reason, the timestamp will contain the time at which the file header is regenerated. e) Broadcast: is reserved for future use. The default value is a space character for fixed format and an empty delimiter for variable format. f) Test Data Flag: is used to indicate whether this file contains test data. Up to 10 values of the Test Data Flag may be used by Users. The values and their purpose are specified below. It is assumed that maintenance of this information will be carried out by an enduring Design Authority. 'OPER' Used to route messages to live environments 'TR01' Used for routing messages to the main testing and trialling environment (i.e. End-to-End (E2E) testing) 'TR02' Reserved for routing of testing and trialling messages outside of E2E 'TR03' Reserved for routing of testing and trialling messages outside of E2E 'TR04' Reserved for routing of testing and trialling messages outside of E2E 'TR05' Reserved for routing of testing and trialling messages outside of E2E 'TR06' Reserved for routing of testing and trialling messages outside of E2E Note TR02-TR06 have not been allocated to any individual test and may be used freely by Market Participants using bilateral agreements, 'TE01' Reserved for other testing (e.g. informal inter-participant, internal integration etc.) 'TE02' Reserved for other testing (e.g. informal inter-participant, internal integration etc.) 'TE03' Reserved for other testing (e.g. informal inter-participant, internal integration etc.) COUNT: Page 14 of 45

15 43. The File trailer has the Group name ZPT and the following structure: Data Item Status Format Comments File identifier M CHAR(10) File identifier - unique within Market Participant Total Group Count M INT (10) total number of Groups in file excluding header/trailer Checksum O INT (10) Checksum Flow count C INT (8) Number of flow instances excluding file header/trailer File completion timestamp O CHAR(14) DATETIME (GMT) Table 3: ZPT File trailer structure 44. Notes a) File identifier: is repeated from the header as a file integrity check. b) Total Group count: is used to count all Groups within all flows except the file header and trailer. If this sum exceeds INT(10), the 10 least significant digits shall be encoded. c) Checksum: This is an optional item which enables application generated checksums to be transparently carried over the DTN. Note that if the DTS converts the file from one format to another during transit, the checksum may become invalid. This is outside the scope of the DTS, which uses other methods to ensure that data is valid. d) Flow count: is a count of the number of instances of a flow occurring in that file. Must be present except for Pool Transfer File Format as Flow Count does not exist for the Pool Transfer File Format. e) File completion timestamp: contains the time (GMT) at which the User File trailer is generated. The main purpose of this timestamp is for audit and query purposes between two Users. It will not be used by the network and is not intended to be used for sequencing purposes. If it is required to regenerate the file for any reason, the timestamp will contain the time at which the file trailer is regenerated. COUNT: Page 15 of 45

16 Groups 45. Flows will be structured in Groups. A Group is defined as the contiguous set of data items at the same level in the DTC. Groups can be repeated any number of times, according to the rules of the DTC. Groups are identified as aaa, a three character identifier where a is alphanumeric. The User File header and trailer are also considered as Groups, although these are independent of any particular data flow. These are designated as ZHV (variable format header), ZHF (Fixed format header) and ZPT (trailer). 46. Each occurrence of a Group within each data flow is assigned a globally unique Group identifier (defined in the Data Transfer Catalogue) for non-elexon-owned Data Flows. This means that if the same Group of data items appears in a different flow, it will be allocated a different Group identifier. The allocation and management of Group identifiers for these data flows is part of the DTC production and maintenance process which is managed by Gemserv on behalf of the MRA Service Company. Non-ELEXON owned flows encompass all flows except those indicated by BSC in the DTC Flow ownership. 47. ELEXON s specifications of Group Identifiers are not unique across flows, but only within flows. In addition, the same Group Identifiers may not contain the same data items. This affects flows defined and owned by the BSC. ELEXON-defined Group Identifiers are of the form aaa where a is alphanumeric. 48. It is necessary to preserve strict order of Groups within a flow and data items within a Group as defined in the DTC. 49. Groups can be repeated within a flow according to the rules of the DTC e.g. (004, 005, 005, 006) or (004, 005, 005, 004, 005, 005, 005, 006). 50. A delimiter is used to identify ends of occurrences of Groups. The delimiter line feed (<LF>) occurs at the end of each occurrence of a Group. Note that some text editors automatically include the carriage return (<CR>) data item at the end of each line. These must be removed before the file is sent to the DTS. The ElectraLink Web Tools includes a tool called D-FlowMaster which will do this. There are other tools available which would do the same, including a free tool called the Programmer s File Editor (PFE) 2 which can be found here: If a Group is optional and does not appear there will be no corresponding <LF>. 52. The beginning of a Group is identified by the Group identifier (e.g. 315 ). 53. Fixed format variant: If an optional item is not present within a Group, the field must be padded (i.e. filled with space characters) for its complete length. 54. Variable format variant: If an optional item is not present within a Group, this shall be indicated by the data item delimiter on its own. 2 ElectraLink cannot be held responsible for this tool or the effect it may have on your data files. COUNT: Page 16 of 45

17 USER FILE GENERATION RULES Generic Rules 55. These rules apply to all User Files, regardless of variant. Encoding Rules - General Rule 1. A separate User File must be created for each different type of data flow as defined in the DTC and for each different destination (a combination of Market Participant ID, Market Participant Role Code, Data Flow, Flow Version Number and Test Data Flag). Rule 2. The File Header is created according to the structure defined in Table 2 in the previous section. Rule 3. All mandatory elements shall be present. Rule 4. For each instance of the flow, Groups shall be constructed according to the rules for that flow in the DTC and in the order specified in the DTC. Rule 5. Data items shall be encoded according to the rules for that flow in the DTC. The physical encoding shall follow the encoding rules defined in this document. Rule 6. The order of Groups and data items specified in the DTC shall be preserved in the User File. Rule 7. Each Group shall be terminated by the delimiter <LF>. Rule 8. If a Group is optional and does not appear there will be no corresponding <LF>. Rule 9. There will be no delimiters between flow instances. Rule 10. The delimiters ( <LF> and ) must not be used within data items, irrespective of the variant used by the sender. Rule 11. The File Trailer shall be created according to the structure in Table 3in the previous section. COUNT: Page 17 of 45

18 Encoding Rules Domain Formats Rule 12. The DTC identifies the following domain formats: BOOLEAN, CHAR, DATE, DATETIME, INT, NUM, TIME, TIMESTAMP. This section defines precise rules for the encoding of each domain format type into the User File Format. Rule 13. BOOLEAN shall be encoded as either the value T or F. Lower case is not permitted. No other value is permitted, except for <SPACE> for padding purposes in the case of the fixed format variant. Rule 14. CHAR(n) is a set of characters of length n. Any characters supported by the EDIFACT Level B character set, which is a sub-set of ISO 646, may be used (the complete list in Appendix B). In addition, the underscore character _ and character (not included in the EDIFACT Level B character set) may be used. If the length of the character string to be encoded is less than n, characters shall be left justified. If the fixed format variant of the User File Format is used, the remainder of the character string shall be padded with spaces. If the variable length format variant is used, the character string may be terminated prematurely by the data item delimiter. When converting from a fixed format file to either a variable file format or Pool Transfer File format file, the DTN will ensure that trailing spaces are removed from each CHAR data item. Rule 15. DATE is a valid date encoded in a CHAR(8) field as per the domain picture for DATE in the DTC. Rule 16. DATETIME is a valid date/time encoded in a CHAR(14) field as per the domain picture for DATETIME in the DTC. The time units are specified individually for each data item, for example, in the Header and Trailer of User Files the File Creation Timestamp must be GMT. Rule 17. For INT(n), ±INT(n), NUM(n,m) and ±NUM(n,m) n and m are total logical lengths as specified in the DTC. The physical length will differ to include decimal point, sign and leading zero in the special case of NUM (n,n) (see rules 21a, 21b), where required. If the field width is smaller than the maximum length, the standard rules for variable and fixed format apply. i.e. for variable format there are no leading spaces or zeros (except in the case of 0 or 0.nnn ) and the field width is reduced; for fixed format leading spaces are used for padding. This applies to all the numeric and integer rules. All INT and NUM values shall be encoded right justified. If the field width is smaller than the maximum length, in the case of ±INT or ±NUM, the negative sign shall immediately precede the first significant digit. Rule 18. INT(n) is an integer of total length n. COUNT: Page 18 of 45

19 Rule 19. ±INT(n) Signed numbers are represented in the DTC by an initial ±. The physical length is n plus 1 because the sign is included. A negative sign must always be included in the User File. A positive sign shall not be included. For fixed format only, the positive sign will be represented by a <SPACE>. For variable variant the field width will be reduced by 1. Rule 20. NUM(n,m) is a real number of total physical length n plus 1 to include a decimal point, with m decimal places. (n,m) means that n is bigger than m. For example the logical format NUM(5,2) would physically be represented as Num(6,2), ie Rule 21. ±NUM(n,m) Signed numbers are represented by an initial ±. This is a real number of total physical length n plus 2 to include a sign and decimal point, with m decimal places. A negative sign must always be included in the User File. A positive sign shall not be included. For fixed format only, the positive sign will be represented by a <SPACE>. For variable variant the field width will be truncated. For example the logical format ±NUM(5,2) would physically be represented as Num(7,2), i.e or if positive (padded with a leading <SPACE> for fixed format and a reduced size for variable). Rule 22. A NUM(n,m) value shall always be encoded with the correct number of decimal places. Unused decimal places shall have the value 0. E.g. a value of 3.5 encoded into a Num(5,2) shall be represented as Rule 23. NUM(n,n) ie where n equals m. This is a real number of total physical length n plus 2 to include a zero and decimal point (ie 0.). For example NUM(3,3) is physically represented as Num(5,3) i.e Rule 24. ±NUM(n,n) This is a real number of total physical length n plus 3 to include a sign, a zero and decimal point (e.g. -0. ). A negative sign must always be included in the User File. A positive sign shall not be included. For fixed format only, the positive sign will be represented by a <SPACE>. For variable variant the field width will be truncated. For example ±NUM(3,3) will be physically represented as Num(6,3) i.e or Rule 25. TIME is a valid time encoded in a CHAR(6) field as per the domain picture for the TIME domain in the DTC. Rule 26. TIMESTAMP is a valid time stamp encoded in a CHAR(21) field as per the domain picture for the TIMESTAMP domain in the DTC. Encoding Rules Data Items Rule 27. Data items are always of logical type of one of domain format as defined in the DTC. The encoding rules above must be followed, together with any further rules for the item itself. For example, the use of delimiters within certain data items e.g. settlement period label, or a constraint in the numeric range or valid characters supported. COUNT: Page 19 of 45

20 Encoding Rules Additional Characters Rule 28. The representation of the data in the file will be as one data stream. <LF> may be used for Group delimiters only. Rule 29. The User File will contain no end of record markers. The split of data across records is a local matter which shall not be manifested in the physical layout of the User File. Rule 30. The file transferred between the DTN Gateway and the user system will be treated as a continuous sequence of 7 bit ISO 646 data starting with the File header and terminating with the file trailer and end of file marker. Rule 31. There may be delimiters (other than <LF> or ) or special characters within a data item if required, providing that they are within the set of allowable characters defined in Appendix B. This is the case for some identifiers. This is a matter for the Data Item Catalogue definition and is transparent to the structure of the User File. Fixed Length Variant Rule 32. Fields shall be of the correct length as defined in the domain format of that data item in the DTC and shall be padded when necessary according to the encoding rules. Rule 33. If an optional item is not present in a particular instance of a Group, the field shall be padded with n spaces, where n is the length of the field. As a consequence, a text string consisting of n spaces is not a valid value for any optional data item, since otherwise there would be confusion about the presence or absence of the data item. Rule 34. There will be no separators between data items within a Group. Variable Length Variant Rule 35. Data Items within a Group shall be delimited with the bar ( ) character at the end of each data item. Rule 36. The Group identifier ( aaa ) shall be followed by the data item delimiter ( ) then the first data item in the Group (e.g. 002 abcde ). Rule 37. An optional item that is not present in a particular instance of a Group will be denoted by its end delimiter ( e.g. abcde defgh which indicates that an optional data item between the data items abcde and defgh is not included in this instance of the Group). Rule 38. CHAR(n) character strings shall not be padded to length n. COUNT: Page 20 of 45

21 Pool Transfer File Format Rule 39. This section applies to all flows which are to be transmitted or received in Pool Transfer File Format. Rule 40. The DTN gateway will support multiple File Formats, including for DTS Users; User File Format, Variable and Fixed Variants and Pool Transfer File Format. Rule 41. The remainder of this section deals with the Pool Transfer File Format requirements. It provides guidance about the options for using and converting between Pool Transfer File Format and User File Format (variable or fixed variants). Appendix C clarifies the precise content of the User File header and trailer when translating between User File (fixed or variable) File Format and Pool Transfer File Format. File Identifier and Optional Elements Rule 42. The User File Format (Variable Variant) and Pool Transfer File Format are identical except that the Pool Transfer File Format does not have a data item delimiter ( ) between the last data item in a Group and the Group delimiter <LF>. The format is: Group identifier data item data item... data item<lf> Checksum requirements Rule 43. The various file formats allow a checksum value to be supplied within the trailer of each flow. It is likely that the algorithm used to generate and validate the checksum will rely upon the format of the User File being used. This should be considered if sending User Files that may be received in a different File Format. Pool Transfer File Format Header and Footer Rule 44. The following tables show the structure and content of the Pool Transfer File Format Header and Footer. Rule 45. The Pool Transfer File Format includes the Header and Footer in the record count i.e. it is 2 greater than the corresponding User File Format Group count. ZHD-File Header Field Field Name Type Comments 1 Record Type Text(3) = ZHD 2 File Identifier Text(10) COUNT: Page 21 of 45

22 3 File Type Text(8) 5 character type (ranges allocated for DTS, BSC or internal use) plus 3 character version 4 From Role Code Text(1) 5 From Participant Id Text(4) 6 To Role Code Text(1) 7 To Participant Id Text (4) 8 Creation Time Text (14) Time file processing was started. Specified in GMT. 9 Sending Application Id Text (5) Application Identifier. For possible future use 10 Receiving Application Id Text(5) Application Identifier. For possible future use 11 Broadcast Text (1) For possible future use 12 Test data flag Text(4) Indicates whether this file contains test data Table 4: Pool Transfer File Format Header ZPT-File Footer Field Field Name Type Comments 1 Record Type text(3) = ZPT 2 Record count integer(10) Includes header and footer 3 Checksum integer(10) Readers should apply to the ELEXON Helpdesk for further information. Table 5: Pool Transfer File Format Footer 56. The DTN no longer uses an intermediate Network File Format. All DTS flows are transmitted over the DTN in User File Format (UFF). COUNT: Page 22 of 45

23 57. When a flow has been transmitted across the network (in UFF), it is automatically converted (on route) by the DTS Hub into the appropriate User File variant (fixed, variable or Pool Transfer), depending upon the configuration data in the receiving DTN gateway for that particular flow. The flow is passed on to the receiving user system in the relevant File Format. 58. This means that a user system can generate flows to the DTN in any supported File Format variant, irrespective of how the receiving application expects the flow to be formatted when it is received. The DTS will automatically convert the flow into the correct format. 59. Details of the individual data item conversions between Pool Transfer File Format and User File Format (fixed and variable variant) are contained in Appendix C. COUNT: Page 23 of 45

24 EXAMPLE USER FILE FORMAT File Structure 60. The D0155 (Notification of New Meter Operator or Data Collector Appointment and Terms Report) flow has been chosen as an example to illustrate how the User File Format design can accommodate the different types of complexity of flows in the DTC. The complete DTC description of the flow (taken from DTC version 9.2) can be found in Appendix A. 61. The D0155 flow is illustrated schematically in figure 4, where the elements of the data flow have been grouped into sets of items at the same level in the DTC. These elements are represented in the boxes in the diagram. The DTC specifies a number of points where repetitions of groups of data items can occur, denoted by 1-*. The exact number of repetitions of each Group cannot always be defined in the data flow catalogue, since it can vary from flow to flow and therefore each repetition needs to be identified to avoid ambiguity. These are represented by blue rectangles in the diagram and have been assigned Group identifiers in the User File Format itself. 62. Note that groups 320, 316 and 317 have a condition specified within the DTC denoting whether they should be populated. 63. Table 6 gives an illustration of a DTC representation of the D0155 flow, showing the representation of Groups. The example is from DTC version Group Group Description Rang e Condition L1 L2 L3 L4 L5 L6 L7 L8 Item Name 315 MPAN Cores 1-* G 1 MPAN Core 1 Effective from Settlement Date {REGI} O Metering Point Address Line 1 O Metering Point Address COUNT: Page 24 of 45

25 Group Group Description Rang e Condition L1 L2 L3 L4 L5 L6 L7 L8 Item Name Line 2 O Metering Point Address Line 3 O Metering Point Address Line 4 O Metering Point Address Line 5 O Metering Point Address Line 6 O Metering Point Address Line 7 O Metering Point Address Line 8 O Metering Point Address Line 9 O Metering Point Postcode COUNT: Page 25 of 45

26 Group Group Description Rang e Condition L1 L2 L3 L4 L5 L6 L7 L8 Item Name 1 Contract Reference 1 Retrieval Method 1 GSP Group Id 320 Reading Cycle Details 1 If Data Collector Flow G 1 Regular Reading Cycle O Required First Scheduled Read Date 316 DC Effective Date 1 If Data Collector Flow G 1 Effective from Date {DCA} 317 MOP Effective Date 1 If Meter Operator Flow G 1 Effective from Date {MOA} 318 Agreed Service Details 1-* G 1 Service Reference 1 Service Level Reference Table 6: Layout of D0155 in the DTC COUNT: Page 26 of 45

27 Level 1 Level 2 315: MPAN Cores 1-* -MPAN Core -Effective from Settlement date -Address lines 1-9 -MPAN post code -Contract reference -Retrieval method -GSP group Id 320: Reading Cycle Details 1 -Regular reading cycle -Required first scheduled read date 316: DC Effective Date 1 -Effective from date 317: MOP Effective Date 1 -Effective from date 318: Agreed Service Details 1-* -Service reference -Service level reference Figure 4: Schematic example of D The following groups are used in this flow: ZHV/ZHF General File Header Group 315 MPAN Cores 320 Reading Cycle Details 316 DC Effective Date 317 MOP Effective Date 318 Agreed Service Details ZPT General File Trailer Group 65. The following Groups contain data items defined in the DTC version 11.5: 315 Contains the following elements: COUNT: Page 27 of 45

28 J0003 MPAN Core J0049 Effective from Settlement Date {REGI} J1036 Metering Point Address Line 1 J1037 Metering Point Address Line 2 J1038 Metering Point Address Line 3 J1039 Metering Point Address Line 4 J1040 Metering Point Address Line 5 J1041 Metering Point Address Line 6 J1042 Metering Point Address Line 7 J1043 Metering Point Address Line 8 J1044 Metering Point Address Line 9 J0263 Metering Point Postcode J0048 Contract Reference J0098 Retrieval Method J0066 GSP Group Id 320 Contains the following elements: J0277 J0696 Regular Reading Cycle Required First Scheduled Read Date 316 Contains the following elements: J0219 Effective from Date {DCA} 317 Contains the following elements: J0210 Effective from Date {MOA} 318 Contains the following elements: J0275 J0274 Service Level Reference Service Reference 66. The physical representation of the file format in the next section is shown on a Group by Group basis; for a file containing two instances of the D0155 flow. 67. Optional header fields are populated in this example. Fixed Length Variant 68. The following conventions are used to improve readability of the example. These should not be regarded as legitimate encoding practices: a) The % sign is used instead of <space> to illustrate the use of padding characters. COUNT: Page 28 of 45

29 b) For long, padded character strings, the convention... is used to indicate a large number of the same character. E.g. %%...%% to indicate a large number of padding characters. c) In the following example each Group is represented on a new line in the following example, for ease of reading, with the Group Delimiter typed as <LF>. Use of <CR> in the User File itself is not allowed. d) Annotations are in italics, preceded by ; ZHFAPP D XMIDEAFIX% BTR02<LF> ; file header Address%Line%1%%%%%%%...%%%%%%%%%%%%%Address%Line%2%%%%%% %%...%%%%%%%%%%%%Address%Line%3%%%%%%%%%...%%%%%%%%%%%Address%Line%4%%%%%%%% %%%%%...%%%%%%%Address%Line%5%%%%%%%%%%%...%%%%%%%%%Address%Line%6%%%%%%%%...% %%%%%%%%%%%Address%Line%7%%%%%%%%%%...%%%%%%%%%%Address%Line%8%%%%%%%%%...%% %%%%%%%%%Address%Line%9%%%%%%%%%...%%%%%%%%%%%MARCB%%%%%ABCDEFGHIJHID<LF> ; 1st group of 1st instance of D A <LF> <LF> <LF> <LF> Address%Line%1%%%%%%...%%%%%%%%%%%%%%Address%Line%2%%%%%%...%%%%%%%%%%%%%%Address%Line%3%%%%%%...%%%%%%%%%%%%%%Address%Line%4%%%%%%%%% %%%...%%%%%%%%Address%Line%5%%%%%%...%%%%%%%%%%%%%%Address%Line%6%%%%%%...%%%% %%%%%%%%%%Address%Line%7%%%%%%%%%...%%%%%%%%%%%Address%Line%8%%%%%%%%...%%%% %%%%%%%%Address%Line%9%%%%%%%%...%%%%%%%%%%%%MARCB%%%%%ABCDEFGHIJHID<LF> ; 1st group of 2nd instance of D A <LF> <LF> <LF> <LF> ZPTAPP %%%%%%%%10%%%%%%%%%%%%%%%%% <LF> ; file trailer COUNT: Page 29 of 45

30 Delimited Variant 69. The following conventions are used to improve readability of the example. These should not be regarded as legitimate encoding practices: a) In the following example each Group is represented on a new line in the following example, for ease of reading, with the Group Delimiter typed as <LF>. Use of <CR> in the User File itself is not allowed. b) Annotations are in italics, preceded by ; ZHV APP D X MIDE C VAR B TR02 <LF> ; file header Address Line 1 Address Line 2 Address Line 3 Address Line 4 Address Line 5 Address Line 6 Address Line 7 Address Line 8 Address Line 9 MARCB ABCDEFGHIJ H ID <LF> ; 1st group of 1st instance of D A <LF> <LF> <LF> <LF> Address Line 1 Address Line 2 Address Line 3 Address Line 4 Address Line 5 Address Line 6 Address Line 7 Address Line 8 Address Line 9 MARCB ABCDEFGHIJ H ID <LF> ; 1st group of 2nd instance of D A <LF> <LF> <LF> <LF> ZPT APP <LF> ; file trailer Understanding the examples 70. The annotated examples above illustrate the physical file format for the catalogue version 11.5 flow D The example shows one instance of a Notification of New Meter Operator or Data Collector Appointment and Terms report. The flow type is D0155, version The User File version is 4.0. The file identifier is constructed as an alphanumeric field: APP , the structure of which is the responsibility of the owner of MPID MIDE. This identifier is unique within the MPID MIDE. COUNT: Page 30 of 45

31 72. There is a repetition of the MPAN Cores id Group (315). This could be repeated many times. 73. The File Trailer (ZPT) is shown as the final Group at the end of the file. The file identifier is repeated. There are two instances of the flow with 10 Groups in that flow. COUNT: Page 31 of 45

32 APPENDIX A D0155 FLOW STRUCTURE 3 Flow reference: Flow name: D0155 Notification of New Meter Operator or Data Collector Appointment and Terms Flow description: This is a notification of a new or changed appointment to a metering system and of the contractual terms to be applied Flow Version: 001 Flow Ownership: MRA From To Version Supplier HHDC 2.0 Supplier MOP 2.0 Data Items: Supplier NHHDC 2.0 Reference J0048 J0219 J0210 J0049 Item Name Contract Reference Effective from Date {DCA} Effective from Date {MOA} Effective from Settlement Date {REGI} J1036 Metering Point Address Line 1 3 Note: This is an example flow used to illustrate the User File structure and content. This version of the flow is NOT a baselined version and must NOT be used for implementation. Implementers must refer to the appropriate version of the DTC. COUNT: Page 32 of 45

33 J1037 Metering Point Address Line 2 J1038 Metering Point Address Line 3 J1039 Metering Point Address Line 4 J1040 Metering Point Address Line 5 J1041 Metering Point Address Line 6 J1042 Metering Point Address Line 7 J1043 Metering Point Address Line 8 J1044 Metering Point Address Line 9 J0066 J0263 J0003 J0277 J0696 J0098 J0275 J0274 GSP Group Id Metering Point Postcode MPAN Core Regular Reading Cycle Required First Scheduled Read Date Retrieval Method Service Level Reference Service Reference COUNT: Page 33 of 45

34 Flow Structure: Group Group Description Range Condition L 1 L 2 L 3 L 4 L 5 L 6 L 7 L 8 Item Name 315 MPAN Cores 1-* G 1 MPAN Core 1 Effective from Settlement Date {REGI} O Metering Point Address Line 1 O Metering Point Address Line 2 O Metering Point Address Line 3 O Metering Point Address Line 4 O Metering Point Address Line 5 O Metering Point Address Line 6 O Metering Point Address Line 7 O Metering Point Address Line 8 COUNT: Page 34 of 45

35 Group Group Description Range Condition L 1 L 2 L 3 L 4 L 5 L 6 L 7 L 8 Item Name O Metering Point Address Line 9 O Metering Point Postcode 1 Contract Reference 1 Retrieval Method 1 GSP Group Id 320 Reading Cycle Details 1 If Data Collector Flow G 1 Regular Reading Cycle O Required First Scheduled Read Date 316 DC Effective Date 1 If Data Collector Flow G 1 Effective from Date {DCA} 317 MOP Effective Date 1 If Meter Operator Flow G 1 Effective from Date {MOA} 318 Agreed Service Details 1-* G 1 Service Reference 1 Service Level Reference COUNT: Page 35 of 45

36 Notes: Required First Scheduled Read Date is an optional field to be used where there is a service level agreement between a Supplier and an Agent which allows the Supplier to specify the Scheduled Read Date. This flow includes Unmetered Supply Requirements. Though all of the address data items included in this flow are defined within the structure as being optional, the address itself is mandatory and must be included in the flow. The use of any individual address item cannot be made mandatory as, in the absence of an agreed address structure for all flows, an address may be constructed from any combination of the Address Line and Postcode items. Postcode should only be omitted if the Post Office has not generated one for the premises. COUNT: Page 36 of 45