Directorate B: Quality, methodology and information systems

Transcription

1 EUROPEAN COMMISSION EUROSTAT Directorate B: Quality, methodology and information systems EDAMIS Dataset Naming Convention (DSNC) The naming convention for 1. Datasets, 2. Dataset occurrences This document covers the 2 possible levels of names related to datasets: 1. Dataset: mainly used in edamis components (Dataset inventory, EWP, EWA) as well as in GESMES and SDMX-ML 2. Dataset occurrence: - a dataset occurrence identifies a dataset for one country and one period (or time series or sequence), - when used by data senders to identify the files to be sent to Eurostat, it allows automatic transmission, or at least provides metadata for the construction of the envelope (same approach as when GESMES is used) Examples in this document are based on existing dataset IDs. Due to new regulations, directives, etc. these dataset identifiers may change. These changes do not affect the examples as these only intend to support the understanding of the presented definitions. Document prepared by Eurostat Unit B3

2 Version Nr. Version Date Author 4 1 February 2006 H. Cloodt V. Tronet Version Control Description Of the modification A clear split into two different documents between the dataset occurrence naming convention which should be used by data senders and the data files delivered naming convention which is used by EDAMIS at the delivery of data. This document is now in line with this current situation and is for Eurostat internal use. Modified Page(s) and/or Chapter(s) All Further improvements: - 1: cosmetic changes and new example 2-2 dataset occurrence naming convention PERIOD field, change of description for others replaced by for other than those and cosmetic changes + new examples + add optional field and format to the name + special case related to different periods for data and transmission + Special case related to the country code used when data is sent by Eurostat to a partner country. - 3 data file naming convention examples updated and action code is no more optional - 4 the naming convention at a glance, correction in the Dataset naming convention of Periodicity ID for data into Periodicity or periodicities, deleted Transmission part in the example of the dataset occurrence naming convention. Furthermore new examples and change in the structure of the dataset occurrence and data file delivered /12/2006 V. Tronet In Annex-1 (code for International Organisations) changed the yellow background from code "4A" to code "4D" because "4D" is the code to be used for Eurostat or more globally the European Commission /07/2008 H. Cloodt In 3 a remark has been introduced indicating the foreseen phase out of the action code field and explaining how to use the filename of data files delivered by EDAMIS /09/2009 H. CLoodt - definition of dataset occurrence updated to cover outgoing datasets - data file delivered definition: action code removed -naming convention at a glance updated /11/2010 D. Suranyi - Updated layout and formatting - Updated length of extension (3-4) - Support for 10-year and longer periodicity - Support for datasets with mixed periodicity - Added examples for mixed periodicity - Updated overview page /07/2011 Sogeti - Replaced edamis by EDAMIS, ewp by EWP and ewa by EWA. - Made the DSNC compliant to RFC 479: the (TO) field after the (TRANSMISSION PERIOD) field - Made the (TO) field optional Page 7 Page 4, 5, 6 All chapters Chapters 2, 3 Chapter 1 Chapter 2 Chapter 2.1, 3 Chapter 4 All chapters Chapters 2, 3, 4 Chapter 2, 4 Version 6.01, Date: 20 July 2011 Page 2 of 6

3 The naming convention described below is used by all the Data Transmission Tools and Services. 1. DATASET ID: Definition Dataset : concept of one or several statistical tables (having one single data structure), with a specific periodicity and to which is usually associated a maximum delay for transmission. Identification rule: DATASET ID = (DOMAIN ID) + _ + (DATASET STRUCTURE ID) + _ + (PERIODICITY or PERIODICITIES) Field: Length: Description/Remark: DOMAIN ID: 1..8 Identifies the statistical domain (group of datasets closely linked together) DATASET STRUCTURE ID: 1..7 Identifies the Dataset Structure (associated to one or several statistical tables). If no Dataset Structure ID is defined, the periodicity id for data will be used as default value. If 2 positions are used for the Periodicity or periodicities field, then the Dataset structure ID should be on 6 positions maximum. PERIODICITY or PERIODICITIES: 1..2 The first position, identifies the periodicity of the data to be reported: A for Annual 0 for multiannual > 10 years 1 for every 10 years 2 to 9 for every 2 to 9 years S for Semester Q for Quarterly M for Monthly W for Weekly D for Daily O for Other periodicity N for Non periodic (e.g. Sequential) defined in EDAMIS up to version 3.2, to be discontinued I for Irregular / A-periodic (e.g. Sequential) defined for future use, not yet implemented see also EDAMIS RFC504 The second position identifies the periodicity of transmission. It has to be provided only if the periodicity for transmission is different from the periodicity of the data. This case is called mixed periodicity. The second position uses the same code as the first position. Composition constraints and limitations for fields: Only the uppercase characters from A to Z and the figures from 0 to 9 are allowed For DOMAIN ID, the first character must be alphabetic The _ (underscore) character can only be used as a field separator. Based on the above the theoretical maximum length for a dataset ID is or = 18 (in compliance with the maximum length defined in GESMES). Example 1: Statistical domain: RAIL TRANSPORT, Dataset structure: Annex E (tables E1 and E2 quarterly goods) Periodicity for data and transmission: Quarterly (Q) If RAIL is chosen as identification for the statistical domain, then the dataset id to be used is: RAIL_E_Q Example 2: Statistical domain: Structural Business Statistics - Annex 4 - Construction, Dataset structure: 4L2 (Purchases of Energy Products (year 2002 onwards)) Periodicity for data: Annual (A) Periodicity for transmission: Every 2 years (2) If SBSCON is chosen as identification for the statistical domain, then the dataset id to be used is: SBSCON_4L2_A2 Version 6.01, Date: 20 July 2011 Page 3 of 6

4 2. DATASET OCCURRENCE ID: Definition Dataset occurrence : An occurrence of a dataset, for 1 sending country responsible (or 1 international organisation) and 1 primary receiving country (or 1 international organisation), for 1 period (or time series or sequence). Identification rule: DATASET OCCURRENCE ID = (DATASET ID) + _ + (FROM) + _ + (YEAR) + _ + (PERIOD) + _ + T + (TRANSMISSION PERIOD) [+ _ + (TO)] [+ _ + (optional field(s))] +. + (FORMAT) Field: Length: Description/Remark: DATASET ID: See above See above FROM: 2 The country code of the country to which the primary data providing organisation belongs. ISO2 country codes should be used except for: UK replaces GB, EL replaces GR, EU replaces 4D for the European Union, IO replaces for all other international organisations YEAR: 4 Format: YYYY, identifies the reporting year. For non-periodic datasets: 0000 or reporting year PERIOD: 4 Identifies the reporting period for periodic datasets or the sequence number for non-periodic datasets: 0000 for Annual or multiannual, 0001 to 0004 for quarterly, stating the number of the quarter, 0001 to 0012 for monthly, stating the number of the month, 0001 to 9999 for sequential, stating the number in the sequence. TRANSMISSION PERIOD 4 4 digit numeric value representing the transmission period for a datasets with mixed periodicity. It follows the same rules as the PERIOD identifier. Note: For datasets with standard periodicity this field is not used. TO: 2 The country code of the country to which the primary data receiving organisation belongs. ISO2 country codes should be used; the same exceptions as listed in the field FROM apply. Note: This field is optional. Optional field(s) Though not recommended, optional information given by the data sender (not taken into account by Eurostat data transmission tools) FORMAT 3 Examples: "XML"; GES : GESMES, CSV, FLR : Fixed length Records,. Composition constraints and limitations for fields: - YEAR and PERIOD : If several years/periods are covered by a dataset occurrence (case of time series), then only the last year/period should be specified, - Optional field(s): Only characters A to Z, the figures from 0 to 9 and _ are allowed. Version 6.01, Date: 20 July 2011 Page 4 of 6

5 2.1. EXAMPLES FOR DATASET OCCURRENCE ID: Example 1: Standard Transmission, incoming dataset For Dataset RAIL_E_Q Country: United Kingdom (UK) Data reference year: 2003 Quarter: 2 Format: CSV (Comma Separated Values) The Dataset occurrence ID to be used is: RAIL_E_Q_UK_2003_0002_EU.CSV Example 2: Transmission for Annual Dataset with optional field For Dataset ESANAAGR_0109_A Country: France (FR) Data reference year: 2001 Optional information: TEMP (Temporary) Format: Excel The Dataset occurrence ID to be used is: ESANAAGR_0109_A_FR_2001_0000_EU_TEMP.XLS Example 3: Transmission for Mixed Periodicity Multiple Transmissions Periods per Reference Period For Dataset occurrence MRTM_F7B_AS Country: Austria (AT) Data reference year: 2009 Annual data sent twice a year (e.g. 1 estimated and 1 final transmission) Format: CSV (Comma Separated Values) The Dataset Occurrence ID" to be used is: Transmission in the first semester: MRTM_F7B_AS_AT_2009_0000_T0001_EU.CSV Transmission in the second semester: MRTM_F7B_AS_AT_2009_0000_T0002_EU.CSV Period of: Reference Year Transmission Example 4: Transmission for Mixed Periodicity Multiple Reference Periods per Transmission Period For Dataset occurrence MRTM_F8A_QA Country: Sweden (SE) Data reference periods: 2009: Q1, Q2, Q3, Q4 Quarterly data sent yearly (e.g. data for all quarters 2009 to be sent in July 2010) Format: XML (SDMX-ML formatted XML file) The Dataset Occurrence ID" to be used is: Transmission for the first quarter data: MRTM_F8A_QA_SE_2009_0001_T0000_EU.XML Transmission for the second quarter data: MRTM_F8A_QA_ SE_2009_0002_T0000_EU.XML Transmission for the third quarter data: MRTM_F8A_QA_ SE_2009_0003_T0000_EU.XML Transmission for the forth quarter data: MRTM_F8A_QA_ SE_2009_0004_T0000_EU.XML Period of: Reference Year Transmission Version 6.01, Date: 20 July 2011 Page 5 of 6

6 3. THE NAMING CONVENTION AT A GLANCE: DOMAIN ID _ DATASET STRUCTURE ID _ (MIXED) PERIODICITY Dataset ID Examples: RAIL_E_Q SBSCON_4L2_A2 DATASET ID _ FROM _ YEAR _ PERIOD _ T TRANSMISSION PERIOD Dataset Occurrence ID [_ TO] [_ Option]. FORMAT Examples: RAIL_E_Q_UK_2003_0002_EU.CSV MILK_TABLEA_M_EL_2002_0012_EU.XML ESANAAGR_0109_A_FR_2001_0000_EU_TEMP.XLS MRTM_F7B_AS_AT_2009_0000_T0002_EU.CSV MRTM_F8A_QA_SE_2009_0003_T0000_EU.XML Delivered File Name DATASET OCCURRENCE ID _ V VERSION. FORMAT Examples: RAIL_E_Q_UK_2003_0002_EU_V0001.GES RAIL_E_Q_UK_2003_0002_EU_V0002.GES ESANAAGR_0109_A_FR_2001_0000_EU_V0001.XLS EGROUT_X_Q_EU_2008_0002_UK_V0001.CSV MRTM_F7B_AS_AT_2009_0000_T0002_EU_V0001.CSV Version 6.01, Date: 20 July 2011 Page 6 of 6