Excel to SDMX Templates for Fisheries Statistics

Transcription

1 EUROPEAN COMMISSION EUROSTAT Directorate E: Sectoral and regional statistics Unit E-1: Agriculture and fisheries Excel to SDMX Templates for Fisheries Statistics User Manual Version 2.1 Friday, 17 May 2013 Unit E1 IT section Excel to SDMX Templates Page 1 of 13

2 Table of contents What are the templates for?... 3 What do I need to use them?... 3 How do I create SDMX-ML data and load it to edamis?... 4 How can I enter data into the template?... 5 Are the fields validated?... 8 What are the different worksheets in the templates for? Will templates be updated? Are there restrictions? What is the XML source for the templates? Excel to SDMX Templates Page 2 of 13

3 What are the templates for? Regulatory data which has to be submitted to EUROSTAT through the edamis system can either be entered directly into WebForms or uploaded through the "Send Datafile" form. Due to size limitations it has not been possible to create WebForms for fisheries. Therefore, SDMX has been adopted as the data collection format, in-line with Eurostat policy and in recognition of its position as an international standard for data exchange. To help produce these SDMX data Eurostat has built some Excel templates. In 2010, a first template with VBA macros was provided for aquaculture. It was well accepted by many; however there was a problem for some Member States who are not allowed to download Excel files with macros. This is a protection against possible viruses. Since that time, the IT section of Unit E2 has developed new templates and a way to make them work without macros has been find out. Currently 7 templates are customized for aquaculture, catch and landings datasets. They are safe and don t contain any macros. What do I need to use them? The templates provided make use of Excel s built in XML functionality, but in order to use this, you must have the Professional Edition of Microsoft Excel 2003 or higher. The Professional Edition of Excel 2003 includes an XML menu item on the Data menu which is required. Excel to SDMX Templates Page 3 of 13

4 The Professional Edition of Excel 2007 and above has a Developer ribbon which contains the XML menu. How do I create SDMX-ML data and load it to edamis? To use the templates, open the DATA ENTRY worksheet and enter the data into the Data List (cells marked with a blue border). For almost all input cells there are drop-down lists to help you with data entry. When your data is complete, select the Excel menu "Data-->XML-->Export" to create an XML file of your data. If you are using Excel 2007 or later then this menu is available from the Developer ribbon rather than the Data ribbon. Once you have created your XML file, login to edamis and upload the file. Excel to SDMX Templates Page 4 of 13

5 How can I enter data into the template? You can key the data in directly or select the appropriate codes from the drop down lists. After having completed the first row, you can fill in data automatically in rows This is a great way to reduce the amount of data to input. In all worksheets, the first 3 columns are always the same: "Frequency", "Year" and "Country" and are repeating the same value over and over. You can quickly fill these cells by using either the fill command or the fill handle (it's the small black square in the bottom right of the active cell). The "Fill" command Start by selecting the cells to copy, then drag the mouse down (or hold down the Shift key and click in the last cell you want to fill), and then choose Edit > Fill > Down. This can also be achieved by pressing Ctrl-D. Excel to SDMX Templates Page 5 of 13

6 The other alternative: the "Fill handle" Select the cells you want to copy, right-click on the fill handle and while holding the mouse button, drag down, release the button and select "Copy Cells". Note: If you left-click on the fill handle then the year column will fill in series (2010, 2011, 2012, ) which is not what we want. Excel to SDMX Templates Page 6 of 13

7 You can paste directly from the keyboard You can also copy and paste data from elsewhere, or use software such as VBA to populate the data from elsewhere. Note, however, that the format of the fields in the template MUST NOT be changed, otherwise the data might not export to XML correctly. For this reason we d recommend using the paste special values option in Excel rather than simply pasting. The following window appears if the data is coming from outside the current Excel application. Excel to SDMX Templates Page 7 of 13

8 Are the fields validated? There is validation on each cell to ensure that only valid data can be entered. For instance, the frequency can only be A (Annual). Any other data will result in an error message appearing. Two remarks: 1. Data validation is designed only to catch input that a user types directly into a cell. Pasted data and data that is dropped in by a VBA procedure will always bypass the validation setup. If you use for instance Paste Special > Values, the data validation rules are preserved, although they don t execute. It's only if you subsequently edit the cells that the validation applies. 2. Excel provides three levels of warning: Stop forces the user to retry until valid data is entered; Warning allows the user to enter invalid data if they insist; Information readily accepts invalid data. In the provided templates the Stop level was chosen for almost every cell with the exception of the species cells, which have the Warning level. There is a final validation done when you export your data to XML. If the resultant XML is not schema valid you get a warning message. That means it does not conform to the correct XML format. Excel to SDMX Templates Page 8 of 13

9 Check your data for wrong codes At any time you can force a validation check on all cells as follows: 1. Show the Formula Auditing Toolbar The quickest way to display it is by selecting (View > Toolbars). Alternatively you can select (Tools > Formula Auditing > Show Formula Auditing Toolbar). 2. Click on the Circle Invalid Data image button 3. Excel draws red circles around all the cells with invalid values If you correct an invalid entry, the circle disappears automatically. Excel to SDMX Templates Page 9 of 13

10 It is strongly recommended that you correct all red circle validations prior to exporting your data. If you are using Excel 2007 or later, Data validation options are located in the Data Tools group. What are the different worksheets in the templates for? The template workbooks contain several worksheets. The DATA ENTRY worksheet is used for the purposes of data input and is the only one that can be directly modified by the user. The RULES worksheet describes the validation rules applied to the input cells in the DATA ENTRY worksheet and shows also the content of the SDMX Header. This header is filled automatically by formulas. All the other worksheets contain the codifications tables called "Codelists" in the SDMX world. Excel to SDMX Templates Page 10 of 13

11 Will templates be updated? If the dataset structures change in the future (for instance if there are new fields), then we will publish new versions of the templates. If you have been using a modified version of the template, then you would need to either update your own version with the equivalent changes or redo your modifications to the new templates. Are there restrictions? The templates contain the following limitations: 1. SDMX-ML files must be generated from the templates. For example, do not move the DATA ENTRY sheet to another Excel spreadsheet. The templates use the XML mapping feature in Excel and can result in undesirable effects if the sheets are moved. 2. Templates are customized to allow direct exportation of data in SDMX format (see explanations below). You can use them only to prepare your data for ONE SPECIFIC YEAR. What is the XML source for the templates? The templates use an XML map as their "XML source". This map is a schema that has been customized for Excel based on the Compact XSD Schemas (XSD) that can be directly downloaded from the SDMX Registry. Excel to SDMX Templates Page 11 of 13

12 List of lists problem In Excel, the XML mapping cannot be exported if the schema contains a List of lists. A list of lists is a primary list of items which contains other lists of items. It is a common way to represent complex data. Unfortunately all the schemas published in the registry are constructed that way: one DataSet can contain several Series (maxoccurs=unbounded) and each Series can contain several observations Obs (maxoccurs=unbounded). To handle that problem in Excel, the number of occurences in one of the two lists (Series, Observations) must be limited to 1. By chance, in the case of fisheries statistics, the data is collected annualy for a specific reference year. We have therefore changed the list which contains the time period (= the list for observations): maxoccurs is not unbounded anymore but equal to 1 (one reference year). For the user of the templates it means that he is only allowed to provide data for only one refrence year. The following text shows the "revised" schema used by the Excel template for catch statistics. Excel to SDMX Templates Page 12 of 13

13 Excel to SDMX Templates Page 13 of 13