User Manual for Reporting Officers and Reporters for Mapping Member State Standard Terminology to EFSA Standard Terminology

Size: px

Start display at page:

Download "User Manual for Reporting Officers and Reporters for Mapping Member State Standard Terminology to EFSA Standard Terminology"

Oliver Brooks
5 years ago
Views:

Supporting Publications 2012:EN-282 TECHNICAL REPORT User Manual for Reporting Officers and Reporters for Mapping Member State Standard Terminology to EFSA Standard Terminology ABSTRACT European Food

1 Supporting Publications 2012:EN-282 TECHNICAL REPORT User Manual for Reporting Officers and Reporters for Mapping Member State Standard Terminology to EFSA Standard Terminology ABSTRACT European Food Safety Authority 1 European Food Safety Authority (EFSA), Parma, Italy European Food Safety Authority (EFSA) is charged with coordinating the reporting of zoonoses, zoonotic agents, animal population, antimicrobial resistance and foodborne outbreaks in the European Union under the Directive 2003/99/EC as well as analysing and summarising the data collected. For data collection purposes EFSA has created a sample Excel based mapping tool to allow Member States to manage the mapping of their standard terminology to that published by EFSA. This technical report is a user manual document that describes how to map terms and generate XML for submission of 2011 data, via the published BIOMO data models. European Food Safety Authority, KEY WORDS Zoonoses, XML, BIOMO Data Models, SD, DCF, Excel. On request from EFSA, Question No EFSA-Q , approved on 11 May Correspondence: zoonoses@efsa.europa.eu Suggested citation: European Food Safety Authority; User Manual for Reporting Officers and Reporters for Mapping Member State Standard Terminology to EFSA Standard Terminology. Supporting Publications 2012:EN-282. [27 pp.]. Available online: European Food Safety Authority, 2012

2 SUMMARY This user manual document provides guidance on use of the BIOMO mapping tool for the submission of 2011 data by the Member States and other reporting countries. The manual provides a general introduction to the mapping tool. The report covers all aspects related to the management and use of the mapping tool. In particular detailed guidelines are provided for the mapping of standard terminology and the creating of a locally validated XML file for submission of data to EFSA via the DCF (Data Collection Framework) Instructions are also given on the updating of EFSA standard terminology dictionaries, and how to incorporate these updates in the Mapping tool. This manual is specifically aimed to guide the mapping of member specific standard terminology to those published by EFSA, thus creating a much needed documented and centralized mapping of controlled terminology. Supporting publications 2012:EN-282 2

3 TABLE OF CONTENTS Abstract... 1 Summary Introducing the Mapping Tool Getting Started by Profiling Your Data Mapping Your Profiled Terms Extending a mapping section with the copy paste buffer Coding your data The Coded_Data_Check worksheet The LookUpTerm Function The Coded_Data Worksheet Generating your XML Copy, pasting and preparing the data for XML export Activating the XML Developer tab Adding your data model XSD as an Excel XML Map Mapping the XML Map to the data in the Coded_XML worksheet Exporting our XML Map table to an XML File Supporting publications 2012:EN-282 3

1. Introducing the Mapping Tool The Mapping tools main goal is to provide a simple yet useable platform for member states to map their country specific standard terminology to those published by

4 1. Introducing the Mapping Tool The Mapping tools main goal is to provide a simple yet useable platform for member states to map their country specific standard terminology to those published by EFSA, producing an XML file for the submission of sample or aggregated based zoonoses monitoring data via the DCF. This User Manual introduces the basic functionalities available in the Mapping Tool, such as: Mapping terms explicitly Providing lookup and encoding of explicitly mapped terms Generating and validating XML via published EFSA data mode XSD This manual will include some suggestions on how to proceed when you want to profile your data, these are simply suggestions on our behalf and do not need to be followed to the letter. The mapping tool is essentially two Excel workbooks that are linked via a custom visual basic function called LookUpTerm. Each of the coded files describes one of the published data models, and consists of four worksheets, Data, Coded_Data_Check, Coded_Data, and Coded XML. The first work sheet is an exact implementation of a data model, and is the format in which Member States must organise their data for successful mapping and submission of data to EFSA. The Coded_Data_Check worksheet contains the LookUpTerm function that will query your mapping file in the PICK_LISTS_MAP work book. The function will return the mapped terms EFSA counterpart and the EFSA description, this allows for ease checking of terms, making sure that the mappings you have created are correct. Supporting publications 2012:EN-282 4

5 The Coded_Data work sheet simply references the cells in the Coded_Data_Check worksheet and via a formula removes the descriptive part of the coded term, leaving just the mapped EFSA codes. The Coded_XML work sheet is where the coded data will finally be pasted a prepared for the production of validated XML via the published BIOMO XSD schema. The PICK_LISTS_MAP workbook consists of two worksheets, Mapping and Pick Lists. The Pick Lists worksheet contains an up to data list of BIOMO standard term pick lists; these are the pick lists that are available on the online Zoonoses Web reporting tool. The Mapping worksheet is where all the mapping between Member State standard terms and those of BIOMO standard term Pick Lists is created. This worksheet will is used referenced by all the data model CodedFile_ excel Work books via the LookUpTerm function. It is essential that when working with a data model CodedFile_ excel workbook that you have both the PICK_LISTS_MAP workbook open at the same time. Supporting publications 2012:EN-282 5

1.1. Getting Started by Profiling Your Data It is very important to open both the CodedFile_ and the PICK_LISTS_MAP workbooks at the same time while working in Excel.

6 1.1. Getting Started by Profiling Your Data It is very important to open both the CodedFile_ and the PICK_LISTS_MAP workbooks at the same time while working in Excel. This is because the CodedFile_ LookUpTerm function needs to reference data in the Mapping worksheet. When you open any of the data models CodedFiles you must enable Marcros so that the custom function LookUpTerm can run. The figure below shows you the sequence of events to follow in doing this. Supporting publications 2012:EN-282 6

If you do not already have a list of all the terms that need to be mapped per data model to EFSA terms, one

clicking the select all table feature indicated in the figure below. 2.

7 If you do not already have a list of all the terms that need to be mapped per data model to EFSA terms, one simple way to profile your data is to use an Excel pivot table. To profile quickly your data follow the following steps: 1. Before inserting a Pivot table into the CodedFile Data worksheet, first select all data in the Data worksheet by clicking the select all table feature indicated in the figure below. 2. Then select the Insert Tab from the control Ribbon, and choose Pivot table. 3. The table range is already set by virtue of the select all table feature we performed in step 1, now we want to create a pivot table in a new sheet. Supporting publications 2012:EN-282 7

4. We now have a pivot table that has all the data values that are in the Data worksheet, so we can now start profiling the elements like zoonosis and matrix which need to have there terms matched

The below figure shows the pivot table that we created, where we have selected Zoonosis, as you can see, all the unique terms in the Zoonosis column are displayed.

8 4. We now have a pivot table that has all the data values that are in the Data worksheet, so we can now start profiling the elements like zoonosis and matrix which need to have there terms matched with the BIOMO standard term pick lists ZOO_CAT_PARAM_ZOO and ZOO_CAT_MATRIX. The below figure shows the pivot table that we created, where we have selected Zoonosis, as you can see, all the unique terms in the Zoonosis column are displayed. These are the terms that we need to insert into the Mapping worksheet in the PICK_LISTS_MAP workbook. The unique Zoonosis terms are copied and passed into a column below the Pivot table so that we can reference them later. 5. We repeat the above step for all data model elements that need to be mapped to the EFSA standard terms, copy and pasting the result unique terms into columns below the pivot table. 6. Eventually the list below the pivot table will have all the unique terms that need to be mapped like the figure below. Supporting publications 2012:EN-282 8

1.2. Mapping Your Profiled Terms At this point you have profiled and saved the lists of unique terms that need to be mapped from your data set to those of EFSA s.

9 1.2. Mapping Your Profiled Terms At this point you have profiled and saved the lists of unique terms that need to be mapped from your data set to those of EFSA s. Step six in profiling your data lists the profiled terms that we will use in our example mapping. The following sequence of steps will map the values we profiled for the Zoonosis element. The figure below highlights the sections of the Mapping file. A mapping section is linked to a specific standard term pick list. Area one is a hyperlinked index that will bring you to any mapping section in the worksheet. Area two is where Member State specific terms are inserted and area three where the BIOMO standard term that matches the Member State term is selected from the dropdown box highlighted in area four below. The Copy and Paste buffer section highlighted in area five is very important when you need to extend the size of the mapping area for sections two and three (see section 1.3). Supporting publications 2012:EN-282 9

10 The following steps describe the mapping process. 1. Open the Mapping worksheet in the PICK_LISTS_MAP workbook. As we know from the BIOMO data models, the Zoonosis element is linked to the ZOO_CAT_PARMA_ZOO standard term pick list. By using the hyperlinked index on the left of the page we can find the correct mapping pick list. Once we have selected the ZOO CAT PARMA ZOO standards terms pick list we paste in our profiled terms in column D,. We paste our terms into this column to check if the default mapping area is the correct size. If the number of profiled terms we need to map is larger than the default size, we need to extend the Mapping area using the Copy and Paste buffer provided. To see how to extend the mapping area, please go to section 1.3 Extending a mapping section with the copy paste buffer. When we are finished checking our term against the default size and any adjustments needed to the mapping area have been made, we can then copy out profiled terms into column E under the MAPPED COUNTRY SPECIFIC TERM/CODE column. Supporting publications 2012:EN

11 Supporting publications 2012:EN

2. Once you have pasted your profiled terms into the MAPPED COUNTRY SPECIFIC TERM/CODE column, it s time to start looking for their matching term in the Zoonoses data models standard terms pick list.

standard term that matches your Member State specific standard term. When you find a match, select it, and now your standard term is physically matched to EFSA s term.

12 2. Once you have pasted your profiled terms into the MAPPED COUNTRY SPECIFIC TERM/CODE column, it s time to start looking for their matching term in the Zoonoses data models standard terms pick list. In column F, CODE and TERM to MATCH, there is a dropdown list of all the standard terms in the EFSA pick lists for the element Zoonosis, you have to now search through this list until you find a standard term that matches your Member State specific standard term. When you find a match, select it, and now your standard term is physically matched to EFSA s term. The LookUpTerm function in the Coded_Data_Check worksheet of the CodedFile_ workbook will replace every occurrence of your standard term with the matched EFSA terms. 3. We repeat the above step until all your terms are matched for all you profiled elements Supporting publications 2012:EN

1.3. Extending a mapping section with the copy paste buffer User Manual for the 2012 Zoonoses Mapping Tool As you will have by now noticed the default size of the mapping sections is quite small and

13 1.3. Extending a mapping section with the copy paste buffer User Manual for the 2012 Zoonoses Mapping Tool As you will have by now noticed the default size of the mapping sections is quite small and for some standard term pick lists like ZOO_CAT_PARAM_ZOO, the default area will be too small very quickly. As a result each mapping area has a grey block of cells at the bottom that we call the Copy and Paste Buffer. If you click into a cell of this block under the Column F you will see that the buffer block has a reference to the correct BIOMO standard term pick list drop down for that specific mapping region. It is very important that this grey block of cells is never overwritten, if an over-write does occur you will manually need to reference again the correct BIOMO standard term pick list. 1. Always Paste profiled terms in column D. We paste our terms into this column to check if the default mapping area is the correct size. If the number of profiled terms we need to map is larger than the default size, we need to extend the Mapping area using the Copy and Paste buffer provided. In the figure below the pasted Member State terms in column D is indeed too long for the current mapping area. The Section highlighted by number four shows the only remaining space for mapping. Supporting publications 2012:EN

14 2. To use the Copy and Paste Buffer you will first have to select the grey cell block provided below the mapping area, highlighted in red below. Once you have highlighted the cells right mouse click and select copy. Supporting publications 2012:EN

3. Now that you have copied the buffer cells, you need to select the cell below the last entry in the Member State terms column E, this is indicated by number three in the figure below.

15 3. Now that you have copied the buffer cells, you need to select the cell below the last entry in the Member State terms column E, this is indicated by number three in the figure below. Once you have selected this cell, you need to right mouse click and select the Insert Copied Cells menu option, this will activate the Insert Paste Dialog window, highlighted by number five. In this dialog box you need to select the Shift Cells Down option followed by the OK button. The cell will now be pasted into the mapping area. In the figure below it is not enough to copy and paste the buffer cell just once, we will have to repeat copy paste process. Supporting publications 2012:EN

16 3->continued: As can be seen from the figure below the exact same copy and process as the figure above with the exception of number seven, whereby we select the cell below the last grey buffer cell under column E to paste our new copied Copy and Paste buffer. Supporting publications 2012:EN

4. Now that we have create the desired size for the mapping area, whereby we can fit all our terms that we pasted in column D into the Column E in the mapping area.

17 4. Now that we have create the desired size for the mapping area, whereby we can fit all our terms that we pasted in column D into the Column E in the mapping area. It s time to remove the formatting associated with the Copy and Paste Buffer. First we select all the cells in the newly created grey mapping region, highlighted in blue in the figure below. With the cells selected we navigate to the Editing/Formatting tab on the Excel Home Ribbon. We then select the clear menu option followed by Clear Formats. We can now copy and paste the data from column D to column E and begin mapping the terms to BIOMO s standards terms Pick list. Supporting publications 2012:EN

18 4-> continued: In the figure below we have the final result for the Extension of the mapping section with the copy paste buffer. 2. Coding your data After you have mapped your profiled standard terms, it is now time to check that the coding of your mapped terms is correct. The LookUpTerm function in the Coded_Data_Check worksheet of the CodedFile_ workbook is the main feature of the Mapping tool. Its job is to look in the Data worksheet, at the data that you have entered, and look for matching values in the Mapping worksheet of the PICK_LISTS_MAP workbook. When it finds a match between the term you have entered in the Data worksheet and its corresponding mapping in the Mapping worksheet, it will return the EFSA standard term code and description found in column F, CODE and TERM to MATCH. Supporting publications 2012:EN

19 2.1. The Coded_Data_Check worksheet The Coded_Data_Check worksheet is an exact copy of the Data worksheet, every cell in the Coded_Data_Check worksheet references its opposite cell in the Data worksheet. The default CodedFile_ s for all the published BIOMO data models have one hundred pre-filled rows containing all the correct formula references and LookUpTerm function calls. Only data model elements that need to be mapped to BIOMO standard term pick list need to have the LookUpTerm function reference, all the rest of the elements have a simple formula that selects its cell counter parts value in the Data worksheet and copies it, in the case where an empty string is found, the formula replaces this with a string NULL. The NULL string values will be replaced by a true null value, which we will need when we generate XML via an XSD The LookUpTerm Function The LookUpTerm Function is the glue between the CodedFile_ s data models and the Mapping worksheet. In the below example, you can see that the zoonosis element in the Coded_Data_Check worksheet has a LookUpTerm function reference, this is because all zoonosis values must come from BIOMO s standard term pick list ZOO_CAT_PARAM_ZOO_11. Because the LookUpTerm function has only a programmatic (Visual Basic) link to the Mapping worksheet in the PICK_LISTS_MAP workbook, when changes are made in a mapping section of the Mapping worksheet, there is no easy way to automatically tell the LookUpTerm function an update has occurred. This means that the user must re-run the function LookUpTerm by double click into the top cell of the element to update in the CodedFile_ s Coded_Data_Check worksheet and applying the function again to all cells in the element column. This limitation although a little more time consuming ensures that Data worksheets with allot of data are not made unbearably slow due to Excels constant auto updating when changes are made, The LookUpTerm is only run by you, and thus only when needed by you. The following sequence of figures highlights how the mapping of elements results in a correct lookup and retrieval of coded terms via the LookUpTerm function. Supporting publications 2012:EN

1. In these next figures no terms have been mapped in the Mapping worksheet for zoonosis element in the Data worksheet, and as such when the LookUpTerm function in the

standard term Salmonella - Other serovars.

20 1. In these next figures no terms have been mapped in the Mapping worksheet for zoonosis element in the Data worksheet, and as such when the LookUpTerm function in the Coded_Data_Check worksheet searches with the value Salmonella - Other serovars it will find nothing, and as a result will return Not Mapped for the member state specific standard term Salmonella - Other serovars. The above figure is an example of what the LookUpTerm function returns when it search for a member state standard term that is not mapped. The above figure shows the ZOO_CAT_PARAM_ZOO_11 mapping section in the PICK_LISTS_MAP workbook with no terms mapped for zoonosis. Supporting publications 2012:EN

2. The figures below show the result of section 1.2 mapping steps, whereby we have mapped all the profiled terms for the data model elements which require BIOMO standard terms.

21 2. The figures below show the result of section 1.2 mapping steps, whereby we have mapped all the profiled terms for the data model elements which require BIOMO standard terms. The LookUpTerm function has now returned from the Mapping worksheet, based on the mapped member state specific standard term Salmonella - Other serovars the coded value RF MCG Salmonella - Other serovars The above figure shows the ZOO_CAT_PARAM_ZOO_11 mapping section in the PICK_LISTS_MAP workbook with all terms mapped for the profiled zoonosis element in the Data worksheet. The above figure shows the LookUpTerm function returning the BIOMO standard term, based on the mapped member state specific standard term in the Mapping worksheet The Coded_Data Worksheet The Coded_Data worksheet is an exact copy of the Coded_Data_Check worksheet, every cell in the Coded_Data worksheet references its opposite cell in the Coded_Data_Check worksheet. The default CodedFile_ s for all the published BIOMO data models have one hundred pre-filled rows containing all the correct formula references. These simple formulas select its cell counter parts value in the Coded_Data_Check worksheet and copies it, in the case where it encounters a BIOMO standard term code value with description attached like RF MCG Salmonella - Other serovars, it will strip away the descriptive part leaving only the coded standard term, RF MCG Supporting publications 2012:EN

The above figure shows how the formula in the cells in Coded_Data copies and strips away the MCG Salmonella - Other serovars part of the coded term in Coded_Data_Check 3.

Under the Developer tab, an XML tool exists for the creating of what Excel calls XML Maps.

22 The above figure shows how the formula in the cells in Coded_Data copies and strips away the MCG Salmonella - Other serovars part of the coded term in Coded_Data_Check 3. Generating your XML With the release of Office Excel 2003, the importing and exporting of XML via a valid XSD has been supported. Under the Developer tab, an XML tool exists for the creating of what Excel calls XML Maps. When BIOMO publishes its data models, it also publishes an XML schema or XSD, which supports the strict structure and validation of any data model, and as such can be used to generate from our Coded_Data worksheet a valid XML file containing all our coded values The following sequence of steps will produce and exportable validated data set in XML Copy, pasting and preparing the data for XML export 1. Copy all the data from the Coded_Data worksheet. The quick way to copy all cells containing data is to use the hot key combination of Control + Shift +Space as indicated in the figure below. Supporting publications 2012:EN

In the Coded_XML worksheet we only want to paste the actual cell values and not the formulas contained in the now copied Coded_Data.

23 2. In step two we want to paste all the values we copied from the Coded_Data worksheet in step one, into the Excel worksheet Coded_XML. The worksheet Coded_Data used cell formulas to generate its cell contents, by referencing data in the Coded_Data_Check worksheet. In the Coded_XML worksheet we only want to paste the actual cell values and not the formulas contained in the now copied Coded_Data. To do this, first we select all the cells in the active Coded_XML worksheet, and then under Paste we choose Paste Values. 3. This next step will replace all occurrences of the string NULL in cells, with an actual true value null, this is to prevent errors in generating and validating XML via the XSD. Empty string values in Excel cells cause problem for the XML parser used by Excel, that is why we replace empty strings first with a string NULL and later with a real null value. To replace the string NULL, choose the Find and Select tab, then select Replace, when the Find and replace dialog appears type NULL into the Find what: search box, leave the Replace with: text box empty (indicates null to Excel). Finally select the Replace All button which will now update all instances of NULL with a true null value. Supporting publications 2012:EN

3.2. Activating the XML Developer tab The Developer tab allows you to import and create XML files, normally it is not immediately available as a default option and so needs to

24 3.2. Activating the XML Developer tab The Developer tab allows you to import and create XML files, normally it is not immediately available as a default option and so needs to be activated directly by you. The following figure indicates the sequence of selections you need to undergo to activate the Developer tab. Supporting publications 2012:EN

3.3. Adding your data model XSD as an Excel XML Map User Manual for the 2012 Zoonoses Mapping Tool To import the XSD that you want to use to validate the data model you must first go to the Developer

Once the XML Maps button is selected the XML Maps dialog box will appear, you now need to select the Add button, which will activate a File dialog.

25 3.3. Adding your data model XSD as an Excel XML Map User Manual for the 2012 Zoonoses Mapping Tool To import the XSD that you want to use to validate the data model you must first go to the Developer tab and select the Source icon. The Source icon will activate the XML Source window, at the bottom of this window there is a XML Maps button which you need to select. Once the XML Maps button is selected the XML Maps dialog box will appear, you now need to select the Add button, which will activate a File dialog. In the file dialog you now navigate to the location of the data model XSD that you wish to import as an Excel XML Map. Once you have selected the XSD, the Multiple Roots dialog will appear, please select message as indicated in the figure below and then select OK. Supporting publications 2012:EN

26 3.4. Mapping the XML Map to the data in the Coded_XML worksheet User Manual for the 2012 Zoonoses Mapping Tool 1. After we have imported our data models XSD file in as an XML Map, you can now see the structure of the XSDs elements in the XML Source window. The section of the XML Map that we are interested in is under the dataset branch. To map the XML Map to your data you will need to mouse down the left click on your mouse button on the TestResult branch, with the mouse button still held down you will need to drag the TestResult branch across to the Top Left most cell, containing the first element column name in the Coded_XML worksheet. Let go now of the mouse button when the Top Left cell becomes active. Excel will now create a Table with our XML Map as its basis. We have now successfully mapped our XML Map to the elements in our data Model. Supporting publications 2012:EN

2. Setting the Validate against Schema for Export Feature. To activate this feature we need to first select the Developer tab followed by the Map Properties button.

Now when we export the data, the XML file created will automatically be validated by the data model XSD that we imported. 3.5.

27 2. Setting the Validate against Schema for Export Feature. To activate this feature we need to first select the Developer tab followed by the Map Properties button. The XML Map Properties dialog box will now be displayed, under XML Schema validation, select the Validated data against Schema option followed by the Ok button. Now when we export the data, the XML file created will automatically be validated by the data model XSD that we imported Exporting our XML Map table to an XML File To Export our data now we first select the Developer tab followed by the Export button. The Export button will bring up the Export XML File dialog box. Here you can name and navigate to where you want to store your XML file. After naming your file simply select the export button to initiate the export procedure. Supporting publications 2012:EN

User manual for reporting officers and reporters for mapping Member State standard terminology to EFSA standard terminology

TECHNICAL REPORT APPROVED: 09 March 2015 PUBLISHED: 13 March 2015 Abstract User manual for reporting officers and reporters for mapping Member State standard terminology to EFSA standard terminology European