EXTERNAL SCIENTIFIC REPORT

Transcription

1 EXTERNAL SCIENTIFIC REPORT APPROVED: 16 May 2017 doi: /sp.efsa.2017.en-1241 Pilot project of the implementation of SSD2 in the frame of the electronic transmission of harmonised data collection of analytical results to EFSA: Finland Finnish Food Safety Authority (EVIRA) Abstract Pirkko Tavast Finnish Food Safety Authority Evira is participating in EFSA s pilot project of the implementation of Standard Sample Description version 2 data model, in the data collection of analytical results of residues of veterinary medicinal products (Council Directive 96/23/EC). Evira is responsible for producing and reporting data of residues of veterinary medicinal products in Finland. This D1 Final document describes the development steps to save, transform and export the data in SSD2 data model. European Food Safety Authority, 2017 Key words: Standard Sample Description, residues of veterinary medicinal products, SSD2, Finland, data collection, standardisation Question number: EFSA-Q Correspondence: data.collection@efsa.europa.eu EFSA Supporting publication 2017: EN-1241

2 Disclaimer: The present document has been produced and adopted by the bodies identified above as author(s). This task has been carried out exclusively by the author(s) in the context of a contract between the European Food Safety Authority and the author(s), awarded following a tender procedure. The present document is published complying with the transparency principle to which the Authority is subject. It may not be considered as an output adopted by the Authority. The European Food Safety Authority reserves its rights, view and position as regards the issues addressed and the conclusions reached in the present document, without prejudice to the rights of the authors. Acknowledgements: The author would like to thank the members of the project group Kaija-Leena Saraste, Leena Saari, Seija Berg, Niina Pajalin-Myllynen, Kati Manni, Sari Mäkilä, Piia Hollo and Johanna Friman in Finnish Food Safety Authority. Suggested citation: Finnish Food Safety Authority (EVIRA), Tavast P., Pilot project of the implementation of SSD2 in the frame of the electronic transmission of harmonised data collection of analytical results to EFSA: Finland. EFSA supporting publication 2017: EN pp. doi: /sp.efsa.2017.en-1241 European Food Safety Authority, 2017 Reproduction is authorised provided the source is acknowledged. 2 EFSA Supporting publication 2017: EN-1241

3 Summary EFSA launched a new SSD2 data model for reporting analysis data to EFSA. Finnish Food Safety Authority participated in SSD2 pilot project to test the collection and transmitting the data of residues of veterinary medicinal products. The standardisation process started by GAP-analysis of information management system ELMO, which is a national data source of residues of veterinary medicinal products (VMPR) in Finland. Then data mappings between EFSA codes and ELMO terms were made. Several improvements for saving new data were planned and implemented in ELMO. A data warehouse was planned for the storage, from where the data is extracted into an XML file. Due to slow development in LIMS system and changes in personnel during the pilot, the whole process was not fast enough to send data from the data warehouse. Data was reported by BusinessObjects reports, which were saved in Excel and then converted to XML by macro code. Few recommendations were suggested in the end. The recommendations concerned the timetable for changing the data model, counting the animals, and supported character sets. 3 EFSA Supporting publication 2017: EN-1241

4 Table of contents Abstract... 1 Summary... 3 Table of contents Introduction Background and Terms of Reference as provided by the requestor Objectives Software and tools Data collection in Finland Overview of national data source Software Tools Methodologies Development in LIMS system Development in Report server Data to the SSD2 packages Specific requirements to report VMPR data in SSD2 model Experience gained in testing SSD Personnel replacements Extracting the Data and packing to SSD2 format Mapping the field values Limits for the analysis Transforming data from Excel to xml Compound type Counting the sample unit IDs Evaluation, actions Data transmissions to EFSA Data sets sent Experience of error messages Syntax in Guidelines Character set problems sampinfo obligatory Conclusions Recommendations Recommendations concerning Data Model Recommendations concerning Guidelines Glossary EFSA Supporting publication 2017: EN-1241

5 1. Introduction 1.1. Background and Terms of Reference as provided by the requestor EFSA published Standard Sample Description Guidance Document and the Guidance on Data Exchange in In these guidance documents is described the data model and data transmission techniques for reporting the results of laboratory analysis on food and feed samples to EFSA. Since 2010, the use of the SSD has been fully implemented and used in the pesticide monitoring data collection. For chemical contaminants data collection the SSD has been implemented to national data management systems gradually. In Finland, the standardisation process was done to chemical contaminants data in In 2012, a working Group on SSD Extension (WG-SSD2) was established. Its objective was to extend the SSD to include zoonotic agents in food and animals, antimicrobial resistance and food additives. The group was also ordered to provide a framework for the collection of harmonised analytical data on chemical and microbiological contaminants in different matrices (e.g. food, feed, animals, water, environmental samples and food contact materials). The new version of the SSD standard, proposed by the working group WG-SSD2, is called Standard Sample Description version 2 (SSD2). This contract was awarded by EFSA to: Finnish Food Safety Authority (EVIRA) Contractor: Mrs Niina Pajalin-Myllynen Contract title: Pilot project of the implementation of SSD2 in the frame of the electronic transmission of harmonised data Contract number: OC/EFSA/DATA/2015/02 - CT Objectives The objectives of the contract are 1. to design the national data extraction process and define the automatic conversion in SSD2- XML data file by: improving the reporting guidelines documents improving the SSD2 data model for the specific domain(s) improving the SSD2 catalogues improving the specific requirements and the business rules 2. to test the data extraction process using data transmissions in 2016 (2015 data) in order to verify if the adaptation of existing National data collection requirements to the SSD2 data model is working correctly. Finnish Food Safety Authority is contracted to meet the objectives by the reporting domain of residues of veterinary medicinal products (VMPR). The objectives of the final document are to: - Describe software and tools used and/or developed - Summarise the experience gained in testing SSD2 - Provide recommendations to EFSA on the possible improvement of the SSD2 data model in the domain(s) covered in the offer(s). 5 EFSA Supporting publication 2017: EN-1241

6 2. Software and tools 2.1. Data collection in Finland Within the Control Department of Evira, the Chemical food safety has overall responsibility for planning and coordinating the National Residue Control Plan (NRCP) for live animals as well as for meat, fishery products, milk, eggs and honey. The Food Hygiene Unit also has operational responsibility for the NRCP which includes the collection and reporting data on veterinary drug residues. The Chemistry and Toxicology Research Unit and the Food and Feed Microbiology Research Unit in the Research and Laboratory Department in Evira are the producer of the analysis data Overview of national data source The national organisation that collects all the data of VMPR is Evira. Evira uses a customized, LabVantage based, LIMS-system ELMO to manage the laboratory and control data. System development began in 2009 and it was introduced in System development was started to integrate the former plant-, animal- and feed laboratory management systems. The management of control information was also integrated into ELMO system gradually. Nowadays laboratory and control functions use same registers and the research and control measures are handled seamlessly in one system. From the beginning of ELMO the food records were classified based on the EFSAs SSD model. Figure 1: Content of ELMO laboratory and control information management system 6 EFSA Supporting publication 2017: EN-1241

7 Software Tools ELMO s database is based on Oracle technology and as a BI-tool SAP BusinessObjects (BO) is used. National EFSA-reporting databases are in MS SQL Server and MySQL environment. An ETL script with Microsoft SSIS to transmit data from ELMO to reporting database. As a BI-tool SAP BusinessObjects (BO) and MS Reporting Services can be used at the moment. In the future there will be also QlikSense available. The reporting tools are used in quality control of the data. It is not easy to use reporting tool to generate xml- format because the tools are not designed for that purpose. It is not yet decided which technique Evira will use in the future to generate the xml-report. Using SQL server Transact SQL language and For XML technique is under consideration. In the pilot project, Excel VBA was used to create XML files. Figure 2: Information management systems possible to use in reporting 2.2. Methodologies Development in LIMS system At the beginning of the project, we performed GAP-analysis of ELMO and technology architecture to recognise the differences and lack of SSD2 data elements. We also designed the information management of whole the VMPR process (planning, control, laboratory services) to ELMO because we saw that it would improve the data quality and efficiency. In the beginning of project only the laboratory section was used to manage data. In the project we implemented two new parts: handling of the sampling data in control section o Information to the persons who collect the samples, where and what to collect, the reason for sampling, and the possibility to follow up the number of samples collected. a functionality to evaluate results for persons working in control section 7 EFSA Supporting publication 2017: EN-1241

8 o Persons working in control section still have no access to add or change laboratory results. They have a right to see results connected to their control events. Couple of separate column were added to make assessments. Figure 3: VMPR process in ELMO We planned that some part of data transformation to SSD2 data model is done already in ELMO because the operating staff in laboratory also has needs for the information. For example Foodex2 codes are given to samples in laboratory. The rest of the data transformation is done in EFSAreporting database. We have realized that this solution needs more detailed documentation about the maintenance of the system. We made the terminology mapping only for the mandatory elements. In the future we have plans to transmit also the optional data which is potentially collected by the sampler. Within this project we renewed our whole food register in ELMO. We saw that this was more coherent solution than patching the old food register Development in Report server In the report server, databases were updated from MS SQL Server 2008 to MS SQL Server Mapping tables were inserted into the report server database, and new fields were added to the ETL process. Development of ETL process and report database was started after changes in ELMO, and an update of the SQL Server. This caused problems in timetable. The reporting database was not ready for sending test data on summer 2016, when it was needed. Development of report database was continued for future use Data to the SSD2 packages For the year 2015 data, we created the test packages by running a BO report. In the BO report we used formulas for mapping the field values if there were only few possibilities. See in the picture grey column named 'appoint'. 8 EFSA Supporting publication 2017: EN-1241

9 Figure 4: Mapping the fields in BO report Then we saved the report as an excel file. In cases where there were more options for the column values (parameters etc.), we mapped the values in the excel sheet. We were hoping to use xml features in Excel to generate the xml-files as we did with SSD1, but after trying a while we realized that it was not possible. Therefore, we created Excel VBA script to generate xml-files from data saved in an excel sheet according to the SSD2 schema. We used schema validation tools found on the internet Specific requirements to report VMPR data in SSD2 model Evira has control and laboratory functions in the fields of plant health, animal health and food safety. The LIMS is also divided into these three functions (Eläin, Elintarvike, Kasvi Rehu ja Lannoite). Figure 5: Three sections in LIMS 9 EFSA Supporting publication 2017: EN-1241

10 In the grant project of Electronic Transmission of Chemical Occurrence Data (CFP/EFSA/DATEX/2011/01/01) in 2012 Finnish Food Safety Authority developed chemical occurrence data collection to meet the requirements of SSD. One data source in that project was ELMO and the terminology mapping to SSD was made of food safety section of ELMO. Despite the fact that the VMPR concerns food safety, the data is managed in animal health section. The reason is that there are APIs to central databases of animal keepers and holdings of animals (bovine, ovine, caprine, porcine, aquaculture, cervidae, camelidae or fur animals, poultry, pigeons, peacocks, honey bees or bumble bees). Through the APIs it is possible to have details of the animal directly from the central database. In this SSD2-pilot project we made the terminology mapping for animal section. 3. Experience gained in testing SSD Personnel replacements During the test period, we had to replace the person involved twice, which made the project very cumbersome. When personnel replacements occurred, the mail conversations were not accessible for new persons, because mails were sent between specific individuals. In the future, other kinds of conversation tools would be preferable Extracting the Data and packing to SSD2 format Because of the schedule, we had not implemented the data warehouse or the changes in ELMO when the year 2015 data was collected and saved in ELMO. Therefore we had to create excel sheets of our data, and add some fields to fulfil all requirements. While doing so, we gained valuable experience we can use when creating the Data Warehouse data model Mapping the field values One of the most important parts of project was to find mappings between the terms in our system and SSD2 data model. This work has been documented and published as Data standardisation document Limits for the analysis The laboratory department of Evira calculated the ccalpha and ccbeta limits for the combinations of animal, matrix and residue. When sending the test data sets, some of the limit values were still missing for some residues, the empty ccalpha and ccbeta values were filled with estimates and are not for real use Transforming data from Excel to xml We first tried to create the xml file using Excel xml-tools as it we did in SSD1. There were difficulties. When deadlines came closer, we wrote some script with Excel VBA to produce the xml file from data stored in an excel sheet EFSA Supporting publication 2017: EN-1241

11 Compound type XML syntax included some Compound type elements. These allow mixed content of text and nested structure. Compound type format was the reason for problems with Excel xml-tools, in chapter Counting the sample unit IDs In 1st Meeting of EFSA Network on Veterinary Medicinal Product Residues Data Collection, held on February 2016, there was discussion whether animal IDs should be included in the data or not. It is still uncertain, whether we will do this in future, and some guidelines would be helpful. In the pilotproject, we used free text info for sample in the <sampunitids><value name= batchid >. Still, the free text can have a farm name or some other information which is personal data and therefore sensitive. Anonymization or pseudonymization should probably be done Evaluation, actions In 2015 we did not have possibility to insert evaluations of results nor needed actions in our ELMO system. When the test packages were sent, we did not have a possibility to add evaluation of the result or actions in our system. That s why this part is not tested, all the rows had evalcode = J037A, no evaluation and acttakencode= N, no actions Data transmissions to EFSA Data sets sent We started the test transmissions with samples of honey, because the amount of samples is relatively low and the data is quite homogenous for example only one matrix. It took one month to deliver all six packages of data. Matrix #Samples #Rows Honey Milk Egg Fish Animals Meat Total Table 1 Test data sent in the project Experience of error messages When data was sent, the first error messages were received fast. This was when there was an error in the structure of the xml file. If the error was in Business rules, it took relatively long time to get the 11 EFSA Supporting publication 2017: EN-1241

12 error message, minutes. It would help the process considerably if it would be possible to get the error messages immediately Syntax in Guidelines In Guidelines for reporting data on residues of veterinary medicinal products ( PUBLISHED: 31 March 2015), XML syntax for evalinfo is wrong in text. In the example there is a syntax which works. Caption 3.2: The sections of the data model / N Evaluation section / N.06 Additional information on the evaluation (evalinfo): Alternative syntax XML example: <evalinfo> <value name= samptkasses >J037A</value> <value name= sampeventasses >J038A</value> <value name= conclusion >C04A</value> </evalinfo> Appendix D Example of one result in XML: <evalinfo>samptkasses=j037a</evalinfo> Character set problems We used xml encoding="utf-8". In Finnish language there are letters like 'å' 'ä' 'ö' in sample info texts, which should work with this character set. The xml-file was validated against the schema and there were no errors, despite that we found: DISCARDED Invalid byte 2 of 3-byte UTF-8 sequence.06_9906insert5854ssd2pilot_data201502_vmprssd2_pilot_v2fi_evira_ T07:14:22.403KOREJECTED Also in the names of parameters, <paramtext> in our system we have α, β, which caused errors. We had to do replacements: α alpha and β beta sampinfo obligatory It seems that we have to have an empty row for <sampinfo>. It would be nice that the element would not be obligatory EFSA Supporting publication 2017: EN-1241

13 4. Conclusions In the pilot project, some more data was saved to LIMS system, data sets in SSD2 format were built and data was sent to EFSA. To design and implement changes in LIMS system takes quite a long time. The new functionality must also be tested. The changes in ETL procedures and report database should be performed simultaneously. Compound type in XML structure was not possible to create with Excel tools. When personnel changes occurred in our organisation during piloting, the mail conversations with EFSA were not accessible for new persons. In the future other kinds of conversation tools should be considered. Open discussion /Q&A would have helped, and could also aid people starting to build their SSD2 data sets EFSA Supporting publication 2017: EN-1241

14 5. Recommendations We think more time for development is needed, if new kind of data is needed. The first step is to add to the LIMS system new fields with access rights functionality and registers. The second step is to collect data. Thereafter new kind of data can be reported. We have couple of comments on Data Model and Guidelines. Recommendations on catalogues are documented earlier in the Data standardisation document Recommendations concerning Data Model SampleUnitIds SampleUnitIds field was planned to show how many different animals or farms were controlled. Because the animal ID and farm ID are sensitive data, some other way should be considered. One possibility is that we have some ID-number for just EFSA-transfer. In this case the farms and slaughterhouses would be counted by separate ID-numbers, although this might be challenging to the LIMS systems of members Recommendations concerning Guidelines The pilot project was performed using the Guidelines for reporting data on residues of veterinary medicinal products. EvalInfo In the text and in the example the syntax is different for evalinfo. It should be the same. Character set It seems UTF-8 character set in SSD2 does not allow special characters, as α, β, ä, ö,, which can be needed in text columns as sampunitids or parameter names. Limitations of special characters should be documented or special characters should be supported EFSA Supporting publication 2017: EN-1241

15 Glossary API BI BO DCF EFSA ELMO EU Evira GAP-analysis LIMS LOD LOQ NRCP SQL SSD Application Programming Interface Business Intelligence Business Objects -reporting tool Data Collection Framework European Food Safety Authority Evira's control and laboratory information management system European Union Finnish Food Safety Authority A technique that is use to determine what steps need to be taken in order to move from its current state to future state. Laboratory Information Management System Limit of Detection Limit of quantification National Residue Control Plan Structured Query Language Standard Sample Description SSD2 Standard Sample Description version 2.0 TWG-DC VMPR WG-SSD2 XML Technical Working Group on Data Collection Residues of Veterinary Medicinal Products Working Group on SSD Extension Extensible Markup Language 15 EFSA Supporting publication 2017: EN-1241