Conformance Testing. Service Offering Description

Transcription

1 EUROPEAN COMMISSION DIGIT Connecting Europe Facility Conformance Testing Service Offering Description [Subject] Version [3.0] Status [Final] Date: 12/06/2017

2 Document Approver(s): Approver Name CORNEAU Caroline Role CEF einvoicing Document Reviewers: Reviewer Name CORNEAU Caroline DE BOER Christel Role CEF einvoicing CEF einvoicing Summary of Changes: Version Date Created by Short Description of Changes /06/2016 CEF Support Creation of the document /06/2016 CEF Support Review of the document /06/2016 Thomas SMALL Review of the document /06/2016 CEF Support Update the document /06/2016 CEF Support Review of the document /08/2016 Patia Template review TAIMURAZOVA /08/2016 CEF Support Apply comments and remarks /10/2016 Christel DE BOER Added information to section /10/2016 Christel DE BOER Comments by Costas Simatos & Nicolas LOOZEN /10/2016 Christel DE BOER updated and changed drawings /11/2016 Christel DE BOER Final after review period /02/2016 CEF Support Update Template /06/2017 ACHHAB Oussama Review 2nd syntax added to testing Conformance Testing Page 2 / 23

3 Table of Contents 1. INTRODUCTION Purpose of the service GITB test bed software Description Functions Future Users Primary Users Other Potential users Scope Benefits ROLES AND RESPONSIBILITIES Software/Service Providers CEF Support Team HOW TO USE THE SERVICE STEP BY STEP Processes Overview Step 1: Registration Step 2: Preparation Specific on technology chosen: Specific on technology chosen: web form upload Specific on technology chosen: web service Specific on technology chosen: offline tool Specific on technology chosen: edelivery edelivery AS edelivery AS Step 3: Execution & report feedback Specific on technology chosen: Specific on technology chosen: web form upload Specific on technology chosen: web service Specific on technology chosen: offline tool Specific on technology chosen: edelivery AS Specific on technology chosen: edelivery AS Step 4: Closing TERMS AND CONDITIONS GLOSSARY ANNEXE 1 DOCUMENT PARTS CONTACT INFORMATION Conformance Testing Page 3 / 23

4 1. INTRODUCTION This document describes the Conformance Testing service provided by the CEF einvoicing Test Beds. It introduces the purpose of the service, its users, its scope, its benefits, the related roles and responsibilities and the overall process Purpose of the service CEF offers a test infrastructure that will allow Solution Providers and Public Administrations to check their existing einvoicing solutions compliance against a given standard, such as PEPPOL BIS or any other syntax supporting the future einvoicing European standard (after Q2 2017). The goal of the CEF einvoicing Conformance Testing service is to verify that an implementation based on the einvoicing specification is indeed conformant to the said specifications. CEF einvoicing provides ready to use test cases, a testing platform, and supports the users of the CEF einvoicing Conformance Testing service during the entire testing process. The overall approach to Conformance Testing is presented in the figure below: The technical specifications are translated in test assertions (a measurable statement for evaluating the conformance of the einvoices). These test assertions are the main unit for testing and reporting the conformance of the einvoices. An einvoice can only pass the Conformance Testing process in case all mandatory test assertions have successfully been completed. Conformance Testing Page 4 / 23

5 Based on the test assertions a set of test cases are derived which are implemented on the test platform in the form of executable test scripts. The outcome of the test execution is documented in the test report. In case the test report shows that all mandatory test assertions have been passed, the einvoice becomes conformant. The CEF einvoicing Conformance Testing service provides ready to use test assertions, test cases and a test platform. The Conformance Testing service also provides support during the set-up of the connection with the test platform and the actual test execution. A list of key terminology related to Conformance Testing is defined in the table below. Terminology Reference implementation System under test Definition Application component that serves the purpose of realising a service that is made available to systems under test (SUT) to perform interoperability tests against. It is termed a reference implementation because it typically provides an implementation of a specification that is assumed to be correct so that other implementations of the same specification (or just other instances of the reference software) can be tested against it. Note that in case no standard or specification is involved, the reference implementation is effectively a simulation of the service that is exposed for testing. Reference implementations can range from being fully independent IT systems to plug-in components. System that is being tested for correct operation. The owner is the system under test want to perform interoperability testing on it by connecting and running test scenarios against available services. Test assertion Artefact that defines a statement to capture the goals of a testing activity and its prescription level, making the link between the normative source of the assertion (e.g. a legislation article) and the test details elaborated in test cases. An example of a test assertion would be The system needs to respond with a receipt to a valid request within one minute. Test cases Test suite Artefact that defines in detail the message exchange, processing and validation steps to be performed in realising a test assertion. A test case realises an assertion possibly as a part of a set of related test cases and can potentially be reusable between multiple assertions. Collection of test assertions forming a cohesive set. In the same way test assertions group a number of related test cases, test suites group assertions into a cohesive set. As an example, consider a test suite that groups all the assertions needed to test a system s conformance against a specific standard (e.g. AS4). Conformance Testing Page 5 / 23

6 Test Platform A tool that facilitates the creation, maintenance and execution of tests. Its main features/components are: Test repository: typically holds the test assertions and test cases. Both can be edited by the test managers and viewed by the users of the platform. Wrapping support: connecting the system under test to the test bed platform either directly or by means of an application programming interface (API). Core testing: execution of the test cases, either manually by the testers and/or users or automatically through test scripts. All traceable and observable behaviour from the implementation under test is captured in the test log during the test process for further analysis. Content validation: validation of the outputs (e.g. documents) produced by the implementation under test against the requirements specified by the specifications. Test report generation: documenting the results of verifying the behaviour and outputs of the implementation under test into a structured test report. Conformance Testing Page 6 / 23

7 1.2. GITB test bed software Description Software and hardware resources comprising a complete platform (Figure 3) Purpose is to make testing easier (for systems and testers) Conformance testing (messages conform to specifications) Testing can use simulators or test-focused instances (e.g. reference implementations of standards) GITB is a CEN standardization initiative to produce test bed specifications and implementing software Functions Contents validation for CEF einvoicing user community Validation via web service call and through use of a standalone offline validation tool and edelivery Validation of PEPPOL BIS invoices (XML based) over various communication channels o Through web form upload o As attachment Extension of supported communication means to include CEF einvoicing o Send invoice to access point using AS2 o Send invoice to access point using AS Future 1 Extension of supported communication means to include CEF einvoicing o Send invoice to access point using AS2 o Send invoice to access point using AS4 1 Details on these services are listed on the wiki page: +CEF+E-Invoice+-+Validation Conformance Testing Page 7 / 23

8 Extensions linked to compliance to European standard on electronic invoicing for CEN WS8 syntaxes provided by the CEN standardisation organisation 1.3. Users Primary Users "Testing for electronic invoice exchange using the European Commission s CEF einvoicing DSI 2 is offered to its Software / Service Providers across Member States to address invoice content validation. Invoices can currently be provided for automated validation by means of web form upload, and (GITB-compliant) web service call, invoice delivery as AS2 3 and AS4 4 messages using the CEF edelivery DSI." The term "User" will be used in the remainder of this document to refer to any of these types of users Other Potential users "Initiatives (public administrations, funded projects etc.) with relevance to (cross-border) interoperability can apply to have their reference implementations and test suites installed in the test bed run by ISA², for their respective communities to use. Software owners, including privately held companies who build solutions as part of a larger programme for which they would need or want to undertake interoperability testing, can then use the test bed to test their products connectivity against supported reference implementations, or the conformance of their messages against supported specifications. The testing of software providers solutions through the test bed results in benefiting the businesses and citizens that will eventually use the envisioned services, in that the interoperability of the building blocks used to realise them is rigorously tested. 2 CEF building blocks on Joinup: 3 AS2: 4 AS4 profile of ebms 3.0: os.pdf Conformance Testing Page 8 / 23

9 1.4. Scope The scope of the CEF einvoicing Conformance Testing is: Explaining conformance testing and the software used to perform it. Showing the different scenarios and examples on a provided platform Benefits The CEF einvoicing Conformance Testing service has been designed to generate a list of benefits to the users of the service: Testing supported by professional staff of the European Commission. Quick testing cycle with reduced cost and time. Testing anywhere at any time. Tests are fully domain-neutral. Ready to use test cases and test platform with a common and standardised plugin. Conformance Testing Page 9 / 23

10 2. ROLES AND RESPONSIBILITIES This section describes the main roles in the CEF einvoicing Conformance Testing service and their responsibilities. The following table summarises the split of roles and responsibilities between the different actors in the CEF einvoicing Conformance Testing process in the form of a RACI 5 matrix where: Responsible (R): indicates the entities that perform the process-step. Every process-step has at least one responsible entity. Responsibilities can also be shared. Accountable (A): indicates the entity that is ultimately accountable for the process-step. Every process-step has only one accountable entity. Consulted (C): indicates the entities that give feedback or are consulted during the processstep. This is a two-way process. Not every process-step has an entity that is being consulted. Informed (I): indicates the entities that needs to be informed on the results of the processstep. This is a one-way process. Not every process-step has an entity that is being informed. The process is described in detail in Section 3. Process/Service Software/Service Provider Responsibility Entity CEF Support team Step 1: Registration R A/R Step 2: Preparation A/R I Step 3: Execution A/R C CEF Stakeholder Management Office (SMO) Step 4: Closing I A/R R 2.1. Software/Service Providers Role: User Responsibilities: Detect and Initiate an Incident, a Request or a Request for Change. Test and validate the closure of tickets CEF Support Team Role: CEF Support Team (2 nd level support) Responsibilities: Register, classify, investigate, escalate if needed, resolve and close tickets. Act as the technical single point of contact to the software or service provider and provide support during the connection to the test platform and test execution. Conduct a final test run and generate the test report. 5 RACI: R = Responsible; A = Accountable; C = Consulted, I = Informed. Conformance Testing Page 10 / 23

11 3. HOW TO USE THE SERVICE STEP BY STEP This section describes the way the user can apply for the service and the processes that are part of the CEF einvoicing Conformance Testing service Processes Overview The figure below presents the main steps of the CEF einvoicing Conformance Testing service process. Each process steps is described in more detail in the next sections. The figure below presents the multiple technical option the user can choose from to apply for conformance testing, they all start with a file of the invoice in xml format (hereafter called invoice.xml). Depending on the choice he or she makes the user will need to perform some steps to execute the conformance test, these are describes in the following sections.. Conformance Testing Page 11 / 23

12 3.2. Step 1: Registration Purpose: register the service request and exchange the necessary information and documentation to set up an account on the test platform. This registration is only required when applying for the conformance testing service via AS2/AS4 protocols for example edelivery. Actors: Process: Software / Service Provider CEF Support Team 1. The Software / Service Provider sends an to the CEF Support Team to request the CEF einvoicing Conformance Testing service. The request should contain the following information: einvoice to be tested Planned start date of the tests Single Point of Contact at the Software / Service Provider side 2. The CEF Support Team registers the service request. 3. The CEF Support Team provides information and documentation to the Software / Software Provider. This information includes: Information on the specifications Test assertion document Information on the test platform 4. The CEF Support Testing Team assigns a Single Point of Contact for the request. 5. The CEF Support Testing Team creates an account on the testing platform and sends the login details to the Software / Service Provider. The overview of the registration process is shown in the diagram below. Conformance Testing Page 12 / 23

13 3.3. Step 2: Preparation Purpose: The user prepares the invoice data file that he or she will use to execute the conformance testing and depending on the technical test option chosen, downloads the needed certificates or files. Actors: Process: Software / Service Provider 1. The Software / Service Provider uses one of the supported communication means to prepare the testing platform with the einvoice and performs conformance tests. The overview of the preparation process is shown in the diagram below. Remark: The CEF Support Team is available for technical questions Specific on technology chosen: When chosen to apply for the conformance testing service via the user can request for validation of invoices by ing one or more invoices.xml file(s) as attachment(s) to an address (the address to forward the information to can be requested via the service desk) Specific on technology chosen: web form upload When chosen to apply for the conformance testing service via web form upload the user can request validation via uploading the invoice.xml file(s) through This service is provided to all users (i.e. via anonymous access) Once the electronic invoice is uploaded, it is validated and a detailed report is presented to the user that can be downloaded (in GITB TRL format) and cross-checked against the submitted content Specific on technology chosen: web service When chosen to apply for the conformance testing service via web service an invoice can be validated via web service call to a GITB-compliant content validation service. The web service (WSDL) is available at: Conformance Testing Page 13 / 23

14 Specific on technology chosen: offline tool This option can be advised for local validation of sensitive files or use for continuous integration and unit testing. When chosen to apply for the conformance testing service via the use of an offline validation tool, one or more invoices can be validated after download the all-in-one jar file from: Specific on technology chosen: edelivery The edelivery option can be AS2 or AS4 and implies that at least one user is registered in the test bed. Self-registration is currently not possible, a user can request registration by contacting the help desk (see section 3.2 Step 1: Registration) edelivery AS2 When chosen to apply for the conformance testing service via the use of an AS2 message over e- Delivery. An invoice may be submitted for validation using the GITB test bed instance that is available at in which the following test scenarios are possible: Submission using the full e-delivery message exchange, i.e. involving an SML lookup, an SMP query and finally submission of the message to the AS2-capable Access Point. Submission using direct querying of the SMP. In this case the SML lookup is skipped, assuming that the sender can be configured with a direct SMP address for test purposes. Submission using direct messaging to an AS2-capable Access Point. This assumes that the sender can be configured with a specific Access Point address for test purposes. A partner certificate for the receiver that is needed to support encryption and signing this certificate can be obtained here, the certificate needs to be added as a trusted CA certificate in the sender's environment edelivery AS4 When chosen to apply for the conformance testing service via the use of an AS4 message over e- Delivery. An invoice may be submitted for validation using the GITB test bed instance that is available at in which the following test scenarios are possible: Submission using the full e-delivery message exchange, i.e. involving an SML lookup, an SMP query and finally submission of the message to the AS4-capable Access Point. Submission using direct querying of the SMP. In this case the SML lookup is skipped, assuming that the sender can be configured with a direct SMP address for test purposes. Submission using direct messaging to an AS4-capable Access Point. This assumes that the sender can be configured with a specific Access Point address for test purposes. Conformance Testing Page 14 / 23

15 3.4. Step 3: Execution & report feedback Purpose: execute the conformance tests and generate a test report Actors Process: Software / Service Provider CEF Support Team 1. The Software / Service Provider executes the conformance tests using the testing platform and based on the test assertions provided by the CEF einvoicing Building Block. 2. The Software / Service Provider resolves the failed tests (with or without the support of the CEF Support Team. 3. The CEF Support Team provides support to the Software / Service Provider during the execution of the conformance tests and assists in resolving failed tests. 4. The Software / Service Provider informs the CEF Support Team once all conformance tests have passed and the software product or implementation is ready for test report validation. 5. CEF Support Team validates the test report automatically generated by the testing platform and shares it with the CEF Support Team. The overview of the Classification process is shown in the diagram below Specific on technology chosen: When chosen for the conformance testing service via the user receives a reply on the sending address once validation is completed. A reply is returned with an overview of the validation as the body and the detailed validation reports (in GITB TRL format) as attachments. Conformance Testing Page 15 / 23

16 Specific on technology chosen: web form upload When chosen for the conformance testing service via web form upload the electronic invoice(s) uploaded, is validated and a detailed report is presented to the user. This report can be downloaded (in GITB TRL format) and cross-checked against the submitted content Specific on technology chosen: web service When chosen for the conformance testing service via web service call to a GITB-compliant content validation service will return a positive call response when compliant and a negative when not compliant Specific on technology chosen: offline tool When chosen for the conformance testing service via the use of an offline validation tool execution is facilitated via use of java -jar validator.jar (requires Java 8). Instructions on how to use the "jar" file are printed on the console Specific on technology chosen: edelivery AS2 When chosen for the conformance testing service via edelivery AS2 a test scenario can be executed (acting as sender) after the user logs into the test bed at using his assigned credentials. The selected test scenarios are listed under the Conformance Statement menu, with new scenarios added by selecting Create conformance statement. The scenarios relative are listed under domain e-invoicing and specification AS2 as follows: Sender_AS2: For a system that will use the full e-delivery message exchange. Sender_AS2_Direct_SMP: For systems that will directly query the SMP (i.e. skip the SML lookup). Sender_AS2_Direct_AP: For systems that will directly message the receiver Access Point (i.e. skip the SML lookup and SMP query). The SML, SMP and receiver Access Point are simulated by the test bed. In all cases, once the simulated receiver Access Point receives the AS2 message it will extract the invoice and validate it. Important to consider: When selecting to run a test case, the tester is prompted to supply configuration relating to the sender system and test case to the test bed under the Endpoints section. Specifically: The network.host must be set to the public IP address of the sender. Any messages received from other addresses are ignored. The network.port is currently not used. Any numeric value is acceptable. The public.key is the public key of the AS2 sender (*.cer) that is used to sign the AS2 message. The test cases for Sender_AS2 and Sender_AS2_Direct_SMP include an ap.location.cached parameter that accesses a "true" or "false" value. This is meant to accommodate sender implementations that cache an Access Point's address after a first query to the SMP. Setting this to "true" will result in the test bed not expecting an SMP query. Participant IDs are not considered in the test case. Any value can be configured in the sender for the receiver. Once the test case is started, in the Execute Test wizard, and specifically Step 2 Initiate Preliminary, the test bed is listing the addresses to use for its simulated actors. The values listed Conformance Testing Page 16 / 23

17 (e.g. the SMP or the receiver Access Point) are to be configured as appropriate in the sender implementation. Communication with the SMP is done over http, whereas communication with the receiver Access Point is done over https. Both business message encryption and signing are supported but are not required. The Sender_AS2 test case can only support one test session at a time, as the SMP must listen on port 80. The Sender_AS2_Direct_SMP and Sender_AS2_Direct_AP test cases can support any number of parallel test sessions by dynamically assigned port numbers. When running the Sender_AS2_Direct_AP it is possible to use an AS2 implementation directly without going through e-delivery. In this case however the message to be sent must still be wrapped in a StandardBusinessDocument with a StandardBusinessDocumentHeader header. The partner certificate for the receiver that is needed to support encryption and signing is provided here Specific on technology chosen: edelivery AS4 When chosen for the conformance testing service via edelivery AS4 a test scenario can be executed after the user logs into the test bed at using his assigned credentials. The selected test scenarios are listed under the Conformance Statement menu, with new scenarios added by selecting Create conformance statement. The scenarios relative are listed under domain e- Invoicing and specification AS4 as follows: Sender_AS4: For a system that will use the full e-delivery message exchange. Sender_AS4_Direct_SMP: For systems that will directly query the SMP (i.e. skip the SML lookup). Sender_AS4_Direct_AP: For systems that will directly message the receiver Access Point (i.e. skip the SML lookup and SMP query). The SML, SMP and receiver Access Point are simulated by the test bed. In all cases, once the simulated receiver Access Point receives the AS4 message it will extract the invoice and validate it. Important to consider: When selecting to run a test case, the tester is prompted to supply configuration related to the sender system and test case to the test bed under the Endpoints section. Specifically: The network.host must be set to the public IP address of the sender. Any messages received from other addresses are ignored. The network.port is currently not used. Any numeric value is acceptable. The test cases for Sender_AS4 and Sender_AS4_Direct_SMP include an ap.location.cached parameter that accesses a "true" or "false" value. This is meant to accommodate sender implementations that cache an Access Point's address after a first query to the SMP. Setting this to "true" will result in the test bed not expecting an SMP query. Participant IDs and pmodes are not considered in the test case. Any value can be configured in the sender for the receiver. Once the test case is started, in the Execute Test wizard, and specifically Step 2 Initiate Preliminary, the test bed is listing the addresses to use for its simulated actors. The values listed (e.g. the SMP or the receiver Access Point) are to be configured as appropriate in the sender implementation. Communication with the SMP is done over http, whereas communication with the receiver Access Point is done over http. Messages should not be encrypted. 6 To download this certificate you will need an EU login account with the required access settings. Conformance Testing Page 17 / 23

18 The Sender_AS4 test case can only support one test session at a time, as the SMP must listen on port 80. The Sender_AS4_Direct_SMP and Sender_AS4_Direct_AP test cases can support any number of parallel test sessions by dynamically assigned port numbers Step 4: Closing. Purpose: provide and publish the test report Actors: Process: Software / Service Provider CEF Support Team CEF Stakeholder Management Office (SMO) 1. Once the CEF Support Team receives the successful test report it signs the report and sends it to the Software / Service Provider as well as the CEF Stakeholder Management Office (SMO). 2. The Software / Service Provider receives the test report for his software product or implementation. The overview of the Investigation and Escalation process is shown in the diagram below: Conformance Testing Page 18 / 23

19 4. TERMS AND CONDITIONS The general terms and conditions of CEF Building Blocks can be consulted in the Master Service Arrangement, available on the CEF Digital Single Web Portal: The terms and conditions specific to the CEF einvoicing Conformance Testing service are described in the table below. Term / Condition Description It is the responsibility of the Client to : Obligations of the Client a) ensure that sufficient information and documents are provided in due time to enable the required services to be performed; b) fulfil the conformance requirements, including implementing appropriate changes when they are communicated by the European Commission. Delivery of the services The European Commission will provide the conformance testing services using reasonable care and skill and in accordance with the technical specifications and procedures outlined in this document. The timing of completion of the conformance testing services and the delivery of the test results is based on best effort. Warranty disclaimer The conformance testing services and the results thereof are provided on the basis of the information, documents and implementations supplied by the Client. The test report issued by the European Commission is issued on an AS-IS basis and will reflect the facts as recorded at the time of testing. The European Commission is under no obligation to refer to, or report upon, any facts or circumstances which are outside the scope of this service and the procedures outlined in this document. Limitation of liability In no event shall the European Commission be liable to the Client or anyone claiming through the Client, for any kind of damages resulting from reliance on the test results. Conformance Testing Page 19 / 23

20 Proof of Conformance If the product/implementation passes all mandatory test assertions, the Client will receive a Proof of Conformance in the form of a test report. It should be clear that this test report has no legal value. It is a technical report, outlining the test results based on the version of the Test platform and test cases in effect at the time of testing. The Proof of Conformance (test report) is restricted only to the product-with version that was subject to testing for a specific test run at a specific date. Other products offered by the Client or versions of that product which have not undergone testing are not considered to be conformant. The Client is entitled to disclose the fact that the product/implementation passed the conformance test to third parties. Notwithstanding the following conditions: Disclosure Liability a) there should be clear mention of the product-with version to which the Proof of Conformance applies, labelled with the specific test run and date; b) the test reports can only be reproduced in their entirety, if provided by the Client to third parties; c) the test report should not be used in such a manner as to bring the European Commission into disrepute and not make any statement regarding its Proof of Conformance that the European Commission may consider misleading or unauthorized; The conformance testing services use is not intended for production data. Use of production data is discouraged. Conformance Testing Page 20 / 23

21 5. GLOSSARY The key terms used in this Component Offering Description are defined in the CEF Definitions section on the CEF Digital Single Web Portal: The key acronyms used in this Component Offering Description are defined in the CEF Glossary on the CEF Digital Single Web Portal: ary Conformance Testing Page 21 / 23

22 6. ANNEXE 1 DOCUMENT PARTS Process Overview.vsd Registration.vsd Preparation.vsd Execution.vsd Closing.vsd Conformance Testing Page 22 / 23

23 7. CONTACT INFORMATION CEF Support Team By By phone: Standard Service: 8am to 6pm (Normal EC working Days) Standby Service*: 6pm to 8am (Commission and Public Holidays, Weekends) * Only for critical and urgent incidents and only by phone Conformance Testing Page 23 / 23