EDC Documentation. Release 1.0. ONS Digital

Transcription

1 EDC Documentation Release 1.0 ONS Digital Nov 13, 2018

2

3 Contents 1 Introduction License JWT Profile UUID Definition JWS Protected Header JWT Payload Signing JWE Header Encrypting Decrypting Verifying Respondent Management to Electronic Questionnaire Schema Definition Required Fields 11 5 Optional Fields Per Survey Metadata An example JSON claim JWT envelope / transport Flushing responses Electronic Questionnaire to Survey Data Exchange Low-level datatypes Schema Definition Example Json payloads JWT envelope / transport Survey Data Exchange to and from Respondent Account Services Schema Definition Indices and tables 21 i

4 ii

5 Contents: Contents 1

6 2 Contents

7 CHAPTER 1 Introduction This website represents a repository of schema standards for data exchange used and published by the Office for National Statistics (ONS). License The project is licensed under the MIT license and is covered by Crown Copyright 3

8 4 Chapter 1. Introduction

9 CHAPTER 2 JWT Profile Where a definition uses JWT it SHALL conform to the following profile: The payload is contained within a JSON Web Tokens (JWT) ~ RFC 7519 signed as per JSON Web Signature (JWS) ~ RFC The signed JWT is used as the payload to a JSON Web Encryption (JWE) ~ RFC All definitions are as per the RFCs unless otherwise stated. Note: All times will be UTC. UUID Definition All references to UUIDs refer to UUID (version 4); 128-bits in length as defined in RFC 4122 in their textual representation as defined in section 3 Namespace Registration Template without the urn:uuid: prefix e.g. f81d4fae- 7dec-11d0-a765-00a0c91e6bf6. All UUIDs MUST be randomly generated such that there is negligible probability that the same value will be used twice, even across multiple servers. The same UUID value MUST NOT appear twice in the same JWT in any claim. JWS Protected Header JWS Protected Header SHALL contain the following claims: typ will be set to JWT alg will be set to RS256 kid will be set to the SHA-1 hash of the digest of the Public Key for the Private Key used to sign (reference and ) 5

10 JWT Payload All JWT payloads SHALL include the follow claims: tx_id - Set to a random UUID. Transaction ID used to trace a transaction through the whole system. MUST NOT be the same as the jti value. jti - Set to a random UUID. See RFC 7519 definition of the jti claim. jti claim MUST NOT be the same value as the tx_id in the JWT Payload (see UUID Definition). The JWT payload can also contain specific other data (claims) defined in the definitions within this website. Signing JWT Protected Header and JWT Payload are signed with RSASSA-PKCS1-v1_5 using SHA-256 (RSA-SHA256) using a Private Key to form a JWS. JWE Header JWE header will include the alg and enc claims: alg will be set to RSA-OAEP enc will be set to A256GCM kid will be set to the SHA-1 hash of the digest of the Public Key used to encrypt (reference ietf.org/html/rfc7517#section-4.5 and Encrypting The encrypted signed JWT is used as the payload to a JWE. The JWE header is used as the Additional Authenticated Data A 256-bit random CEK and 96-bit IV is generated by the creator of the JWT. The entire JWT is encrypted with AES-256-GCM using the generated CEK and IV and the Additional Authenticated Data. This process outputs a 128-bit Authentication Tag. The CEK is encrypted with the JWT receivers public key using the RSAES-OAEP algorithm to produce the JWE Encrypted Key The JWE is formed from the above values concatenated together (see RFC7516). Decrypting JWE payload is decrypted (reverse of encryption process) by JWT receiver using their private key; revealing the signed JWT. 6 Chapter 2. JWT Profile

11 Verifying Signature of JWT verified by the receiver using the Public Key of the creator of the JWT. Only the RSA-SHA256 algorithm will be accepted, all other algorithms (including None) are rejected Verifying 7

12 8 Chapter 2. JWT Profile

13 CHAPTER 3 Respondent Management to Electronic Questionnaire When a respondent is ready to take a survey hosted on the eq system, a set of details need to be passed to eq to setup the survey correctly. This data is wrapped inside a json web token which is attached to the end of a url, digitally signed and authorised by a respondent management system. This creates a clean interface for any respondent management system to integrate with the eq system. Schema Definition 9

14 10 Chapter 3. Respondent Management to Electronic Questionnaire

15 CHAPTER 4 Required Fields The following metadata keys are always required for the survey runner, they do not appear in individual survey metadata definitions. tx_id see JWT Profile iat JWT Issued At claim, see exp JWT Expiration Time claim, see ru_ref The responding unit reference id - with checkletter appended eq_id The eq questionnaire id collection_exercise_sid A reference number used to represent the collection exercise inside the ONS form_type The particular form_type for a responding unit ru_ref The responding unit reference id - with checkletter appended. ru_name The name of the responding unit. Could be a business name or person name. Not required, however at least one of ru_name and trad_as must be present trad_as Temporary until wider refactor. Not required, however at least one of ru_name and trad_as must be present user_id The id assigned by the respondent management system period_id A numerical reference to either a month or quarter time period case_id The case UUID used to identify a single instance of a survey collection for a respondent account_service_url The url of the account service (i.e. rrm or ras) used to launch the survey response_id A unique identifier for the questionnaire response 11

16 12 Chapter 4. Required Fields

17 CHAPTER 5 Optional Fields The runner can optionally accept the following keys. period_str A display name for the period_id referenced above language_code Language code identifier, used to change language displayed. Format as per ISO (https: //en.wikipedia.org/wiki/list_of_iso_639-1_codes) e.g. en for English; cy for Welsh. This parameter is currently optional; the default is en survey_url A URL for a remote survey JSON. This claim is used to tell Survey Runner to load the schema JSON from a remote location case_ref The case reference identified by the above UUID (e.g ) account_service_log_out_url The logout url of the account service used to launch the survey. Not required for services that don t have a log in function (i.e., respondent home) Per Survey Metadata In addition to the above required fields, some surveys require other data be passed. These can simply have their keys added as a claim in the main JWT body. e.g. {"language_code": "en"}. An example JSON claim { "tx_id": "0f534ffc c-b39f-a756b4adc6cb", "iat": , "exp": , "user_id": " ", "ru_ref": " J", "ru_name": "", "eq_id": "678", "collection_exercise_sid": "789", 13

18 } "period_id": "", "period_str": "", "ref_p_start_date": "", "ref_p_end_date": "", "employment_date": "", "trad_as": "", "form_type": "", "return_by": "YYYY-MM-DD", "region_code": "GB-GBN", "language_code": "en", "flag_1": true, "roles": [ "role1", "role2" ], "response_id": "QzXMrPqoLiyEyerrED88AbkQoQK0sVVX72ZtVphHr0w=" JWT envelope / transport This payload is part of a JWT as specified in JWT Profile. The encoded JWT is appended to the URL of the receiving system as follows: Flushing responses To flush responses to the downstream systems a /flush endpoint is available. This endpoint takes a JWT in the same way as /session but with roles including the role of flusher 14 Chapter 5. Optional Fields

19 CHAPTER 6 Electronic Questionnaire to Survey Data Exchange All collected responses for a collection exercise (a questionnaire within a survey series) are transformed into the following described data format for downstream processing and refinement. The json document is encrypted using the public key of the downstream collection system at submission, placing the cyphertext onto a rabbitmq queue for consumption by eq-submitter component. Low-level datatypes All datetimes are expressed using ISO_8601 and are assumed to be normalised to UTC unless a timezone identifier is given. All character encoding is UTF-8. All Boolean responses are matched to a True or False string representation. The ru_ref is appended by default with a check letter for all responses. If a question has no answer and is optional, we will not give an null or empty string entry for the downstream system. Schema Definition tx_id Transaction ID used to trace a transaction through the whole system. This will be a GUID (version 4) and 128-bits in length as defined in RFC 4122 in its textual representation as defined in section 3 Namespace Registration Template without the urn:uuid: prefix e.g. f81d4fae-7dec-11d0-a765-00a0c91e6bf6. type The unique type identifier of this JSON file. uk.gov.ons.edc.eq:feedback Can be uk.gov.ons.edc.eq:surveyresponse or version The version number of the schema definition used to generate and parse the schema. Will always be 3 numbers separated by two dots e.g with the intention being MA- JOR.MINOR.PATCH no guarantees are given to compatibility across version changes. 15

20 origin The name or identifier of the data capture / data generator system. Currently, the only option for this is uk.gov.ons.edc.eq - however this allows us to futureproof for new collection instruments. survey_id The numerical survey number as used across the ONS. case_id The case UUID used to identify a single instance of a survey collection for a respondent [optional] case_ref The case reference identified by the above UUID (e.g ) [optional] flushed Whether the survey was flushed or not. This will be true if the survey has been flushed through EQ (surveys that haven t been submitted could be flushed through at the end of their collection period) and false otherwise. collection exercise_sid Collection exercise ID instrument_id The collection instrument ID - used by legacy downstream systems. Referenced by some systems as the form_type. period The collection period, currently represented as a string due to downstream systems lack of support for correct date formats. started_at The datetime of the first answer saved in a survey submitted_at The datetime of submission by the respondent. metadata data user_id The respondent user_id as specified by the respondent management system in use. ru_ref Reporting Unit reference number to which the collected data represents. This allows the downstream system to map the responses to individual business/household/person in the original sample as created by the survey team. Version A key, value pairing of responses from a respondent, making use of the box_code / stat_var / q_code as the key to identify the captured respondence from a user. Version data example { } "001": " ", "002": " " Version A sorted array of answers in the order the questionnaire was answered. Dictionary of values value: The answer given in the questionnaire for the answer. answer_id: The identifier of the answer. group_instance: The sub-identifier of a group of block_id/pages if that group repeats. answer_instance: The sub-identifier of an answer if that answer repeats. Version data example [{ "value": "Joe", "answer_id": "household-first-name", "group_instance": 0, 16 Chapter 6. Electronic Questionnaire to Survey Data Exchange

21 }, { }] "answer_instance": 0 // Example of multiple values for a checkbox question "value": ["Eggs", "Bacon", "Spam"], "answer_id": "favourite-breakfast-food", "group_instance": 0, "answer_instance": 0 Example Json payloads { } "tx_id": "0f534ffc c-b39f-a756b4adc6cb", "type" : "uk.gov.ons.edc.eq:surveyresponse", "version" : "0.0.1", "origin" : "uk.gov.ons.edc.eq", "survey_id": "021", "flushed": false, "collection":{ "exercise_sid": "hfjdskf", "instrument_id": "yui789", "period": " " }, "started_at": " T14:10:08Z", "submitted_at": " T15:28:05Z", "metadata": { "user_id": " ", "ru_ref": " " }, "data": [{ "value": "Joe", "answer_id": "household-first-name", "group_instance": 0, "answer_instance": 0 }, { "value": ["Eggs", "Bacon", "Spam"], "answer_id": "favourite-breakfast-food", "group_instance": 0, "answer_instance": 0 }] { "type" : "uk.gov.ons.edc.eq:feedback", "origin" : "uk.gov.ons.edc.eq", "metadata": { "user_id": " ", "ru_ref": " " }, "data": { "url": " "name": "John Appleseed", " ": "john.appleseed@ons.gov.uk", 6.3. Example Json payloads 17

22 } "message": "Feedback message string" }, "started_at": " T14:00:59Z", "submitted_at": " T15:28:05Z", "collection": { "instrument_id": "0001", "exercise_sid": "739", "period": " " }, "survey_id": "021", "tx_id": "0f534ffc c-b39f-a756b4adc6cb", "version" : "0.0.1" JWT envelope / transport This payload is part of a JWT as specified in JWT Profile. 18 Chapter 6. Electronic Questionnaire to Survey Data Exchange

23 CHAPTER 7 Survey Data Exchange to and from Respondent Account Services Collection Instruments are uploaded from the ONS estate to a RabbitMQ queue for consumption by RAS. To facilitate this the excel files are base64 encoded and then wrapped in a JWT. This JWT is encrypted and signed before being placed on the queue. Once the Collection Instruments have been completed by the respondent RAS re-encodes them and wraps them in a signed and encrypted JWT to be transferred back into the ONS estate. Schema Definition tx_id see JWT Profile file The contents of the Collection Instrument base64 encoded filename The name of the collection instrument file case_id The case UUID 19

24 20 Chapter 7. Survey Data Exchange to and from Respondent Account Services

25 CHAPTER 8 Indices and tables genindex search 21