Corporate Data Warehouse File Extract Specification CHAD Project District Nursing Data Mart

Transcription

1 National ervices cotland Architecture & olutions ervice N Information Technology Corporate Data Warehouse File Extract pecification CHAD Project District Nursing Data Mart Version 1.0 Page 1 of 22

2 Document Control Version Date Author Description /06/2015 Angela Forbes Initial draft /06/2015 Angela Forbes Other Aim of Care (4) and Other Intervention (4) added and validation updated accordingly /07/2015 Angela Forbes Contact tart Date changed to Contact Date /07/2015 Angela Forbes Denise added to Business Contacts /07/2015 Angela Forbes For sign off /07/2015 Angela Forbes igned off by Margaret P in Lee s absence upporting Documentation Document Business User Requirements Reference Information Functional Requirements Additional Information Help Text Document Corporate Data Warehouse ecurity Owner Business N IT N IT N IT N IT N IT Business Contacts Name Role Telephone Elaine McNish Data Manager elainemcnish@nhs.net Margaret Parsons Information Manager Margaretparsons1@nhs.net Margaret Quinn Principal Information Development Margaret.quinn@nhs.net Manager Lee Davies ervice Manager Lee.davies2@nhs.net Denise Hastie enior Information Analyst denisehastie@nhs.net N IT Contacts Name Role Telephone Irene Gow Project Manager i.gow@nhs.net Angela Forbes Analyst Angela.forbes@nhs.net Neil inclair Technical Lead neilsinclair@nhs.net Jill mith Head of Business Intelligence & Corporate Applications Jill.smith1@nhs.net Version 1.0 Page 2 of 22

3 Glossary Acronym/Word BI BOXI BOXI Universe Case CDD CDW CHI Contact Data mart Episode ETL Fact WIFT Description Business Intelligence. Business Objects XI is a web-based reporting tool that allows users to create reports which can be made available to other users who have the appropriate access permissions. The BOXI universe is the reporting environment which is built on and allows access to the associated data mart. A case is used to identify a patient s single episode with the nursing service. Within a single case a patient may have multiple contacts with health care professionals. Common Dimensions Database the database which holds all of the common dimension tables used by the data marts within the Corporate Data Warehouse. The Corporate Data Warehouse is a central data store maintained by N IT. It is made up of a collection of independent data marts which share a common database for reference information. Community Health Index number a unique patient identifier which is issued to residents in cotland when they register with NH cotland. An interaction between the patient and the District Nurse during an episode of care. A subset of a data warehouse which has been designed to satisfy a particular function or reporting area within an organisation. A data mart can be designed and developed independently of other data marts. A data warehouse will contain multiple data marts. The period of care whilst the patient is part of the District Nurse s caseload. Extract, Transform and Load - The process of extracting data from external sources, transforming the data if necessary to meet operational needs and loading it into the data warehouse. A fact is a table in a data mart which contains data, usually measures, which can be reported on. The measures can usually (but not always) be aggregated at a dimensional level, e.g. a count of patients can be aggregated to give the counts per location, organisation etc. ubmission with internet file transfer Version 1.0 Page 3 of 22

4 Contents 1. Introduction Purpose Overview General Data Assumptions File Extracts File ubmission File Processing File Processing chedule Validation Accessing Validation Results Automated s File Format File Name Header Record General File Rules Data Element Rules Episode File Content Contact File Content Appendix A Data Validation pecification Appendix A ending Locations Version 1.0 Page 4 of 22

5 1. Introduction 1.1. Purpose The purpose of this document is to detail the file schemas and processing rules for files submitted to a District Nursing data mart. The signed off document will be used in conjunction with other supporting documentation by the BI team to develop the data mart as specified. This document is intended to give details of the file specifications and how the files will be processed into the data mart. It is not intended to provide recording guidance Overview District nursing data will be submitted by the NH Boards and will be held in the Corporate Data Warehouse in a District Nursing data mart. This will be delivered as part of the first phase of the Community Health Activity Data (CHAD) project General Data Assumptions The following general assumptions have been made about the data being submitted All files will be sent from a single location within a NH Board, i.e. multiple sites will not be responsible for submitting information for the NH Board. The NH Board code used in the header record to identify which NH Board has submitted the data will be included as a data item in the data mart to identify the NH Board of Treatment. A patient can have more than one episode or contact. An episode can have zero or many contacts. A contact does not have to be associated with an episode. There is no interdependency between the files, e.g. a submission does not need to contain both an episode and a contact record for the patient. Unique Record ID It must be possible to uniquely identify an episode and contact record to avoid duplicate records being processed and to allow records to be updated if required. All NH Boards will be able to provide a system generated Unique Record ID o The Episode ID will uniquely identify an episode within the NH Board. The Episode ID will be fixed for a given episode and will not differ between the original episode and any update to that episode. This will allow Episode information to be updated. o The Contact ID will uniquely identify a contact within the NH Board. The Contact ID will be fixed for a given contact and will not differ between the original contact and any update to that contact. This will allow Contact information to be updated. Version 1.0 Page 5 of 22

6 2. File Extracts Extract Type Description Number of Records per file EPIODE The file will contain episode records which have been added or updated during the previous quarter. 50,000 per quarter CONTACT An episode starts when a patient is added to a District Nurse s caseload and ends when the patient is discharged from the caseload. The file will contain contact records which have been added or updated during the previous quarter. 200,000 per quarter A contact is an interaction between the patient and District Nurse during an episode of care. An episode can have multiple contacts. NOTE 1. There is no requirement for a separate historic data load. Any historic data submissions will conform to this specification. 3. File ubmission The files will be submitted using WIFT. WIFT is currently being used to submit test files. The destination location of the WIFT account will be modified so that files are transferred to the source directory for the data mart. 4. File Processing The files will be processed using an insert and update process. The following logic will be applied If a record is submitted and the Unique Record ID does not exist in the data table the record will be inserted/added to the data table. If a record is submitted and the Unique Record ID already exists in the data table it will overwrite the record already held in full - it is not a selective update so the update record must contain all of the relevant information. It is important that the latest record submitted for a Unique Record ID contains all of the relevant information. Records can be submitted or resubmitted from any historic timeframe if required. Records cannot be automatically deleted. If a record needs to be deleted a fast track CR will be raised to have the record removed. Version 1.0 Page 6 of 22

7 4.1. File Processing chedule The automated file load process will run at 6pm on weekdays. The file polling will increase 6 weeks after the end of a Quarter to run three times daily at 11am, 3pm and 6pm for 2 weeks only. 5. Validation The validation rules are specified in Appendix A. If any records fail a validation rule the file will fail and the error must be corrected and the file resubmitted. The file will go through two stages of validation as detailed in the table below. For the data validation to begin the file must pass the initial file validation stage. Validation tage File Error Type Description Outcome of Validation Rule Failure File Name Header chema The format of the file is validated, e.g. file name, header record, number of columns. The file will fail. The error must be corrected by the NH Board and the file resubmitted. Data Data The content of the file is validated. The record will fail. The error must be corrected by the NH Board and the record resubmitted Accessing Validation Results Validation results will be accessed using Business Objects. 6. Automated s The following s will be generated. The s can be sent to multiple recipients. Description Recipient Passed Contains details of the file submitted and that it passed validation. NH Board ( address(s) to be provided). CHAD support mailbox N.CommunityActivityData@nhs.net Failed Contains details of the file submitted. The number of records processed. The number of records which passed validation. The number of records which failed validation. If the file has failed totally it will include an Version 1.0 Page 7 of 22 ID Data Management mailbox nss.datamarts@nhs.net NH Board ( address(s) to be provided). CHAD support mailbox N.CommunityActivityData@nhs.net

8 Description Recipient indication of why the file could not be processed. ID Data Management mailbox Loaded The data will not be available immediately after NH Board ( address(s) to be validation because of the processing required. provided). This will advise when the data are available for analysis. CHAD support mailbox ID Data Management mailbox 7. File Format The data extract(s) will follow the general standards File Name The file will fail validation if the file name does not conform to the following format. DATAMART_EXTRACT TYPE_ENDING LOCATION_EXTRACT DATE. EXTENION e.g. DNURE_EPIODE_N_ csv Where: Item Description Example Data Mart Identifies the data mart which the files are submitted to DNURE Extract Type The type of file extract: Episode or Contact EPIODE ending Location The cipher for the sending location e.g. N = Grampian N ee Appendix A for valid sending locations. Extract Date The date the file was compiled. Date format: YYYYMMDD File extension Identifies the type of file, must be a.csv.csv 7.2. Header Record The first row in the file must contain a header record which contains details of the file. All data items in the header record are mandatory and must be in the specified format. The information in the header record should be consistent with the information supplied in the file name. The file will fail validation if the header format does not conform to the following comma delimited format. DATA MART,EXTRACT TYPE,ENDING LOCATION,EXTRACT DATE,UBMIION REFERENCE NUMBER,RECORD COUNT e.g. DNURE,EPIODE,N, ,1,12100 Version 1.0 Page 8 of 22

9 Where: Item Format Description Example Data Mart Varchar (6) Identifies the data mart which the files are DNURE submitted to Extract Type Varchar (15) The type of file extract: Episode or Contact EPIODE ending Location Varchar (1) The cipher for the sending NH Board e.g. N = N Grampian. ee Appendix B for valid sending locations. Extract Date Varchar (8) The date the file was compiled. Date format: YYYYMMDD ubmission Integer (6) A unique number identifying the number of the 1 Reference Number submission for the sending location. Record Count Integer A count of the number of records in the file (excluding the header record and column names) NOTE 1. The header record is not case sensitive. 2. The ubmission Reference Number is required to allow multiple files for the same location to be processed in the correct order, and to flag if there are any gaps in the received files. The ubmission Reference Number should increment per successful validation for each sending location. Files which are invalid and rejected for any reason must be corrected and resubmitted with the same ubmission Reference Number as the original. 3. There should be no characters (including commas) after the last item General File Rules The following table include the general rules which apply to the file format Category Columns Column Names Column Width Blank Rows Null/Missing Values paces (nonprintable values) Description The columns are comma delimited to separate fields and must be in the order specified in the file specification. If a field is empty the column must still be included in the extract. The file will contain column name headings. The headings are for local use only and will be ignored when the file is loaded. The name can only contain a comma when surrounded by double quotes. The column width is the minimum width from a development perspective. The data mart will be developed to process columns with a width wider than documented to ensure that the field is not truncated to a valid value. Blank data rows must not be included in the data file. Empty fields will indicate where a value is null (missing). Zero must not be used. paces should not be present between fields. The comma alone should delimit fields. Version 1.0 Page 9 of 22

10 7.4. Data Element Rules The following table includes the data element rules which apply to the data in the file Category Number Type Text/Character Type Description Numeric data items will appear in the text file as a string of continuous numeric digits. For large numbers the thousand display format characters such as the comma will not be included. Where the number contains decimals, the period character. will be used as the decimal point. The text string will contain any alphanumeric character including the space character other than double quotes. The format of the text field will be defined as Varchar (n) where Varchar (n) is a variable length string of maximum length n. Free text fields should be qualified/surrounded by double quotes. Codes and descriptions will be converted to upper case for all data items. Date Type A column containing date strings will be formatted as DDMMCCYY (e.g ). Time Type Leading Zeros If a 7 digit date is submitted it will be left zero padded to 8 digits. A column containing date time strings will be formatted as HH:MM using the 24 hour clock (e.g. 17:35). Leading zeros must not be stripped off codes which have been identified as having leading zeros. Data Item Priorities If a 9 digit CHI number is submitted it will be left zero padded to 10 digits. The priority categories for including data items are as follows. M = Mandatory the data item must be populated = hould be populated if available O = Optional 7.5. Episode File Content The table below gives details of each data item in order it will appear in the Episode file. Data Item Description Format Priority Episode ID Identifier used within the NH Board to uniquely identify an Integer (100) M episode. CHI Number The identifier used by NH cotland to uniquely identify a patient. Varchar (10) 9 digit codes will be left zero padded to 10 digits. urname The surname of the patient. Varchar (35) Required for CHI seeding. Version 1.0 Page 10 of 22

11 Must be provided if a valid CHI Number is not Forename The first forename of the patient. Varchar (35) Required for CHI seeding. Must be provided if a valid CHI Number is not Date of Birth The patient s full date of birth. Date Required for CHI seeding and will be used to derive age where a valid CHI number is not available. Must be provided if a valid CHI Number is not Postcode The patient s postcode of residence. Varchar (8) Required for CHI seeding and will be used to derive the area of residence where a valid CHI number is not available. Must be provided if a valid CHI Number is not Gender The gender of the patient, e.g. 1 = Male, 2 = Female (see the reference information document for valid codes). Varchar (1) Required for CHI seeding and to populate gender where a valid CHI number is not available. Must be provided if a valid CHI Number is not Ethnicity The client s ethnic group (see the reference information document for valid codes). Varchar (2) Date Referral Received The date the healthcare service receives a referral (not the date in the referral letter). Date The date could be after the Contact Date. ource of Referral The source of referral code. Varchar (1) Main Aim of Care A desired achievement of a programme of care e.g. the primary Varchar (1) reason for a referral to a Community Health Care Professional. Other Aim of Care Other desired achievement of a programme of care, but not Varchar (1) O (1) deemed to be the primary desired achievement. Other Aim of Care (2) Other Aim of Care (3) Other Aim of Care (4) Other desired achievement of a programme of care, but not deemed to be the primary desired achievement. Other desired achievement of a programme of care, but not deemed to be the primary desired achievement. Other desired achievement of a programme of care, but not deemed to be the primary desired achievement. Version 1.0 Page 11 of 22 Varchar (1) Varchar (1) Varchar (1) Episode tart The date care began for the episode. Date O O O

12 Date Episode End Date The date care finished for the episode. Date Outcome of Episode The reason the patient was discharged from the episode of care. Varchar (1) NOTE 1. The recording guidance states that CHI number is mandatory however if it cannot be provided the demographic information must be submitted and this will be used for CHI seeding digit dates will be left zero padded to 8 digits. 3. A combination of the ending Location and Episode ID will uniquely identify a record in the data mart. 4. Episode tart Date is not mandatory because it is likely that initially the episode data will be less complete than the contact data. As the episode data completeness improves a change request may be raised to make the Episode tart Date mandatory Contact File Content The table below gives details of each data item in order it will appear in the Contact file. Data Item Description Format Priority Contact ID Identifier used within the NH Board to uniquely identify a contact. Integer (100) M Episode ID Identifier used within the NH Board to uniquely identify an Integer (100) episode. CHI Number The identifier used by NH cotland to uniquely identify a patient. Varchar (10) 9 digit codes will be left zero padded to 10 digits. urname The surname of the patient. Varchar (35) Required for CHI seeding. Must be provided if a valid CHI Number is not Forename The first forename of the patient. Varchar (35) Required for CHI seeding. Must be provided if a valid CHI Number is not Date of Birth The patient s full date of birth. Date Required for CHI seeding and will be used to derive age where a valid CHI number is not available. Must be provided if a valid CHI Number is not Postcode The patient s postcode of residence. Varchar (8) Version 1.0 Page 12 of 22

13 Required for CHI seeding and will be used to derive the area of residence where a valid CHI number is not available. Must be provided if a valid CHI Number is not Gender The gender of the patient, e.g. 1 = Male, 2 = Female (see the reference information document for valid codes). Varchar (1) Required for CHI seeding and to populate gender where a valid CHI number is not available. Must be provided if a valid CHI Number is not Ethnicity The client s ethnic group (see the reference information document for valid codes). Varchar (2) Contact Date The date the contact between the Healthcare Professional and the patient began within an Episode of Care. Date M Contact tart Time The time the contact between the Healthcare Professional and the patient began. Time Contact End Time The time the contact between the Healthcare Professional and the patient ended. Time Duration of Contact The length of time contact occurred between the Healthcare Professional and the patient in minutes. Varchar (5) Location of Contact Patient Contact Category Planned/Unplann ed Must be completed if Contact tart Time and Contact End Time are not recorded. The type of location where the contact occurred, e.g. Health Varchar (1) Centre. The category of patient contact, e.g. direct, indirect or other. Varchar (1) Indicates if the contact with the patient was planned or unplanned. Varchar (1) Joint Contact Indicates if two or more staff members were required for the patient contact. Varchar (1) Number of staff present in joint contact The value can be 0 or 1 where 1 indicates that joint contact was required. The number of staff members in attendance during a joint visit. Varchar (1) Visit tatus Indicates if the patient attended/was seen. Varchar (1) Patient Related Other patient related activities undertaken by the Healthcare Varchar (1) Activity Type Professional as a result of the contact. Duration of The number of minutes taken by staff to carry out any patient Varchar (5) Version 1.0 Page 13 of 22

14 Patient Related Activity Travel Time related activities which serve to enhance the delivery of care. The number of minutes taken to travel by the staff member to the location of contact. taff Pay Band The current staff band as featured on staff payslip. Varchar (2) Time ervice Team The name of the ervice Team. Varchar (100) Free text Primary Intervention An activity which is undertaken to maintain or potentially improve the patient s state of health and well-being, relieve distress or reduce risk. Varchar (8) Other Intervention (1) Additional activity other than the main activity s undertaken to maintain or potentially improve the patient's state of health and well-being, relieve distress or reduce risk. Varchar (8) O Other Intervention (2) Additional activity other than the main activity s undertaken to maintain or potentially improve the patient's state of health and well-being, relieve distress or reduce risk. Varchar (8) O Other Intervention (3) Additional activity other than the main activity s undertaken to maintain or potentially improve the patient's state of health and well-being, relieve distress or reduce risk. Varchar (8) O Other Intervention (4) Additional activity other than the main activity s undertaken to maintain or potentially improve the patient's state of health and well-being, relieve distress or reduce risk. Varchar (8) O NOTE 1. Demographic information and episode specific information is included in the Contact file because not all NH Boards will be able to provide the Episode file initially. 2. Contact End Date is not included and it will be assumed that the contact will start and end on the same date even though some visits could straddle midnight digit dates will be left zero padded to 8 digits. 4. A combination of the ending Location and Contact ID will uniquely identify a record in the data mart. Version 1.0 Page 14 of 22

15 8. Appendix A Data Validation pecification The table below includes details of the File Validation and Data Validation Errors. The File Extract Type ALL refers to both the Episode and Contact File Type. If a validation rules needs to be turned off a Change Request will be raised. File Extract Type Error Code Error Type Error everity Error Message ALL FE1 File Name Error Invalid File Name: Data mart is missing or not recognised ALL FE2 File Name Error Invalid File Name: File extract type is missing or not recognised ALL FE3 File Name Error Invalid File Name: ending location is missing or not recognised ALL FE4 File Name Error Invalid File Name: Extract Date is missing or invalid. hould be a date in the format YYYYMMDD ALL HE1 Header Error Invalid Header Record: Could not be read check format ALL HE2 Header Error Invalid Header Record: Data mart is missing, not recognised or does not match the file name ALL HE3 Header Error Invalid Header Record: File extract type is missing, not recognised or does not match the file name ALL HE4 Header Error Invalid Header Record: ending Location is missing, not recognised or does not match the file name ALL HE5 Header Error Invalid Header Record: Extract Date is missing, invalid or does not match the file name. hould be a date in the format YYYYMMDD. ALL HE6 Header Error Invalid Header Record: The ubmission Reference supplied is out of sequence for the supplied ending Location ALL HE7 Header Error Invalid Header Record: The Additional Information Must be DNURE Must be EPIODE or CONTACT Must be one of A, B, F, G, H, L, N, R,, T, V, W, Y and Z. Version 1.0 Page 15 of 22

16 number of records is inconsistent with the number of records submitted ALL E1 chema Error Data record could not be read check file format EPIODE E2 chema Error Episode ID must be provided and must be unique within the file CONTACT E3 chema Error Contact ID must be provided and must be unique within the file ALL VE1 Data Error CHI number must be valid CHI number has been 9-digit codes will be left zero padded to 10 digits. ALL VE2 Data Error urname must be provided and in the correct format CHI number has not been ALL VE3 Data Error Forename must be provided and in the correct format The following criteria must be met The first character must be alphabetic. Can contain alphabetic characters, a hyphen, a space or an apostrophe. Must be a minimum of 2 characters. Maximum field length 35 CHI number has not been The following criteria must be met The first character must be alphabetic. Can contain alphabetic characters, a hyphen, a space or an apostrophe. Must be a minimum of 2 characters. Version 1.0 Page 16 of 22

17 ALL VE4 Data Error Date of Birth must be provided and must be a valid date in the format DDMMCCYY Maximum field length 35 CHI number has not been ALL VE5 Data Error Postcode must be provided and must be valid CHI number has not been ALL VE6 Data Error Gender must be provided and must be valid CHI number has not been ALL VE7 Data Error Ethnicity must be valid Ethnicity has been EPIODE VE8 Data Error Date Referral Received must be a valid date in the format DDMMCCYY Date Referral Received has been EPIODE VE9 Data Error ource of Referral must be valid ource of Referral has been Must not be 0 (Community Health ervice) 0 is not valid for this dataset. EPIODE VE10 Data Error Main Aim of Care must be valid Main Aim of Care has been Version 1.0 Page 17 of 22

18 EPIODE VE11 Data Error Other Aim of Care (1) must be valid Other Aim of Care (1) has been EPIODE VE12 Data Error Other Aim of Care (2) must be valid Other Aim of Care (2) has been EPIODE VE13 Data Error Other Aim of Care (3) must be valid Other Aim of Care (3) has been EPIODE VE14 Data Error Other Aim of Care (4) must be valid Other Aim of Care (4) has been EPIODE VE15 Data Error Episode tart Date must be provided and must be a valid date in the format DDMMCCYY Episode tart Date has been EPIODE VE16 Data Error Episode End Date must be provided and must be a valid date in the format DDMMCCYY Episode End Date has been EPIODE VE17 Data Error Outcome of Episode must be valid Outcome of Episode has been Version 1.0 Page 18 of 22

19 CONTACT VE18 Data Error Contact Date must be provided and must be a valid date in the format DDMMCCYY CONTACT VE19 Data Error Contact tart Time must be a valid time in the format HH:MM Contact Date is mandatory. Contact tart Time has been CONTACT VE20 Data Error Contact End Time must be a valid time in the format HH:MM Contact End Time has been CONTACT VE21 Data Error Duration of Contact must be numeric Duration of Contact has been CONTACT VE22 Data Error Location of Contact must be valid Location of Contact has been CONTACT VE23 Data Error Patient Contact Category must be valid Patient Contact Category has been CONTACT VE24 Data Error Planned/Unplanned must be valid Planned/Unplanned has been CONTACT VE25 Data Error Joint Contact must be 0 or 1 Joint Contact has been CONTACT VE26 Data Error Number of taff Present in Joint Contact must be numeric Number of taff Present in Joint Contact has been Version 1.0 Page 19 of 22

20 Must be a whole number. CONTACT VE27 Data Error Visit tatus must be valid Visit tatus has been CONTACT VE28 Data Error Patient Related Activity Type must be valid Patient Related Activity Type has been CONTACT VE29 Data Error Duration of Patient Related Activity must be numeric Duration of Patient Related Activity has been CONTACT VE30 Data Error Travel Time must be numeric Travel Time has been CONTACT VE31 Data Error taff Pay Band must be valid taff Pay Band has been The code supplied will be mapped to the relevant common dimension therefore must exist in the mapping specified. CONTACT VE32 Data Error Primary Intervention must be valid Primary Intervention has been Can be Category or ubcategory code. CONTACT VE33 Data Error Other Intervention (1) must be valid Other Intervention (1) has been Can be Category or ubcategory code. CONTACT VE34 Data Error Other Intervention (2) must be Version 1.0 Page 20 of 22

21 valid Other Intervention (2) has been Can be Category or ubcategory code. CONTACT VE35 Data Error Other Intervention (3) must be valid Other Intervention (3) has been Can be Category or ubcategory code. CONTACT VE36 Data Error Other Intervention (4) must be valid Other Intervention (4) has been Can be Category or ubcategory code. Version 1.0 Page 21 of 22

22 9. Appendix A ending Locations NH Board Ayrshire & Arran Borders Dumfries & Galloway Fife Forth Valley Grampian Greater Glasgow and Clyde Highland Lanarkshire Lothian Orkney hetland Tayside Western Isles ending Location Code A B Y F V N G H L R Z T W Version 1.0 Page 22 of 22