Data Integration Best Practices

Transcription

1 (Healthy Habits for SAS Data Integration Studio Users)

2 Abstract: Version 9 of the SAS System offers tools to help developers and business users manage and organise the wealth of data and processes that face SAS professionals today. SAS Data Integration Studio benefits from many features that support healthy habits for data integration, but they can only 'be of use' if they are 'being used'. DI Studio allows customisation of the custom tree, error monitoring, job status handling, data validation, conformed data model support, selfdocumentation, and role assignment. Identification of the benefits behind using these functions is often enough to motivate users into controlled and organised methods of working. This paper describes examples of best practice for developing data integration suites to ensure quality, efficiency and resilience is built into the heart of your enterprises information estate.

3 Subjects: Data Integration Structure Data Integration Organisation Capture Control (CCT Tables) Error Monitoring Data Validation Data Protection (Scrambler) Conformed Modelling SQL Optimisation Self Documentation Role Assignment Rename Standard Transforms SAS DI Studio Version 3.4 under SAS Intelligence Platform 9.1.3

4 Data Integration Structure Challenge: How can you best deliver Business Intelligence from a variety of source systems across a diverse consumer base? Solution: Employ a Data Integration flow structure.

5 Data Integration Structure Challenge: How can you best deliver Business Intelligence from a variety of source systems across a diverse consumer base? Solution: Employ a Data Integration flow structure. Source Systems

6 Data Integration Structure Challenge: How can you best deliver Business Intelligence from a variety of source systems across a diverse consumer base? Solution: Employ a Data Integration flow structure. Source Systems Detailed Data Model

7 Data Integration Structure Challenge: How can you best deliver Business Intelligence from a variety of source systems across a diverse consumer base? Solution: Employ a Data Integration flow structure. Source Systems Detailed Data Model Subject Specific Data Marts

8 Data Integration Structure Challenge: How can you best deliver Business Intelligence from a variety of source systems across a diverse consumer base? Solution: Employ a Data Integration flow structure. Source Systems Detailed Data Model Subject Specific Data Marts Subject Specific Business Intelligence

9 Subjects: Data Integration Structure Data Integration Organisation Capture Control (CCT Tables) Error Monitoring Data Validation Data Protection (Scrambler) Conformed Modelling SQL Optimisation Self Documentation Role Assignment Rename Standard Transforms SAS DI Studio Version 3.4 under SAS Intelligence Platform 9.1.3

10 Data Integration Organisation Challenge: How can you keep track of the thousands of jobs typically created in a data integration suite? Solution: Utilise the custom tree in SAS Data Integration Studio.

11 Data Integration Organisation Create folders for each integration layer. Sub divide them by: Jobs Libraries Tables Number the folders preserve order. Stick to methodology: (e.g. don t transform in capture layer)

12 Subjects: Data Integration Structure Data Integration Organisation Capture Control (CCT Tables) Error Monitoring Data Validation Data Protection (Scrambler) Conformed Modelling SQL Optimisation Self Documentation Role Assignment Rename Standard Transforms SAS DI Studio Version 3.4 under SAS Intelligence Platform 9.1.3

13 Capture Control Challenge: How can I perform incremental extracts from several source systems? Solution: Define Capture Control Tables for each source table. Status To ensure smooth running of DI suite. (Started, Failed, or Success) From/To Datetimes To extract against the last updated column in the database. Also useful to determine processing times as data increases day by day.

14 Capture Control Send Job Status to dataset with same name as the job.

15 Capture Control Only extract records which have updated since last run. Capture Job Source Systems Conformed Model

16 Capture Control Only extract records which have updated since last run. Capture Job Source Systems Conformed Model CoreInfo Tables

17 Capture Control Only extract records which have updated since last run. Pre Capture Job Post Source Systems Conformed Model CoreInfo Tables

18 Capture Control Pre-Processing Is this the first time the job has run successfully today? Yes No Warn that duplicate facts will occur. Did the previous run fail, or not finish? Yes No Warn that this is a replacement run. Update dates in CCT table for this source. (&source_table._cct)

19 Capture Control Post-Processing Did the job run successfully? Yes No Update CCT table with Status= Failed. Update dates in CCT table for this source. (&source_table._cct)

20 Subjects: Data Integration Structure Data Integration Organisation Capture Control (CCT Tables) Error Monitoring Data Validation Data Protection (Scrambler) Conformed Modelling SQL Optimisation Self Documentation Role Assignment Rename Standard Transforms SAS DI Studio Version 3.4 under SAS Intelligence Platform 9.1.3

21 Error Monitoring Challenge: How can I keep my production support department informed of job failures/successes? Solution: job statistics to designated mailbox. Create User Transform called _Stats. Add _Stats transform to each job.

22 Error Monitoring Add _Stats transform to Job.

23 Error Monitoring Drag Target table to one input. Drag _Stats to other input. ( _Stats table contains addresses of recipients). Don t hard-code addresses. What happens when people leave? Different recipients for dev/prod.

24 Error Monitoring _Stats transform properties. Only s if job has failed.

25 Error Monitoring Last job in flow always sends to Admin & Support. Set Last Job to Yes.

26 Subjects: Data Integration Structure Data Integration Organisation Capture Control (CCT Tables) Error Monitoring Data Validation Data Protection (Scrambler) Conformed Modelling SQL Optimisation Self Documentation Role Assignment Rename Standard Transforms SAS DI Studio Version 3.4 under SAS Intelligence Platform 9.1.3

27 Data Validation Challenge: How can I ensure only clean data gets loaded into the warehouse? Solution: Use the Data Validation transformation.

28 Data Validation Challenge: How can I ensure only clean data gets loaded into the warehouse? Solution: Use the Data Validation transformation. Use the standard Invalid, Missing, Duplicate tabs. Employ custom validation and apply a severity rating: 1 = Exclusion 2 = Correction 3 = Improvement Store exceptions in permanent dataset for further analysis.

29 Data Validation e.g. Check for Truncation of Key columns

30 Data Validation 1) Create each condition

31 Data Validation 1) Create each condition 2) Determine validation

32 Data Validation 1) Create each condition 2) Determine validation 3) Define corrective action if required

33 Data Validation 1) Create each condition 2) Determine validation 3) Define corrective action if required 4) This gets written to temp dataset ETLS_EXCEPTIONS.

34 Data Validation 1) Create each condition 2) Determine validation 3) Define corrective action if required 4) This gets written to temp dataset ETLS_EXCEPTIONS. 5) Run %Append_Data_Quality Macro in post-processing.

35 Data Validation 1) Create each condition 2) Determine validation 3) Define corrective action if required 4) This gets written to temp dataset ETLS_EXCEPTIONS. 5) Run %Append_Data_Quality Macro in post-processing. 6) Use BI tools to investigate Data Quality issues (e.g. Particular source system requires cleansing)

36 Data Validation %Append_Data_Quality Macro Logic. Does ETLS_EXCEPTIONS exist? Yes No Halt macro as no errors to process. Append exceptions to permanent table DQ_Error_Event.

37 Data Validation Table Properties for DQ_ERROR_EVENT. Column name Description Type Length Row_Extraction_Date Date-timestamp when the row was exported or extracted from the source system. Num (8) Exception_Event_Date Date-timestamp when the exception was identified by the data warehouse processes. Num (8) Job_Name The name of the ETL job which identified the exception. Char (64) Table_Name The library and table name which contains the row and column containing the exception. Char (41) Row_Number The row number containing the exception. Num (8) Column_Name The column name containing the datum of the exception. Char (32) Screen_Description The screen (data quality test) description. Char (256) Exception_Description Standardised description of the exception. Char (256) Exception_Action Automated data conform action (if any). Char (256) Exception_Severity The severity level of the DQ Error Event (1=Exclusion, 2=Correction, 3=Improvement ). Num (8) Unconformed_ValueN Original value (numeric) before conforming. Num (8) Conformed_ValueN Conformed (numeric) value. Num (8) Unconformed_ValueC Original value (character) before conforming. Char (256) Conformed_ValueC Conformed (character) value. Char (256)

38 Subjects: Data Integration Structure Data Integration Organisation Capture Control (CCT Tables) Error Monitoring Data Validation Data Protection (Scrambler) Conformed Modelling SQL Optimisation Self Documentation Role Assignment Rename Standard Transforms SAS DI Studio Version 3.4 under SAS Intelligence Platform 9.1.3

39 Data Scrambling Challenge: How can I ensure I m not holding sensitive production data on development/test systems. Solution: Use Data Scrambling routines in non-production environments. Often development source systems are created using production data, and warehouses can propagate the risk of breaching the data protection act.

40 Data Scrambling Custom Transform The %data_scrambler macro allows for columns to be scrambled or passed through normally.

41 Data Scrambling Custom transform Edit Paramters: Select Pass don t scramble key fields! Scramble method: Ranuni Function MD5 Function Translate Function

42 Data Scrambling What about Production? %let liveenvironment = PROD; %let thisenvironment= %sysfunc(substr(%sysfunc(upcase(%sysfunc(getoption(metaserver)))),1,4); Don t perform scramble routine if thisenvironment = liveenvironment. When runnning in Dev the METASERVER option should be different. Could set up a table with environment value in.

43 Subjects: Data Integration Structure Data Integration Organisation Capture Control (CCT Tables) Error Monitoring Data Validation Data Protection (Scrambler) Conformed Modelling SQL Optimisation Self Documentation Role Assignment Rename Standard Transforms SAS DI Studio Version 3.4 under SAS Intelligence Platform 9.1.3

44 Conformed Model Challenge: How can I track trends in my data when the source systems don t hold history. Solution: Use a conformed data model in a warehouse, using slowly changing dimensions where appropriate. Re-Useable Dimensions Fact Tables

45 Conformed Model In the Integrate layer use the SCD Type II Loader transform to make use of effective date processing.

46 Conformed Model In the Integrate Layer use the Surrogate Key Generator to determine keys for dimension tables.

47 Subjects: Data Integration Structure Data Integration Organisation Capture Control (CCT Tables) Error Monitoring Data Validation Data Protection (Scrambler) Conformed Modelling SQL Optimisation Self Documentation Role Assignment Rename Standard Transforms SAS DI Studio Version 3.4 under SAS Intelligence Platform 9.1.3

48 SQL Optimisation Challenge: How can I ensure the best possible SQL performance is achieved through my SQL Join transform. Solution: Use the undocumented _Method option on the SQL procedure to determine processing.

49 SQL Optimisation: _Method Option (SAS Note 33604)

50 Subjects: Data Integration Structure Data Integration Organisation Capture Control (CCT Tables) Error Monitoring Data Validation Data Protection (Scrambler) Conformed Modelling SQL Optimisation Self Documentation Role Assignment Rename Standard Transforms SAS DI Studio Version 3.4 under SAS Intelligence Platform 9.1.3

51 Self Documentation Challenge: How can I ensure the executed warehouse code is documented to an acceptable standard? Solution: DI Studio self documents the code, based on descriptions in in the job and transform properties.

52 Self Documentation Meaningful Job names Descriptions of why not just what.

53 Self Documentation Use Notes and Document Attachments.

54 Self Documentation Descriptions & Notes are propagated through to the executable code, benefitting production support teams.

55 Subjects: Data Integration Structure Data Integration Organisation Capture Control (CCT Tables) Error Monitoring Data Validation Data Protection (Scrambler) Conformed Modelling SQL Optimisation Self Documentation Role Assignment Rename Standard Transforms SAS DI Studio Version 3.4 under SAS Intelligence Platform 9.1.3

56 Role Assignment Challenge: How can I address who is responsible for which job / entity? Solution: Use Role Assignment in DI studio.

57 Role Assignment Allocate names and roles where required.

58 Subjects: Data Integration Structure Data Integration Organisation Capture Control (CCT Tables) Error Monitoring Data Validation Data Protection (Scrambler) Conformed Modelling SQL Optimisation Self Documentation Role Assignment Rename Standard Transforms SAS DI Studio Version 3.4 under SAS Intelligence Platform 9.1.3

59 Rename Standard Transforms Challenge: How can I keep track of processing in a job which has a lot of transformations. Solution: Don t use the default transform names, but rename the default to something meaningful. E.g. Rename SQL Join to Merge Agent_Dim with Broker_Dim

60 Subjects: Data Integration Structure Data Integration Organisation Capture Control (CCT Tables) Error Monitoring Data Validation Data Protection (Scrambler) Conformed Modelling SQL Optimisation Self Documentation Role Assignment Rename Standard Transforms SAS DI Studio Version 3.4 under SAS Intelligence Platform 9.1.3

61 Contributors Mick Collington Jethro Day Steve Morton Nick Treadgold

62 Contributors Mick Collington Jethro Day Steve Morton Nick Treadgold Data Integration Developer Group (SAS Professionals) Julien Heijster John Robertson forum/topics/data-integration-best

63 Contributors Mick Collington Jethro Day Steve Morton Nick Treadgold Data Integration Developer Group (SAS Professionals) Julien Heijster John Robertson forum/topics/data-integration-best SAS.COM