Self-documenting Data Processes with SAS

Transcription

1 Queensland Users Exploring SAS Technology Self-documenting Data Processes with SAS 26 Sept 2013 Dinu Corbu Suncorp - Credit Risk Reporting

2 The Problem SAS Data Processes of High Complexity: The Details How tracking: The Solution 1. Understanding 2. Controlling 3. Documenting 1. Flows of data 2. Changes of variables (values/types) 3. New variables creation 4. Delivery of results Self-documenting process: 1. Each operation/step at run time 2. Customised logs

3 Case Study 1: Self documenting a Data Cleaning - Standardising Process Used for projects employing identity resolution techniques: 1. Master data management (MDM) 2. Statistical Integration (Records Linkage )

4 Steps: 1. Cleaning/standardising: Surname Sir Brown- Smith Jnr Maiden Name Air Vice Marshal First Name Middle Name Nickname DoB Douglas Alexand er (Doug) 20/05/1916 Brown Smith Douglas Alexander Doug 20/05/ Blocking: - To restrict comparison numbers 3. Matching: Brown Smith Douglas Alexander Doug 20/05/ Brown Smiths Doug Alexander 20/05/ Classifying: - matches, non-matches, possible matches

5 The self-documenting method: - Uses SAS - Records all cleaning/standardising operations - At both record and dataset levels At record level - oper_codes : operations (filters) applied e.g.: C3-C2-C1-I1-G1-A1-E1 - updated_codes : what operations changed the record e.g.: C3-C1-I1-G1-E1 At dataset level - operation code and description - numbers of records filtered - number of records manually inspected/reviewed - number of records transformed (updated) - date of the operation

6 An interactive three-steps cycle: Step 1 - Automatically filtering the data Step 2 - Manually Reviewing /inspecting/modifying the changes (user may update the filters and re-run Step 1) Step 3 - Validating all changes and self-documenting the operation Each operation runs through a customised SAS code e.g.: This generally contains: - a macro for running Step 1 - instructions for interactively running Step 2 - a macro for running Step 3

7 Step 1: - imports a list of tokens - filters the records based on tokens - creates a temporary dataset of proposed changes Automatically run by a macro, e.g: Imports list of tokens and/or lookup tables, e.g Filters the records against tokens Retains the subset matching the filters for manual reviewing

8 Step 2 - Interactively in Edit Mode Needed because: a. some situations require human decision b. new tokens for filters may appear in data Example: The token NEE which may: - be part of a real surname such as MC NEE or - indicate a maiden name such as SMITH NEE JOHNES Options: - update the look-up table and run the Step 1 again - eliminate records that do not need transforms - do more transforms in Edit Mode

9 Step 3: Validating 1. Updates the permanent dataset with all the changes 2. Concatenates updated_codes (self-documenting at row level) 3. Updates the general log (self-documenting at dataset level) Example: Macro to validate surnames The sequence that validates the cleaning

10 The general log: 1. Self-documented process at a glance (dataset level) 2. Short description of the operations 3. Principal counts and operation date For each data cleaning operation a row is appended to the general log E.g.: incrementing the log E.g.: reviewing counts and date

11 At observation level 1. A cross-tabulation of update_codes and oper_codes gives an image of the data quality 2. Tracking data quality changes from one period to another and prioritising data issues 3. More information can be obtained by sub-setting the data by operation codes E.g.: looking for correlations between types of data quality issues:

12 Case Study 2: Data Lineage for a Regular Data Process Conditions: 1. Multiple Data Sources from disparate systems 2. Big Number of Variables and Transforms 3. Hard-coded, complicated computing solutions Challenges: 1. No metadata, no flow diagram 2. Only a few comments in SAS codes 3. Old codes relying on the author s specific knowledge 4. Task assigned to new employees How to approach this?

13 Possible approaches (among many): A. Based on Business - Rejected 1. Learning the business rules 2. Finding extra information about data 3. Reading/understanding the codes B. Based on Codes - Rejected 1. Reading and running the codes 2. Reading the logs 3. Trying to understand C. Based on Self-documenting - Accepted 1. Inserting self-documenting sequences the codes 2. Filtering/analysing the customised logs

14 How this works Building three types of self-documenting macros, for: 1. importing data into SAS 2. transforming SAS datasets 3. exporting data outside SAS Self-documenting the process 1. Identifying and numbering the steps 2. Inserting self-documenting macro for each step 3. Running the process Using the customised logs 1. They are SAS datasets built at dataset and variable level 2. Contain the information about each step 3. Can be filtered, sorted, interrogated

15 Usage examples: Importing data Transforming data 1. one or more inputs/outputs 2. a relation table for transformed/newly created variables Exporting data

16 The logs at a glance At dataset level May have one or more inputs/outputs per step At variable level 1. An observation for each I/O variable 2. The field comes_from shows the transforms details

17 Interrogating the logs Examples of queries a. To see all datasets whose names include the string riskpoint_ b. To analyse the variables containing child_ and/or parent_ from the above dataset Using queries results 1. Describing/documenting/optimizing/controlling the process 2. Comparing a month against another month 3. Eliminating redundancies such as unused variables

18 Thank you. Questions and Suggestions