Common Sense Validation Using SAS

Transcription

1 Common Sense Validation Using SAS Lisa Eckler Lisa Eckler Consulting Inc. TASS Interfaces, December 2015

2 Holistic approach Allocate most effort to what s most important Avoid or automate repetitive tasks Ask ourselves the right questions

3 Defining terms: QA Data quality assurance is the process of profiling the data to discover inconsistencies, and other anomalies in the data and performing data cleansing activities to improve the data quality. Wikipedia

4 Defining terms: Verification Verification is the act of reviewing, inspecting, testing, etc. to establish and document that a product, service, or system meets the regulatory, standard, or specification requirements. à Does it meet the structural requirements? à Is it complete?

5 Defining terms: Validation Validation refers to meeting the needs of the intended end-user or customer. à Does it answer the user s question? à Does it meet all of the needs? à Structure and completeness, data integrity, appropriateness

6 Computers are useless. They can only give you answers. Pablo Picasso

7 How do I know if I got it right?

8 Is Validation a programming task? à Yes mostly à The routine parts can and should be automated and repeatable à That leaves more resources for the parts which require human attention

9 q PROC COMPARE q PROC CONTENTS q PROC CONTENTS with compare (using PROC COMPARE or TRANSPOSE, MERGE and flag) q PROC FREQ +/- PROC FORMAT q PROC SUMMARY Wrap a macro around this and you have a flexible, re- usable tool! q PROC SUMMARY + compare (using PROC COMPARE or TRANSPOSE, MERGE and flag)

10 Does this mean writing more SAS code after I thought I was finished writing SAS code? Yes and no We can save time and improve the quality of results by using code that isn t part of the final program. Don t think of it as disposable, though: this code can be set up once and saved to use for all future validation efforts. Additional benefits Automated validation provides a log Easily repeatable

11 What are the questions? Should this be a replication of something I have seen before? If not, is it similar to something I ve done before? Is it or some part of it supposed to be different from anything I ve seen before? Is the result packaged properly?

12 Mantra for Validation Check your assumptions Confirm similarities Focus on differences

13 How is this result expected to compare with what we ve seen before? Entirely different Some overlap Complete overlap Subset

14 Some possibilities not an exhaustive list!

15 ** This is the simplest form of **; ** comparison between two sets of data **; proc compare compare = SHOES run; base = OLD_SHOES;

16

17 ** PROC CONTENTS gives us metadata **; proc contents run; data = OLD_SHOES;

18

19 ** CONTENTS with select facts saved to **; ** a data set --> a table of metadata **; proc contents data = OLD_SHOES run; out = CONTENTS_OLD_SHOES (keep=name type length);

20 ** Same as previous slide except for the **; ** new data set **; proc contents data = NEW_SHOES run; out = CONTENTS_NEW_SHOES (keep=name type length);

21 ** Comparing metadata tables rather than **; ** data tables **; proc compare compare = CONTENTS_OLD_SHOES run; base = CONTENTS_NEW_SHOES;

22

23

24 proc contents data = OLD_SHOES run; out = CONTENTS1(keep=name type length); proc contents data = NEW_SHOES out = CONTENTS2(keep=name type length); run; proc compare run; compare = CONTENTS1 base = CONTENTS2;

25 %macro COMPARE_STRUCTURE1; proc contents data = OLD_SHOES run; out = CONTENTS1(keep=name type length); proc contents run; data = NEW_SHOES out = CONTENTS2(keep=name type length); proc compare compare = CONTENTS1 base = CONTENTS2; run; %mend COMPARE_STRUCTURE1; %COMPARE_STRUCTURE1;

26 %macro COMPARE_STRUCTURE(DS1,DS2); proc contents run; data = &DS1 proc contents run; proc compare run; out = CONTENTS1(keep=name type length); data = &DS2 out = CONTENTS2(keep=name type length); compare = CONTENTS1 base = CONTENTS2; %mend COMPARE_STRUCTURE; %COMPARE_STRUCTURE(OLD_SHOES, NEW_SHOES);

27 %macro COMPARE_STRUCTURE(DS1,DS2); proc contents run; data = &DS1 proc contents run; proc compare run; out = CONTENTS1(keep=name type length); data = &DS2 out = CONTENTS2(keep=name type length); compare = CONTENTS1 base = CONTENTS2; %mend COMPARE_STRUCTURE; %COMPARE_STRUCTURE(OLD_SHOES, NEW_SHOES);

28 ** We've just built a generic tool for comparing **; ** the STRUCTURE of any two SAS data sets **; %COMPARE_STRUCTURE( <any SAS data set name>, <any other SAS data set name> );

29 Reasonableness: complete overlap

30 Reasonableness: complete overlap _character_ gives the list of ALL vars in the table with data type character, which may include some vars with too many values

31 Reasonableness: complete overlap This code also gives a list of ALL vars in the table with data type character

32 Reasonableness: complete overlap The above code lets us customize our list to exclude non-categorical character columns and include the others

33 Reasonableness: complete overlap

34 Reasonableness: complete overlap Similar to the way we compared the structure of two tables, we can compare the frequency counts of values in two tables

35 Data correctness: complete overlap proc compare run; compare = OLD_SHOES base = NEW_SHOES; à Judicious use of unrestricted PROC COMPARE -- after confirming reasonableness

36 If we are expecting a result that is a complete replication of something that already exists Confirm that the structure is identical Confirm that the data is the same at a high level Confirm that the data is the same at a detailed level à fully automated

37 Data correctness: completely new What if we don t have an existing results table to compare to? Similar SAS data in an existing table or produced by someone else? Similar data in some other format that can be imported into SAS for comparison? Do we have a data requirements document? The truly original data will require much greater attention to validation and the involvement of a subject matter expert

38 Packaging: completely new

39 Packaging: completely new Assuming we have a Requirements document Import REQUIREMENTS into SAS data set run PROC CONTENTS on new data set to get CONTENTS_NEW_SHOES run PROC COMPARE, comparing CONTENTS_NEW_SHOES to REQUIREMENTS OR Join REQUIREMENTS with CONTENTS_NEW_SHOES and flag non-matching rows

40 Packaging: completely new

41 Packaging: completely new

42 Reasonableness: completely new

43 Reasonableness: completely new

44

45 What if part of our result should be the same as an existing result but there should be some differences? Treat it as a hybrid and split the validation exercise into two parts Expected same (by rows, columns, data or metadata) Expected different (by rows, columns, data or metadata)

46 For each of the two parts Confirm (expected) similarities Focus efforts on (expected) differences Run the validation procedures we ve alreay looked at as appropriate for the same and different aspects

47 Recall the scenario where our data sets should be identical record _id a b c 1 * * * 2 * * * 3 * * * record _id a b c 1 * * * 2 * * * 3 * * *

48 When some columns should be the same record _id a b c 1 * * 2 * * 3 * * record _id a b d 1 * * 2 * * 3 * * 48

49 When some cells (parts of rows and columns) should be the same record _id a b c 1 * * 2 * * 3 * * record _id a b d 1 * * 3 * * 4

50 Review:

51 Summary: q Ask the right questions q Confirm similarities with known things quickly and programmatically then focus time and effort on validating unknown or new things q Basic base SAS procedures for validation q vary the technique based on how much is similar/different from what you ve validated previously and what types of data are involved

52 You can find my related conference papers at Don t Forget About Small Data (SESUG 2015) When Good Looks Aren t Enough (NESUG 2009) If you have comments or questions lisa.eckler@sympatico.ca