Paper DM07. An Animated Guide: Comparing Files without Proc Compare Alejandro Jaramillo (Data Means Corp) and Russ Lavery contractor

Transcription

1 Paper DM07 An Animated Guide: Comparing Files without Proc Compare Alejandro Jaramillo (Data Means Corp) and Russ Lavery contractor ABSTRACT This paper provides an easy to understand SAS-based programmed method for comparing files and has several advantages over Proc Compare when the files are expected to be different This method produces a small volume of output and allows easy creation of reports to summarize and analyze the differences This automated analysis of differences helps resolve differences It should be noted that Proc compare is excellent if differences are small This paper discusses comparing files in the context of a system migration, or a change of vendors, where many (hundreds) files from an old vendor/system must be compared to files from a new vendor/system If the programmer is asked to compare hundreds of files and there are issues with the new files- the programming problem becomes embedded in a management problem Some effort must be spent tracking: file checked, files passed/rejected, status of rework of rejected files, communicating results and managing the timeline of the project This paper discusses code for doing the comparisons (it provides fully functioning sample code) and the project structure and task flow DETAIL LEVEL COMPARISON Proc compare is the method of choice for comparing files that are expected to have no, of few, errors If the comparison is expected to produce many failures to match, (thousands of observations fail for a variety of reasons), there is a need to post process information on failures and SAS is an excellent tool for doing this A logic for comparison is explained below Logic ---Left File--- Store Reg Prod LQ1 LQ2 LQ Merged File--- Store Reg Prod LQ1 LQ2 LQ3 1 RQ1 RQ2 RQ Match Match Data Checking Keys and Data gives exact answers key error Key ODD Data Right File--- Store Reg Prod RQ1 RQ2 RQ ODD Key Match lines on keys and use array & loop to compare data values This method checks EVERY value in the file Figure 1 Please look at Figure 1 We are going to be comparing two files (call the two files left and right) and will separate variables in the files into Key variables and data variables Store, region and prod (in red boxes) are key information and identify a unique row in the file These files have three numeric variables: Q1, Q2 and Q3 When we finally do a detailed comparison of the files we will merge the files on key information and use arrays to loop over data elements to check that the data associated matches We can have four states of nature and they are illustrated above Our eventual task is to report on all these

2 conditions: 1) keys and data match see black above stores 001 and 022 2) keys match but data elements differ see blue above-stores 044 (all data are bad) and 311 (Q3 data element is bad) 3) there was a data entry issue with the key and the observation can not be matched In our example, these were actually from the same store 956, but a 056 was keyed instead of a 956 4) a file has an observation that is not in the other file green above-stores 211 and 555 When switching from one vendor to another, the mechanisms the vendors use to collect data can differ and in that case a 100 percent match might not possible Sometime, management must decide on close enough %let Fuzz=00; /*how close is the match*/ data both badleft badright; merge Left (in=l rename=(q1=lq1 q2=lq2 q3=lq3)) right(in=r rename=(q1=rq1 q2=rq2 q3=rq3)); by store reg Prod; if L*R=1 then infile="both "; else if L=1 and R=0 then infile="left "; else if L=0 and R=1 then infile="right"; Array ALsales(*) Lq1-LQ3; Array ARSales(*) RQ1-RQ3; Array ADiff(*) Diff1-Diff3 (0 0 0) ; Left_vs_Right=L*10 +R*1; /*two measures of how much is at risk if obs are not "good"*/ if L then LSales=sum(of LQ1-LQ3); else LSales=0; if R then RSales=sum(of RQ1-RQ3); else RSales=0; if L*R EQ 1 then Total_Sales=sum(Lsales,Rsales)/2; else if L=1 and R=0 then Total_Sales=sum(LSales); else if L=0 and R=1 then Total_Sales=sum(RSales); SumDiff=0; Diff1=0; Diff2=0; Diff3=0;/*measures of size of data mismatch*/ if L*R NE 1 then KeyFlag="Bad ";/*this flags if key good/bad*/ Mismatch="N";/*Lower the data element mismatch flag*/ if L*R then /*if keys match, loop through an array to check the data elements*/ do; do i= 1 to 3;/*do loop-check data elements only if keys match*/ ADiff(i)=0; if ALsales(I) LT ARSales(i)*(1-&fuzz) /*if "close is OK, set fuzz to be GT 0*/ or ALsales(I) GT ARSales(i)*(1+&fuzz) then do; ADiff(i)=ALsales(i)-ARsales(I); if abs(adiff(i)) GT MaxDiff then MaxDiff=abs(Adiff(i));/*biggest difference */ SumDiff+abs(Adiff(i)); /*sum of differences */ mismatch="y"; /*raise the data element mismatch flag */ end; end; end; else mismatch="y"; Absolute value output both; if L=1 and R=0 then Output BadLeft; if L=0 and R=1 then Output BadRight; run; Absolute value use fuzz if exact match not achievable Rename data elements Infile is a flag for observation source If we had char & num data elements, we would need char & num arrays How important is obs? Use Avg if keys match cause data elements might not match /*if keys are bad, raise the data element mismatch flag*/ 2

3 The resulting file BOTH is shown below L e f T t o _ t v a M s l S K i M i _ L R _ u e s a S n D D D R S S S m y m x t P f i i i i a a a D F a D o r r L L L R R R i f f f g l l l i l t i r e o Q Q Q Q Q Q l f f f h e e e f a c f e g d e t s s s f g h i f Both N Both N Both Y Left Bad Y Left Bad Y Both Y Right Bad Y Right Bad Y The file both can be used for creating managerially useful reports as below: proc format ; value ObsFrom 1 =" 1 = obs in Right only" 10="10 Obs in Left only" 11="11 obs in both "; value fflag 0="Yes, Problems with key or data" 1="NO Problems with Key or data"; value $ YandN Y ="Yes: Problems with key or data" N ="NO problems with key or data"; run; Figure 2 shows that there were two observations that were only in the left file and that there were two observations that were only in the right file The keys not matching caused this situation This situation, keys failing to match, is worth some consideration If keys do not match, there is little more to be done at the detail level It is very difficult 3

4 for a QC person to fix, or even do much worthwhile investigation, of bad keys As we can see above, where a 9 in the store ID was coded as a 0 (the number next to it on the keyboard), if keys are miscoded and the data is sorted by keys, observations that should match can be anywhere in the resulting file If keys fail to match, the files likely have different sources (or the same source but the data was pulled from different time periods) Figure 3 shows that there were two observations that matched both keys and data and that there were two observations that matched on keys and had data issues In this example, the percent of bad observations was very high What if the data table were like this pretty good mismatch Left_vs_Right 1= Obs 10= Obs 11= Obs Total Frequency in Right in in both Percent Only LEFT only NO Problems Yes:Data Problems Total Management might want to know the importance of the six observations that were in the problem cells If they were a small enough percentage of total sales, management might accept the file mismatch Left_vs_Right 1= Obs 10= Obs 11= Obs Total Frequency in Right in in both Percent Only LEFT only NO problems with key or data Yes: Problems with key or data Total proc freq data =both; tables mismatch*left_vs_right; format mismatch $ YandN Left_vs_Right ObsFrom ; run; Store Reg Prod LQ1 LQ2 LQ3 R Q1 R Q2 R Q we are comparing data in one file to missing values - there is a data problem we are comparing data in one file to missing values there is a data problem obs in only one file obs in only one file Figure 2 Note that if an observation was classified as bad if 1) keys failed to match or if 2) ANY data of the data elements failed to match If there are many data elements on a row, classifying an observation as bad if one pair of data elements differs by just a small amount can be a harsh judgment As an example, Store 311 was classified as a mismatch because one data element failed to match It is possible this classification is too extreme Management often wants to know how important these bad data elements are in relation to the total sales in the file Management might want to consider more than the percent of bad elements Information about the importance of bad observations can be gotten form a Proc Means and from using weighted frequencies in a Proc Freq Code and output for Proc Freq and Proc Means are shown in Figure 4 The total sales in the whole data set is 2435 and only 2382 percent of the total is in observations that are perfect in keys and 4

5 data If the data sets have mismatches in keys and on data elements and those mismatches make up a small percent of the total sales (or whatever number the file contains) management may decide to accept the file This is the final piece of detail analysis covered in this paper mismatch Left_vs_Right 1= Obs 10= Obs 11= Obs Total Frequency in Right in in both Percent Only LEFT only NO problems with key or data Yes: Problems with key or data Total Ideally, all obs are be here Keys Match, issues w/data Figure 3 Store Reg Prod LQ1 LQ2 LQ3 RQ1 RQ2 RQ Match Match Data Key ODD Data ODD Key Frequency 1 = obs 10 Obs i 11 obs i Total Percent in Righ n Left o n both Col Pct t only nly NO problems with key or data Yes: Problems with key or data Total proc freq data=both; tables mismatch*left_vs_right / missing norow; format mismatch $ YandN Left_vs_Right ObsFrom; weight Total_Sales; proc means data=both n min mean max sum; var maxdiff sumdiff total_sales; total Max Sum Mis Store Reg Prod LQ1 LQ2 LQ3 R Q1 R Q2 R Q3 sales Diff Diff match N N Y 056 Variable N Min 12 Mean 10 8 Max Sum Y MaxDiff SumDiff Y 311 Total_Sales Y Y Y Figure 4 OVERVIEW AND PROJECT MANAGEMENT The SAS code for comparison of files is all one would need if there were only one file that needed to be compared However, if this is part of s system migration or of changing a to a new data supplier, many files might need to be compared In the case of a large project, the one bit of SAS code, shown above, needs to be put in context of it being part of a large project A large project would need an audit train of documents showing that files were proven good when accepted and providing actionable feedback to the vendor when they fail A large project would need inter-departmental co-ordination, reports, tracking of jobs completed, time estimates and to run the above code 5

6 (which can take time to program and to run), only when necessary Starting with the last item- running the code when necessary If the project is one where many files are being delivered that do not pass, labor can be saved by making the running of the code, shown above, the last step in a drill down process Vendors have been known to deliver the wrong file and it saves time if the QC team can just do a quick check and then do as much further work as the quality of the file warrants We first run Proc Contents on each file and paste the results into a word document If the number of obs in one file is ten times the number of obs in the other file, the keys and data comparison that we did above is not warranted If variables are missing or of the wrong type, the comparison of the files might stop at this point and the Proc Contents information sent to the vendor This output also helps decide what variables are to be used as key, what variables are to be numeric data elements and what variables are character data elements While our example only has numeric elements, in real life we need to loop over both numeric and character variables The Proc Summary information, as basic as it is, documents that the proper files were compared The run dates are important if the new file has been delivered many times Having the documentation for a file start with the Proc Contents information below puts to rest any question of the QC being done on improper files or version of the files Data Set Name WORKLEFT Observations 6 Created 01 Dec 2007 Sat 19:03:13 oclock Last Modified 01 Dec 2007 Sat 19:03:13 oclock File Name C:\DOCUME~1\user\LOCALS~1\Temp\SAS Temporary Files\_TD464\leftsas7bdat Alphabetic List of Variables and Attributes # Variable Type Len 3 Prod Char 8 4 Q1 Num 8 5 Q2 Num 8 6 Q3 Num 8 1 Store Char 8 2 reg Char 8 Figure 5 Data Set Name WORKRIGHT Observations 6 Created 01 Dec 2007 Sat 19:03:13 oclock Last Modified 01 Dec 2007 Sat 19:03:13 oclock File Name C:\DOCUME~1\user\LOCALS~1\Temp\SAS Temporary Alphabetic List of Variables and Attributes # Variable Type Len 3 Prod Char 8 4 Q1 Num 8 5 Q2 Num 8 6 Q3 Num 8 1 Store Char 8 2 reg Char 8 The creation of the Proc Summary output is only the first step in process of peeling back the onion on the data and stopping as soon as soon as we learn of major problems That process is shown in Figure 6 below The output from the Contents date and size step is shown in Figure 5, above Any data cleaning that need to be done can be done here This data cleaning has the object of fixing errors and making the files worth comparing 6

7 Process for QC Left File New or Right File Contents: date & size Check for duplicates Check for bad codes R P T Contents: date & size Check for duplicates Check for bad codes Freq - important vars Freq - important vars Clean the file Merge-Calc Diff by important Vars Rpt Clean the file Identify every row with problem Merge- Calc High Level Diffs Rpt electronic copy electronic copy Key Analysis Rpt Problem Analysis Row Problem Analysis sales electronic copy Figure 6 We suggest checking both files for duplicate observations Errors have been found in files that have been in use for years Sometimes the QC of a new data stream cleans up errors in old data streams Here we move to some basic principals of the peeling back of the onion Summary data is easier/faster to read than detail data If the data matches at the detail level, it will match at the summary level If the summaries are off, consider stopping It is NOT true that data that matches at the summary level will match at the detail level- ERRORS cancel If there are important variables, like dates, if is often useful to compute and compare counts by these important variables we have sees situations where both left and right had the same range of dates, but one file was missing several weeks- simply had no obs for a few weeks The totals for the files were close in number of obs and for total sales, but one file was bad A useful way to do this was to do Proc Summaries on both files and merge the summary files This provides a wealth of data at many different levels of summary Code, output, and is shown below proc summary data=left missing; class store reg prod; output out=lefthisummry n()= sum(q:)= /autoname; /*Left has the Sum in the name*/ types () store reg prod Store*reg*prod; proc summary data=right missing; class store reg prod; output out=righthisummry n()= sum(q:)=; /*Right has original vars*/ types () store reg prod Store*reg*prod; 7

8 *merge the file and do a comparison of the Sales info; data hiboth(keep=_type_ vartype FreqDiff store reg prod source salesdiff:) ; source=" "; /*wide enough for *Total* computed in Proc Report*/ merge LeftHiSummry(in=L rename=(_freq_=lfreq)) RightHiSummry(in=R rename=(_freq_=rfreq)); by _type_ store reg prod; if L*R then source="both "; Else if L then source="left " ; Else if R then source="right "; Array LeftSales(3) Q1_Sum Q2_Sum Q3_Sum; Array RightSales(3) Q1 Q2 Q3; Array flagsales(3) SalesFlag1-SalesFlag3; Array SalesDiff(3) SalesDiff1-SalesDiff3; array numvars(*) _numeric_; do i=1 to dim(numvars); if NumVars(i)= then numvars(i)=0; end; Zero fill in obs that are only from one file /* calculate differences tween Left and Calculate right*/ differences for freq and data values FreqDiff=LFreq-RFreq; do i=1 to 3; salesdiff(i)=leftsales(i) - RightSales(i); Knowing the order of subtraction is helpful end; vartype="sales (Left-Right)"; Output; run; options linesize=100 ps=50; proc report data=hiboth spacing=0 missing nowd; columns _type_ store reg prod source vartype FreqDiff salesdiff:; define _type_ /order width=4 "Type"; define store /order width=6 spacing=1; define reg /display width=7 spacing=1 "Region"; define prod /display width=8 spacing=1 "Product/Code"; define source /display width=7 spacing=1 "data/source"; Define vartype /display spacing=1 width=21 ; Define FreqDiff /sum width=5 spacing=1 "Freq/Diff"; Define SalesDiff1 /sum width=5 spacing=4 "Q1 /Diff"; Define SalesDiff2 /sum width=5 spacing=1 "Q2 /Diff"; Define SalesDiff3 /sum width=5 spacing=1 "Q3 /Diff"; Break after _type_ / suppress summarize ol ul skip; compute after _type_ ; source="*total*"; endcomp; run; 8

9 Product data Freq Q1 Q2 Q3 Type Store Region Code Source vartype Diff Diff Diff Diff 0 Both Sales (Left-Right) *Total* Both Sales (Left-Right) Right Sales (Left-Right) Left Sales (Left-Right) Both Sales (Left-Right) *Total* Right Sales (Left-Right) Both Sales (Left-Right) Both Sales (Left-Right) *Total* Both Sales (Left-Right) Both Sales (Left-Right) Both Sales (Left-Right) Left Sales (Left-Right) Left Sales (Left-Right) Both Sales (Left-Right) Right Sales (Left-Right) Right Sales (Left-Right) *Total* Both Sales (Left-Right) Both Sales (Left-Right) Both Sales (Left-Right) Left Sales (Left-Right) Left Sales (Left-Right) Both Sales (Left-Right) Right Sales (Left-Right) Right Sales (Left-Right) *Total* Figure 7 The red box sows the danger of just checking summary totals for the files Positive and negative errors will cancel out and make the overall numbers look good This canceling happens in real life and not just in made up exercises Totals can look very good and sub-totals or details can look bad The output shown in Figure 7 becomes part of the documentation for the project The QC documentation for a file might be: Proc Summary output - shown in Figure 5 High-level comparisons shown in Figure 7 Detail Level comparison summaries shown in Figures 3 and 4 9

10 QC PROCESS FLOWCHART AND TIMELINES The final section is a short discussion of the time line for QC-ing a file Management sometimes does not understand the process and has unreasonable expectations for timelines The fact is that QC-ing a file is not a time consuming process, in any case Often a programmer can program ahead and run multiple sessions of SAS and be QC-ing several files at the same time If QC- programs are run interactively, output from one section can be assembled into a report while the next section of code is running This quasi-quick process IF THE FILES ARE GOOD If the files are bad, the QC-ing step is still quick, but the approval of the file process becomes much slower as the process starts to loop back on itself If the files are identical, the Green box meeting is very quick and the orange box can be eliminated If the files are close or bad, the meeting can take quite a while If the files are judged to be close the pink box must happen The comparison of standard reports to see if the differences between the files is visible to report users, can take a while Finally, if the files are bad, the green meeting can take quite a long time When the files are bad, the reports shown in this paper have been able to provide good initial direction as the cause of the problem However, when the files are bad, the tasks in the pink box must happen and these tasks can take quite a while Finding discrepancies is an easy task, fixing the discrepancies can be very difficult and requires commitment Timeline Write programs, for series of files, in anticipation of file delivery A batch of files, to be compared, is delivered The QC Process Run QC Programs on the batch of files (using multiple SAS sessions if required/possible) Assemble report on the batch of files (often concurrent w/ run) QC Programming Reviews and annotates report (1 day) Arrange meeting with group producing the series of files (1 week) Discuss report w/ group producing files and create action items (1 day) File is OK or Close Figure 8 File(s) FAIL Investigate / fix action items (1 week) F A I L If files are close, users of files run old reports with new files and see if results look odd (1 week) Create new version of files (2 weeks) File Passes log file as Done CONCLUSION Proc compare provides a quick and easy way to compare files that are well known and expected to have small differences or not differences at all When data sources and process change use of a systematic approach as the outlined in this paper to compare data at the top and record level provides an efficient mechanism to track progress, identify and resolve potential problems CONTACT INFORMATION Your comments and questions are valued and encouraged Contact the authors at: Alejandro Jaramillo, Data Means Corp, ajaramillo@datameanscom, wwwdatameanscom Russ Lavery, Independent Consultant for Numeric Resources, RussLavery@verizonnet SAS and all other SAS Institute Inc product or service names are registered trademarks or trademarks of SAS Institute Inc in the USA and other countries indicates USA registration Other brand and product names are trademarks of their respective companies 10