Cleaning Duplicate Observations on a Chessboard of Missing Values Mayrita Vitvitska, ClinOps, LLC, San Francisco, CA

Transcription

1 Cleaning Duplicate Observations on a Chessboard of Missing Values Mayrita Vitvitska, ClinOps, LLC, San Francisco, CA ABSTRACT Removing duplicate observations from a data set is not as easy as it might sound if the data set has missing values. For example, it is difficult to choose which observation to keep if the duplicate observations have certain values missing in one observation, but present in others (missing values resembling a chessboard pattern.) Important information could be lost or redundant observations could affect your analysis if the user applies such well-known methods as NODUPKEY option in PROC SORT or DISTINCT keyword in PROC SQL. This paper presents a macro solution that populates missing values based on user s judgment and cleans duplicates from the data set. INTRODUCTION Before any analysis of data or during data processing it is important to investigate the integrity of the data. Two of the most common data issues are duplicate observations and missing values. An example of a data set with mixed missing and duplicate values is shown in Figure 1. Duplicate observations in this example could be removed by PROC SORT, PROC SQL, PROC FREQ or data step manipulations. Figure1: Original data set with duplicate observations (left panel) and Updated data set after removing duplicates (right panel). C 1 and C 2 in the first two columns represent the values of two character variables. N 1,, N k are the values of numeric variables. 1 C 2 N k C 2 N k 2 C 1 N 1 C 1 N 1 3 C 2 N k Figure 2 shows an example of a data set with a linked observation that connects several non-duplicate rows together producing a group of duplicate observations. Formally, rows 1, 2 and 3 are all unique but rows 2 and 3 are connected by the same value of the first variable, and rows 1 and 3 are connected by the same value of the second and by the last variables. Figure 2: Original data set with a group of linked non-duplicate observations by the first character variable (left panel) and Updated data set after removing duplicates (right panel). 1 C 2 N k 2 C 1 N 1 C 1 C 2 N 1 N k 3 C 1 C 2 N k It is desirable to collect all information within the group, collapse linked observations into one and delete all redundant observations. A straightforward use of PROC SORT with the NODUP or NODUPKEY options, or PROC SQL with DISTINCT and GROUP BY or PROC FREQ will either keep redundant or delete valuable information from the data set. Multiple data set manipulations could provide a solution but it is complicated. This paper presents a three-step approach to the problem of linked duplicate observations. The first step is to separate the unique observations from the groups of linked non-duplicate observations into two data sets. The second step is to use PROC SUMMARY to populate missing values within the groups, and remove duplicate observations. The last step is to set two datasets back (see Figure 3). 1

2 Figure3: A three-step approach to removing redundant observations. See text for more details 1 C 2 N k 1 C 1 C 2 N 1 N k 2 C 1 N 1 2 C 1 C 2 N 1 N k 1 C 1 C 2 N 1 N k 3 C 1 C 2 N k 3 C 1 C 2 N 1 N k 4 C 4 C 3 K k 4 C 4 C 3 K k 5 C 4 C 5 K k 5 C 4 C 5 K k 4 C 4 C 3 K k 5 C 4 C 5 K k STEP ONE: SEPARATE THE SHEEP FROM THE GOATS DEFINING GROUPS In order to meaningfully define grouped observations, the variables need to identify potential duplicates as uniquely as possible. For example, Social Security Number and ID are very strong candidates for this task whereas Gender and Age should be avoided. STEP ONE First, we need to split the original data set into two. _MY_GOAT data set will contain observations where at least one variable of interest has more than one unique non-missing value (See Figure 3 rows 4 and 5, second column). _MY_SHEEP data set will have observations with either missing or a single non-missing value for each variable (See Figure 3 rows 1, 2 and 3). PROC SQL is used to create a data set that has the original and temporary variables. The values of the temporary variables are counts of unique non-missing values in each GROUP BY group of the variables that define a group of linked observations. SELECT statement lists all variables(*) from the original data set (&DSIN.) and the temporary variables (%LISTCOUNT(&vars_list.)). %LISTCOUNT() macro generates a text string of consecutive words: count( distinct, variable name, ) as _0_, variable name, and, for all variables defined in &var_list. For example, a code %LISTCOUNT(var1 var2) will be resolved to count(distinct var1) as _0_var1, count(distinct var2) as _0_var2. %LISTCOMMA() macro produces a list of comma separated variables specified by &BY macro variable (See Appendix A). create table _temp as select *, %listcount(&vars_list.) from &dsin. group by %listcomma(&by.); Next, a data step is used to break the original data set into _MY_SHEEP and _MY_GOAT data sets by using these temporary variables: data _MY_GOAT(drop=_0_:) _MY_SHEEP(drop=_0_:); set _temp; if %count_ge_2(&vars_list.) then output _MY_GOAT ; else output _MY_SHEEP; %COUNT_GE_2(&vars_list.) macro resolves to a piece of code that checks if at least one of the temporary variable values is greater or equal to 2 (see Appendix A). Finally, all temporary variables are dropped from these datasets (drop=_0_:). STEP TWO: SHEARING THE SHEEP PROC SUMMARY In _MY_SHEEP data set any information from linked observations will be used to populate missing values within the groups and redundant observations will be removed. Since any numeric variables in _MY_SHEEP data set contain 2

3 either missing or single unique non-missing values, PROC SUMMARY with MAX statistic will output the non-missing values. VAR statement lists all numeric variables that the user is interested in (&col_name_num.). NWAY option suppresses output of marginal MAX values for the numeric variables. All character variables that define the group of linked observations are listed in CLASS statement (&by.). Any other character variables that need to be included into the output data set are defined in ID statement (&col_name_char.). MISSING option is not used in PROC SUMMARY because we are not treating missing values of CLASS variables as a unique group. Two automatically generated variables _TYPE_ and _FREQ_ are dropped from the output dataset. proc summary data=_my_sheep nway; class &by.; id &col_name_char.; var &col_name_num.; output out=_my_sheep_shr(drop=_type FREQ_) max()=; _MY_SHEEP_SHR output data set contains the unique observations with as few as possible missing values (See Figure 3 middle panel). THE GOATS The _MY_GOAT data set contains observations with more than one unique value for at least one variable. These observations could be unique or they could indicate potential data problems such as misspelling or mistyping. To investigate the _MY_GOAT data set, a CSV file is produced for a visual inspection in Microsoft Excel (See Appendix A). STEP THREE: PUT EVERYTHING BACK The last step is to change _MY_SHEEP_SHR to the output data set &dsout., to append _MY_GOAT data sets, and to delete all temporary data sets in WORK library that start with _MY_, i.e. _MY_CONT, _MY_GOAT, _MY_SHEEP, _MY_SHEEP_SHR, _MY_TEMP. proc datasets library=work ; change _my_sheep_shr =&dsout.; %if %eval(&nobs_my_goat. >0) %then %do; append out=&dsout. Data=_my_goat(drop=&drop.); delete _MY_: ; The %IF statements checks if _MY_GOAT data set has more than zero observations. If the condition (&nobs_my_goat. >0 ) is TRUE then _MY_GOAT will be append to the final &dsout. data set otherwise the final data set will contain only _MY_SHEEP_SHR. A drop statement defines all variables to be dropped from the final dataset. STEP FOUR: OPTIONAL There are a few useful modifications that can be easily included into this macro. It is easy to check if the original data set is missing from the library; the data set has no observations; all values are missing or ZERO; whether the user provided parameters are valid; and whether or not to output selected variables from _MY_GOAT data set to a CSV file and what number of observations to output. For the sake of simplicity the macro in this paper does not include these modifications. CONCLUSION This paper presents a three-step approach to cleaning grouped duplicate observations. The steps are: PROC SQL to separate an original data set into two data sets based on the number of unique values of its variables, PROC SUMMARY with MAX to collapse grouped observations in a data set that has either missing or at most one unique value for all its variables, and a DATA step to set these two data sets back together. REFERENCES 1. Brett J. Peterson, Finding a Duplicate in a Haystack, 2. Wendi L. Wright, Checking for Duplicates, 3. Ron Cody, Cody s Data Cleaning Techniques 4. Base SAS 9.2 Procedures Guide, 3

4 ACKNOWLEDGMENT The author would like to thank Dr. Frank Han, Director of SAS Department of ClinOps, LLC for discussing the scope of this paper. CONTACT INFORMATION Your comments and questions are valued and encouraged. Contact the author at: Mayrita Vitvitska ClinOps, LLC (Pacific Data Designs, LLC and ClinPro, Inc. are now ClinOps, LLC) 353 Sacramento Street, Suite 800 San Francisco, CA Work Phone: Fax: Web: SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. indicates USA registration. Other brand and product names are trademarks of their respective companies. APPENDIX A: /* This macro collapses groups of _DUPLICATE_ obs. It splits a data set into two: all variables in the one dataset contain either missing or single unique values. The second data set contains all other observations. The macro removes _DUPLICATE_ observations from the first data set. In an example below all groups of observations where LN in ( JIN, MAY, MOM ) are considered _DUPLICATE_. ----Example ==> Original Obs LN v3 v1 v2 1 JIN B. 1 2 JIN B 2. 3 JIN MAY 4. 5 MAY MOM A.. 7 MOM A.. 8 TOM A TOM TOM TOM ==> remove_dups(dsin=t1, by=ln, notby=, keep_num=, keep_char=) Obs LN v3 v1 v2 1 JIN B MAY MOM A.. 4 TOM A TOM TOM TOM */ %macro remove_dups(dsin=, by=, notby=, keep_num=, keep_char=, drop=); /*---- LISTCOUNT() generates a text string for a use in PROC SQL to create counts of provided variables in (VARS) */ %macro listcount(vars); %local vars; 4

5 %let _listcount=count(distinct %scan(&vars,1)) as _0_%scan(&vars,1); %let _listcount = &_listcount., count(distinct %scan(&vars,&i.)) as _0_%scan(&vars,&i.); %put &_listcount.; &_listcount. /*---- COUNT_GE_2() generates a text string for a DATA STEP to split a data Set into two depending on the values of the count variables */ %macro count_ge_2(vars); %local vars; %let _0_count2= (_0_%scan(&vars,1) ge 2) ; %let _0_count2=&_0_count2. OR (_0_%scan(&vars,&i.) ge 2) ; %put &_0_count2.; &_0_count2. /*---- LISTCOMMA() creates a comma separated list of variables ----*/ %macro listcomma(vars); %local vars i; %let _listcomma=%scan(&vars.,1); %let _listcomma=&_listcomma., %scan(&vars.,&i.); &_listcomma. /*---- LISTQUOTE() creates a quoted list of variables ----*/ %macro listquote(vars); %local vars; %let _listquote=%sysfunc(quote(%scan(&vars,1))); %let _listquote=&_listquote %sysfunc(quote(%scan(&vars,&i))); &_listquote /*----- generate a variable list that are in DSIN and NOT in &by. and &notby------*/ select name into: vars_list separated by ' ' from dictionary.columns where libname="work" and upcase(memname)=upcase("&dsin.") and upcase(name) not in (%upcase(%listquote(&by.)) %upcase(%listquote(&notby.))); /* Split the original dataset into two */ create table _temp as select *, %listcount(&vars_list.) from &dsin. group by %listcomma(&by.); 5

6 data _my_goat(drop=_0_:) _my_sheep(drop=_0_:); set _temp; if %count_ge_2(&vars_list.) then output _my_goat ; else output _my_sheep; /*---- Find the number of observations in NOT _DUPLICATE_ data set */ select nobs into: nobs_my_goat from dictionary.tables where libname="work" and memname="_my_goat"; /*---- Generate a CSV file for a visual check of observations with different values */ %if %eval(&nobs_my_goat. >0) %then %do; x echo "NOTE: ========> Check &dsin.. There are observations with different values for &by. variable."; ods listing close; ods csv file="_check_goat_&dsin..csv"; proc print data=_my_goat; ods csv close; ods listing; /*---- define character and numeric variables for a PROC SUMMARY */ proc contents data=_my_sheep out=_my_cont(keep = name nobs type) noprint; %let col_name_char=; %let col_name_num =; select distinct name into: col_name_num separated by ' ' from _my_cont where upcase(name) not in (%upcase(%listquote(&by.)) %upcase(%listquote(&notby.))) and type =1; select distinct name into: col_name_char separated by ' ' from _my_cont where upcase(name) not in (%upcase(%listquote(&by.)) %upcase(%listquote(&notby.))) and type =2; %let col_name_num = &col_name_num. &keep_num.; %let col_name_char= &col_name_char. &keep_char.; /*---- Display a NOTE if there are NO numeric variables and stop the macro execution*/ %if (&col_name_num.=) %then %do; x echo "NOTE: > There are no numeric variables to summarize on."; %goto graceful; /*---- Collapsing grouped observations */ proc summary data=_my_sheep nway; class &by.; id &col_name_char.; var &col_name_num.; output out=_my_sheep_shr(drop=_type FREQ_) max()=; /*---- Create the output data set and clean up */ proc datasets library=work ; change _my_sheep_shr =&dsout.; %if %eval(&nobs_my_goat. >0) %then %do; append out=&dsout. Data=_my_goat(drop=&drop.); delete _MY_: ; %graceful: 6