Cleaning the data: Who should do What, When? José Antonio Mejía Inter American Development Bank SDS/POV MECOVI Program February 28, 2001

Transcription

1 Cleaning the data: Who should do What, When? José Antonio Mejía Inter American Development Bank SDS/POV MECOVI Program February 28, 2001

2 Precious resource Better to answer these questions than to have to wonder: Who? Where? With what? When? Why? Integrated surveys cannot afford to lose observations. The integrity of the data must be preserved.

3 Who does What? Data quality control has been pushed back as close to production as possible. The best person to correct errors is the informer. There is still some work that needs to be done at the central office so that the final data files are ready for analysis and distribution. Definitively in terms of organization, but also some in terms of data quality.

4 Do the job while the questionnaires are still at hand. Go back to them, check and re-check them. The work done at the central office should be one of CLEANING not of CHANGING. Cleaning is based on facts, on reviewing the questionnaires. Changing is altering the data. I don t think that is correct

5 Data quality checks after data entry (Who: Data manager in central office) Gather all files with household data, verify that all hh are included without duplication. A good system of hh identifiers should facilitate this process. Convert household based files into thematic files more useful for analysis. Files should be converted to the software format that will be used for analysis. Convert to popular software languages (SAS, STATA, SPSS). Always keep a master version of the files in ASCII.

6 Data quality checks after data entry Check structural consistency of files, so that different thematic files with hh data can be matched with each other, with individual data and community data. Compile basic univariate statistics for each variable. Frequencies for qualitative variables. For quantitative variables, the minimum, maximum and mean values. Issues with logical consistency have been taken care by a good data entry program and concurrent data entry.

7 Data quality checks after data entry Any issues left related to logical consistency should be left to the analysts. Because there is no consensus as to how to identify or treat outliers and missing observations. Best to give the raw data and let each analyst perform whatever cleaning she/he thinks best.

8 Data quality checks after data entry The statistical office as an analyst can do some more data cleaning. If the office chooses to make public its corrections, imputed values, treatment of outliers and missing values, and aggregated variables, it should always do it with the proper labels and tags, plus a well documented account of what was done. And, of course, this data should be distribute in addition not instead of the original data.

9 Data cleaning Minimize dirty data focus on training, field work, and data entry. Document changes keep program files with explanation of all changes made to the raw data, and construction of variables. Maintain original data give users the option to change assumptions. Use robust estimation techniques use statistics that are relatively insensitive to outliers (e.g. median instead of mean, etc.).

10 Distribution Organize data by section/section sub-part One to one correspondence with questionnaire. File naming convention should be informative, transparent and well documented. Add variable and value labels. Links between variables in questionnaire and data files should be clear and well documented. Distribute original data files with good and complete documentation, users are analysts not detectives.

11 Key messages Extremely important steps! Bad data is just bad data. Good data badly organized is bad data. Try to catch errors early. Documentation is essential. Cleaning is very different from changing. Learn from mistakes.