Data Quality and Cleaning

Transcription

1 Data Quality and Cleaning A Case of Mobile Phone Survey Data INNA KOUPER DATA TO INSIGHT CENTER SCHOOL OF INFORMATICS AND COMPUTING INDIANA UNIVERSITY September,

2 Why DQ Data becomes: Big Frequent Heterogeneous Collaborative Integrated Reusable Shared

3 Agricultural Decision Making and Food Security in Africa When to plant and harvest What to plant How to grow What weather looks like Are all of your maize fields planted now? Did it rain on your fields this week? Did you plant any maize in the last 7 days? How many 50kg bags of maize do you have in storage now? What seed variety did you plant?

4 Common activities impacting DQ Redman, Thomas C. Data Quality: The Field Guide. Digital Press

5 DQ - Database / industry approach Accuracy The data was recorded correctly. Completeness All relevant data was recorded. Uniqueness Entities are recorded once. Timeliness The data is kept up to date. Consistency The data agrees with itself. Exploratory Data Mining and Data Quality, T. Dasu and T. Johnson, Wiley, 2004

6 DQ - Government approach Utility The data is useful to the public. Objectivity The data is accurate, clear, complete, and unbiased. The data is documented and/or reproducible. The data is subject to peer-review. Integrity The data is protected from corruption and unauthorized action.

7 DQ - Research lifecycle approach Validity Accuracy Consistency Integrity Completeness Context

8 Factors affecting mobile data quality Sampling Medium / message No interviewer Technical e.g., poor network signal, discharged device Social / individual Errors Non-response Policy / economical e.g., literacy level, no funds for texting Limitations of mobile and texting platforms

9 Completeness

10 Accuracy 1 Did it rain this week? Please answer yes or no : Yes, yes, YES, YESS, YAS No, no, No!, N0, NO, no rain 50

11 Accuracy / Consistency Did it rain this week? Please answer yes or no : No Yes

12 Consistency How many 50kg bags of maize do you have in storage now? Week 1 30 Week 2 25 Week 3 25 Week 4 20 Week 5 60

13 Context: Can you interpret the data?

14 Approaches to data quality Preemptive Processes (data management) Metadata and domain expertise Diagnostic Statistics Databases (data mining) Retrospective Data cleaning

15 Processes Decide where to store raw data and products Standardize content and formats Assign responsibility: data stewards Monitor data Archive data

16 Data Processing Pipeline TextIt Server Quality Control Metadata Management Cleaning and anomaly detection Build products Development Server Test Server Production Server MongoDB

17 Data monitoring

18

19 Metadata From the platform From the team Country "uuid": "f07a bc-ab11-572f4b562aa9", Season "name": "harvest flow 25 Apr 2016", Creator "runs": 674, Date created "completed_runs": 252, Run start date "label": "coll_fuelwood" Run start time "label": "harvest" Run end date Run end time Flow type List of questions

20 Data cleaning Identify variable types Check for invalid / incorrect values and missing values Conduct frequency analysis (mean, median, min, max, STD) Identify outliers Identify what can be corrected Decide how to treat outliers and missing values Evaluate time needed for automated and manual cleaning Use tools or manual correction Are there patterns in missing data?

21 Using Google (Open) Refine to clean data String to number Remove words ( bags ) Cluster similar values ( NO, no )

22 Difficulties in cleaning How many 50kg bags of maize do you expect to harvest? !50 You cant tell now rains have jast staoted 5O KGS ONLY

23 Questions for discussion How can we define data quality to reflect differences in types of data and its uses and the dynamic nature of research? What indicators can help to track quality processes and improvements? Who is responsible for ensuring high data quality? What preemptive techniques can help to improve the quality of mobile data?