Automating the Capture of Data Transformation Metadata

Transcription

1 Automating the Capture of Data Transformation Metadata H.V. Jagadish Univ. of Michigan

2 George Alter, University of Michigan

3 Why Metadata? Data are useless without Metadata data about data Metadata should: Include all information about data creation Describe transformations to variables Be easy to create Our goal: Automated capture of metadata

4 A few words about ICPSR World s largest archive of social science data Consortium established member institutions around the world Founding member and home office for the DDI Alliance

5 Powered by DDI Metadata ICPSR is building search tools based upon Data Documentation Initiative (DDI) XML Codebooks (pdf and online) are rendered from the DDI.

6 Click here for online codebook Searchable database of 4.5M variables

7 What question was asked? Online codebook shows variable in context of dataset Link to online graph tool How was the question coded? Link to online crosstab tool

8 Click here for variable comparison Searchable database of 4.5M variables

9 Click here for online codebook Variable comparison display

10 Metadata for the American National Election Study What question was asked? Who answered this question? How was the question coded? Who answered this question?

11 Metadata for the American National Election Study Who answered this question? How do we know who answered the question? It s in the pdf. Who answered this question?

12 When data arrive at the archive No question text No interview flow (question order, skip pattern) No variable provenance Data transformations are not documented.

13 How is research data created? Most surveys are conducted with computer assisted interview software (CAI) CATI Computer-assisted Telephone Interview CAPI Computer-assisted Personal Interview CAWI Computer Aided Web Interview There is no paper questionnaire The CAI program is the questionnaire i.e. the program is the metadata

14 Original data Computer Assisted Interviewing Convert to DDI: Collectica MQDS others CAI CAI to DDI We already have tools to convert CAI to machinereadable metadata. Original metadata DDI XML

15 Original data What happens when a project modifies the data. Statistical Packages Computer Assisted Interviewing Command scripts: SPSS SAS Stata R Revised data CAI SPSS SAS Stata R Convert to DDI: Collectica MQDS others CAI to DDI Original metadata The modified data no longer match the metadata. DDI XML

16 Original data Statistical Packages Metadata are recreated after the data are transformed. Computer Assisted Interviewing Command scripts: SPSS SAS Stata R Revised data Convert to DDI: Collectica MQDS others CAI CAI to DDI SPSS SAS Stata R Original metadata SPSSSAS Stata R Transformations are documented by hand Stat Package to DDI DDI XML Extract metadata from SPSS/SAS/ Stata/R Data file DDI XML Extracted metadata

17 Statistics packages have limited metadata Variable names Variable labels Value labels No provenance

18 Original data Statistical Packages Automating the capture of transformation metadata. Computer Assisted Interviewing Command scripts: SPSS SAS Stata R Revised data CAI SPSS SAS Stata R Standard Data Transformation Language Revised metadata Convert to DDI: Collectica MQDS others CAI to DDI Script Parser Original metadata DDI XML SDTL XML Updater DDI XML Missing links that we will build.

19 What statistics packages should be covered? ICPSR Downloads by Format All downloads Studies with all formats Delimited text 43% 29% SPSS 22% 24% SAS 10% 12% Stata 19% 23% R 5% 12% Excel 0% 1% Other 0% 0% 100% 100% Number 378, ,663

20 Why do we need an SDTL? SPSS MISSING VALUES X(-1). IF (X > 3) Y=9. IF (X < 3) Z=8. Stata replace X=. if X==-1 generate Y=9 if X>3 generate Z=8 if X<3 SAS if X=-1 then X=.; if X>3 then Y=9; if X<3 then Z=8; Input Data X X X Output Data

21 Why do we need an SDTL? SPSS MISSING VALUES X(-1). IF (X > 3) Y=9. IF (X < 3) Z=8. Stata replace X=. if X==-1 generate Y=9 if X>3 generate Z=8 if X<3 SAS if X=-1 then X=.; if X>3 then Y=9; if X<3 then Z=8; Input Data Output Data X X Y Z X X Y Z X X Y Z

22 What happens when a missing value is SPSS in a logical comparison? Logical expressions including a missing value are considered Missing. Usually, Missing is equivalent to False. Stata Missing values are treated as numbers equal to infinity. So, any number is less than a missing value. SAS Missing values are treated as numbers equal to minus infinity. So, any number is greater than a missing value.

23 SPSS MISSING VALUES X(-1). IF (X > 3) Y=9. IF (X < 3) Z=8. Stata replace X=. if X==-1 generate Y=9 if X>3 generate Z=8 if X<3 SAS if X=-1 then X=.; if X>3 then Y=9; if X<3 then Z=8; Missing Values in Comparisons Input Data Output Data X X Y Z NULL X X Y Z X X Y Z

24 Benefits of automated metadata Metadata will be better capture All the information in the CAI can be included. Variable transformations can be described Automation will lower costs Metadata will not be discarded and re-created All metadata will be standardized and machine readable Codebooks with rich information can be rendered at will If we make it easy and beneficial, researchers will use it.

25 Continuous Capture of Metadata for Statistical Data (NSF ACI ) Project Partners Inter-university Consortium for Political and Social Research (ICPSR), University of Michigan Colectica Metadata Technology North America Norwegian Centre for Research Data General Social Survey, NORC, University of Chicago American National Election Study, University of Michigan

26 Questions? Ask George Alter