DDI metadata for IPUMS I samples

Transcription

1 DDI metadata for IPUMS I samples Wendy Thomas Workshop Integrating Global Census Microdata : Dublin Ireland, 58th ISI

2 What is DDI DDI is a metadata standard d focused ocusedprimarily on microdata from the social, behavioral, economic, medical, and environmental sciences DDI captures metadata on the intent of the study, data collection, data processing, data file structures, t dissemination i and preservation information Metadata standard which is receivingstrong support from National Statistical Agencies around the world

3 Two DDI Development Tracks DDI Codebook Original DDI structure focused on the standard contents of a complete detailed codebook for a data file, primarily a presentation focus Version used by the IHSN Microdata Toolkit and supported by NESSTAR DDI Lifecycle Focus is on the full lifecycle of the data Developed to support complex studies, longitudinal data files (series), and support data collection processes, adds a production focus GSBMP is based on a DDI Lifecycle model

4 DDI metadata for IPUMS I I samples TheIHSNMicrodata Toolkit is used by over 80 countries to document their internal surveys, many of these countries are also members of IPUMS I Providing a DDI metadata filecompatiblewith the IHSN Microdata Toolkit would facilitate use of this data by the NSO A project to map IPUMS I metadata to DDI was undertaken in 2010 funded by OECD/Paris21

5 Project The project was to extract metadata from the IPUMS database system and to create IHSN Microdata Toolkit compatible metadata for the original data files and for the IPUMS extracts of data for developing countries. IPUMS contains metadata and data for 106 censuses from developing countries world wide Goals: Facilitate access to existing full data sets within the IPUMS member country Populate a number of microdata catalogs based on NADA W.L. Thomas 5

6 IPUMS Metadata Extensive, detailed metadata held as original documents, structured text files, and structured database Basic division is between metadata to: Inform the user Drive the harmonization and/or search and extraction engine Retains the original metadata for all included original variables: Description Source Question Enumerator Instructions Adds information on harmonization process and variables W.L. Thomas 6

7 IHSN Microdata Toolkit Developed by the World Bank Data Group for the IHSN to promote the adoption of international standards and best practices for microdata documentation, dissemination andpreservation preservation. Consists of 3 modules Microdata Editor: Documents using DDI Codebook and Dublin Core standards Explorer: Views metadata and exports to common formats CD ROM Builder: Create user friendly CD ROM and web products W.L. Thomas 7

8 Process The project focused on metadata within the IPUMS database system which could be mapped out to generate a IHSN Microdata Toolkit compatible set of metadata The desired output consisted of A set of Dublin Core descriptions of related technical documents and copies of those documents A set of metadata for the original country data A set of metadata and data for the content of the public extract system (harmonized and country specific variables) All outputs would be delivered to the country NSO for their use Metadata content and document copies would be provided to IHSN for inclusion in their catalog W.L. Thomas 8

9 IPUMS Process: Creating the Database STEP 1: Incoming data and metadata Prepare data for entry into IPUMS harmonization process and database Data arrives in non specific formats and reflects the original structure of the data at the NSO or a standard extract format Metadata arrives in non specific format with inconsistent content coverage (may or may not have English language equivalents) W.L. Thomas 9

10 IPUMS Process: Creating the Database STEP 1: Incoming data and metadata Prepare data for entry into IPUMS harmonization process and database STEP 2: Intermediary data and metadata Clean and restrict for confidentiality Harmonize Establish metadata links Data is selected and reformatted into household and person records, IPUMS generated variables are added for processing Basic metadata is entered into a spreadsheet including; original name, description, IPUMS name, translations, frequency (generated), and all extant content types (including errors). Creation of database W.L. Thomas 10

11 IPUMS Process: Creating the Database STEP 1: Incoming data and metadata Prepare data for entry into IPUMS harmonization process and database STEP 2: Intermediary data and metadata Clean and restrict for confidentiality Harmonize Establish metadata links STEP 3: IPUMS Clean, confidential, harmonized data Metadata includes links to original data, questions, instructions, and transformation ti processes Original data structures have not been retained in the dtb database, includes only the cleaned, restricted content and harmonized variables created by IPUMS W.L. Thomas 11

12 Results The MPC is now capable of creating DDI/IHSN/NESSTAR compatible codebooks db kfor all extracts t on all microdata systems Within 24 hours of submission of metadata copies to IHSN, all metadata files were loaded into the IHSN NADA based catalog and were searchable down to the variable level in that system Countries can create full extracts of their census data to match the intermediary structure (household/person record) found in the metadata, import it to the Microdata Toolkit and explore or create desired output easily Countries can easily compare or create products from the harmonized data from their country or download additional data from the IPUMS International system for cross national comparison W.L. Thomas 12

13 Distribution and Process Metadata IPUMS, like most metadata databases, contains the majority of required objects for supporting distribution metadata in a DDI format (codebook or lifecycle) The process of mapping to a detailed standard raised specific issues and highlighted possible improvements to the database content and structure Brought to light the need to capture more metadata at an earlier point in the process. Metadata produced outside the database structure was lost Object level version control was missing so intermediary content was lost W.L. Thomas 13

14 Conclusions The ability to create distribution metadata in a standard format increases utility for users, provides new product options, provides easy means of populating external catalogs The process of mapping to a well formed standard provides an opportunity for content and process improvement Capture of incoming metadata is critical to shifting from a distribution metadata structure to a process metadata structure Process structures not only can capture the development of the data within the system but can be used to drive the process and data and metadata checks W.L. Thomas 14

15 Contact Information DDI Wendy Thomas