DELIVERABLE D12.6/12.7 IECM Database Extension & User Interface with Tabulator

Transcription

1 Project N : ACRONYM: Data without Boundaries DELIVERABLE D12.6/12.7 IECM Database Extension & User Interface with Tabulator WORK PACKAGE 12 Implementing Improved Resource Discovery for OS Data REPORTING PERIOD: From: Month 37 To: Month 48 PROJECT START DATE: 1 st May 2011 DURATION: 48 Months DATE OF ISSUE OF DELIVERABLE: 20 November 2014 DOCUMENT PREPARED BY: Partners 25, 8 CED, RODA Combination of CP & CSA project funded by the European Community Under the programme FP7 - SP4 Capacities Priority 1.1.3: European Social Science Data Archives and remote access to Official Statistics

2 The research leading to these results has received funding from the European Union's Seventh Framework Programme (FP7/ ) under grant agreement n

3 TABLE OF CONTENTS 1. IECM Database The Integrated European Census Microdata database IECM database extension since the beginning of DwB REDATAM: an online tabulator for IECM samples Advances in the online data browser Work done on REDATAM Converting IECM samples into REDATAM files REDATAM data dictionary Future developments: the SDA software /13

4 1. IECM Database 1.1 The Integrated European Census Microdata database The Integrated European Census Microdata (IECM) database ( provides access to census microdata samples from 16 countries, supported by corresponding metadata. Access to the system is restricted to qualified researchers, but for those who qualify the access is free of charges. Researchers must apply for access online and accept the conditions of access defined by the IECM project. The access rules are specified to be in accordance with the conditions laid down by the participating National Statistical Offices. Table 1 shows the availability of census microdata samples across 24 European countries as of February That table covers both what is presently available and what is in the preparatory phase. We use different symbols to identify the status of each sample within the IECM project. The IECM website offers access to harmonized micro data from 55 censuses (bold and underline). Table 1 - Census microdata samples in IECM (February 2014) Country 60s 70s 80s 90s 00s 10s Austria * Belarus Bulgaria * Croatia Czech Republic France * Germany Greece Hungary * Ireland ** Italy * Latvia Moldova Netherlands Poland * Portugal * Romania * Russia Slovenia Spain Switzerland Turkey Ukraine United Kingdom * Bold Microdata entrusted; Bold Disseminated; * Promised when available; ** 2010 census round sample entrusted 4/13

5 1.2 IECM database extension since the beginning of DwB 1 Since the startup of the Data without Boundaries project, in May 2011, CED has been involved in and administered the continuous updating of the IECM database under WP5 (task 5.2). This work has added 14 new European samples to the IECM database: France (06), Germany (70, 87), Ireland (71, 79, 81, 86, 91, 96, 02, 06) and Turkey (85, 90, 00). Before dissemination, each sample goes through a process of data formatting, harmonization and data documentation. In particular, this harmonization work is extremely important for later comparative analysis. Table 2 shows the distribution of harmonized variables by topic that is currently available at the IECM website. It also provides information about the type of variables that have been harmonized for the 14 new samples during the last 3 years. In total, CED has harmonized 278 variables at the household level and 440 at the person level. The IECM website provides extended documentation on all of these variables, including documentation of the universe, codes and comparability descriptions, in addition to several other aspects. The IECM website is based on the IPUMS data extraction system which allows checking the documentation in an interactive way while or before requesting the data. In addition, users may download complete sets of micro data documentation based on the DDI-C standard XML Codebook from the following address: This documentation includes information at the series, study and dataset level, including documentation for each variable. Table 2 - IECM harmonized variables by topic Integrated variables Unharmonized variables France 06 Germany Ireland Turkey Household Technical Group Quarters Geography Economic Utilities Appliances, mechanicals, other amenities Dwelling characteristics Constructed households Other Total Persons Technical Constructed household interrelationship Demographic Fertility and mortality Nativity and birthplace Ethinicity and language Education Work Income Migration Disability Other Total Total The work performed in that respect is documented in more details in Chapter 3 of deliverable D5.2 available [Here] 5/13

6 The data holdings in the IECM database are growing rapidly. CED is currently working on the harmonization of 19 samples that have been entrusted to the project but not yet documented to a level where they may be disseminated to the research community (see samples marked in bold in Table 1). Regarding the 2010 census round, there have been signed agreements with 9 countries (samples marked with an asterisk in Table 1). We expect that the 2010 micro data samples will be made available for preparatory work by the summer Most of these samples will be harmonized and disseminated in the June release of CED estimates that it on average takes one year of preparatory cleaning and documentation work between receiving the micro data and dissemination. To date, Ireland is the only European country to entrust the 2011 census micro data sample to the project. 6/13

7 2. REDATAM: an online tabulator for IECM samples 2.1 Advances in the online data browser In the description of the planned work of Work Package 12, Task 12.6 refers to the adaptation and use of the REDATAM software to construct comparative tables for integrated and harmonized variables from multiple countries or census rounds. REDATAM is a tabulation software product specially developed for such analytic work by CELADE (Latin American & Caribbean Demographic Center - Economic Commission for Latin America and the Caribbean). REDATAM allows census micro data processing and browsing online. REDATAM was originally designed to disseminate census data from one single census. When writing the application for the DwB project one of the decisions made to promote flexibility in accessing and analyzing micro demographic data was to adapt the REDATAM software to disseminate multiple censuses. This involves changing the interface and the internal code of the software and adapting the census micro data samples to REDATAM s hierarchical file structure. This task has been successfully carried out. There is a beta version of the enhanced REDATAM application working at the CED server: MPLE&MAIN=WebserverMain.inl 2.2. Work done on REDATAM REDATAM is an acronym for REtrieval of DATa for small Areas by Microcomputer. Redatam+G4 (abbreviated R+G4) is the fourth and most recent generation of the software. It can be used in Spanish, English or Portuguese, with Microsoft Windows 95, 98, NT4 or 2000 operating systems, on any IBM compatible microcomputer. The REDATAM website offers access to user's manuals for each of the three R+G4 modules. REDATAM is a free and user friendly software. It deals with hierarchical data files and uses an encrypted data storage system that does not allow the identification of personal data. When dealing with large datasets, as census are, it creates a highly compressed file format that minimizes storage. REDATAM runs on 100% census microdata, often with several million individual cases. Its way of storing the data allows for very fast processing of queries. It also breaks results by small areas, even by census tract. It offers the option to implement filters and weighting results to work with samples and not full counts. REDATAM outputs are compatible with standard data formats such as EXCEL or ASCII. The software presents an interface to map the results in thematic maps and display the results graphically. It also allows attaching external databases in.dbf format. REDATAM is widely used among the National Statistical Offices of developing countries. It is the most used application to disseminate census data in Latin America. 7/13

8 Converting IECM samples into REDATAM files A Redatam database is composed of its Dictionary, plus two other file sets: a) the data files; and b) the index or pointer files. Data files (.rbf). R+G4 stores each variable of an entity in a single file, with a "record" for each element in the entity. Then, each variable file looks like a vector of data. Those files are known as transposed files, the file is flipped to turn rows into columns and columns into rows. This technique distinguish them from the more common statistical data organizations, where all the variables of an element are stored in the same record, and one single file has all the information for all the elements. In a transposed file it is possible to read just the variables that are needed for a specific analysis, while in a traditional statistical file the whole matrix have to be read. Working on transposed files is therefore much more efficient if files have many variables, and are usually the file-structure used for interactive analytic use. In the transposed structure used in Redatam, there is a file for sex, another for household type, etc. This has major processing advantages. The processing speed is very fast, since it allows the system to read and process only the variables that are involved in each specific analysis. By turning every variable into a file of its own it also becomes a very robust system that it is easy to implement in different programming environments. Pointer files (Indexes.ptr) These files are needed to connect the entity elements with the lower entities. Each entity has an index file, with values that "point" from the elements in the higher entity (mother entity) to the elements of the entity itself. For documentation purposes, this produces a catalogue of the contents of the system. The CED team has converted all IECM samples into REDATAM readable files following the aforementioned structure, which combines.rbf and.ptr files. The process works as follows: - Download all IECM samples (sample by sample) - Use SPSS or SAS to create variable files (variable by variable) with a unique id per record (.ptr files) - Create.rbf files to link all.ptr files Once.rbf and.prt files have been created, we had to create and application to disseminate the data online. The structure of the application only allows up to 4 variables in a single tabulation (see images 1 and 2). However, the Area Break option of REDATAM allows adding one more variable to the procedure. When multiple samples are selected, the Area Break options changes by default to Sample break. The IECM online data analysis system (based on REDATAM) works as follows: Table Title: a description limited to 90 characters or less. Geographic Area/Sample Selection: the user can select ALL regions to be processed or a particular one. If the latter option, the user must click on the down arrow menu to display the list of available regions. When the user selects ALL regions, REDATAM collapses in a single table the information from all regions. For multiple sample selections, users have to select first which samples they want to work with. 8/13

9 Columns and Rows: The next panel allows the user to introduce up to four variables: one for the column and 3 for the rows. When using the Combination option the tabulator opens by default the Column window with the list of the Person variables and the Row1 window with the list of the Dwelling/household variables (To remind the user that is about to process dwelling variables with person variables). The Row2 and Row3 area blank. Area/Sample Break: Produces a separate table for each region in the sample or for each sample when multiple samples are selected Weight: This option allows the user to weight the results either by the dwelling/household weight or the person weight. When the use combines dwelling/household characteristics with person characteristics, the person weight is the option by default. Image 1 - REDATAM customization to disseminate IECM data. Only one sample. 9/13

10 Percent Option: Absolute value, Total %, Row % and Column %. Only one option can be used at each time. Filter: Enables the user to define the universe of analysis by selecting cases that meet one or more conditions. The filter option pops up a window that allows the user to build an arithmetic and/or logical query to select the cases that will be included in the universe of the requested table or graph. Output format: The user can choose different formats: Table, Graph (up to 2 variables), Table and Graph (up to 2 variables) Image 2 - REDATAM customization to disseminate IECM data. Multiple samples. 10/13

11 REDATAM data dictionary All the information about the entities and variables are stored in a file called Database Dictionary. The Database Dictionary corresponds to the full catalogue of the system and interrelates all the components of the metadata. This dictionary acts as a bridge between the user and the data files. It makes the visualization of the data easier by showing it in terms of variables and records. The dictionary prevents the users from dealing with the complex structure of storing the data. The dictionary can be visualized as a list of all the variables in the database for each of the hierarchical levels (entity) and a list of the codes (values or categories) for each variable. The dictionary includes a brief description of the meaning of each code. For example, the codes for the sex variable of the person level are 1 and 2, where 1 equals male and 2 female. The user can examine the dictionary, make searches, and print specific sections. Image 3 - IECM Online Data Analysis System, view of the Data Dictionary (educational attainment as an example). 11/13

12 2.3. Future developments: the SDA software The IECM team is considering adopting other tabulating software to overcome some of the limitations of REDATAM. REDATAM was designed to disseminate census microdata for one census at a time. From the user perspective, it is user friendly software. However, from the producer perspective, it requires substantial data reformatting. If there is only one census to be disseminated, this is not a major drawback. However, the IECM database is a live database that gets updated with new censuses and variables at least once a year. Any minor adjustment to the database (change of code, new variable or new samples) requires reformatting the whole database. Considering this time-consuming task, the IECM project team has been exploring the possibility to implement the new tabulator software for the European samples, developed by the Minnesota Population Center (MPC) to disseminate the IPUMS international census microdata samples (of which the IECM project is the European part). This tabulator is based on the SDA software package, developed and maintained by the University of California at Berkeley. A cost/benefit analysis has led to the conclusion that implementing SDA in the IECM project would circumvent the burden described above for adjusting / updating the IECM database (by allowing creating new variables with recode and compute procedures) while facilitating documentation & dissemination. It would also greatly improve the users' experience with the data as it notably allows searching both within a single sample (at the variable level) and across samples (at both the variable and sample level); while providing a number of tools and data analysis capabilities. 12/13

13