EXAMPLE 10: PART I OFFICIAL GEOGRAPHICAL IDENTIFIERS IN THE UNDERSTANDING SOCIETY PART II LINKING MACRO-LEVEL DATA AT THE LSOA LEVEL

Transcription

1 EXAMPLE 10: PART I OFFICIAL GEOGRAPHICAL IDENTIFIERS IN THE UNDERSTANDING SOCIETY PART II LINKING MACRO-LEVEL DATA AT THE LSOA LEVEL DESCRIPTION: The objective of this example is to illustrate how external data can be linked to Understanding Society using look-up files between the household identifier and official geographical identifiers. PART I describes which geographical look-ups are available with the Understanding Society study, and under which conditions. It also describes why you will be using fictitious neighbourhood-level data in the second part of this example. Please do not use the data for your analysis we have made them up! PART II is a step-by-step guide to using a Special Licence data file at the level of Lower Layer Super Output Areas (LSOA) to enrich the Understanding Society data with macrolevel indicators at this spatial scale. The specific example is the Department for Transport (DfT) s Accessibility Statistics series which we will link to an individual-level Understanding Society data file in long format. EXAMPLE RESESEARCH QUESTION(S): Are people who have better access to public transport more likely to enter employment than those with poorer access to public transport? FILES: w_fakelsoa01.dta, w_indall.dta EXTERNAL DATA FILES: fakeacs xls; fakeacs xls WAVES: 1, 2 NEW COMMANDS: import excel, destring, substr(), do filename.do STEPS: Part I 1. Official geographical units in Understanding Society 2. Special Licence official geographical unit look-up files 3. Disclaimer about the data used in the Example Part II 4. Setting up the directories for using Special Licence data 5. Familiarising with the Understanding Society LSOA look-up file 6. Familiarising with the LSOA-level external data file 7. Preparing Accessibility Statistics for merge with the Understanding Society -LSOA look-up file 7.1 Convert the data to Stata format 7.2 Rename variables consistently across years 7.3 Rename variables consistently across domains 7.4 Generate unique identifiers to merge with Understanding Society files 8. Merge files with Understanding Society -LSOA look-up file 9. Merge file with Understanding Society individual level file 1 E x a m p l e 10

2 PART I 10.1 OFFICIAL GEOGRAPHICAL UNITS IN UNDERSTANDING SOCIETY Understanding Society includes a great deal of geographically referenced indicators and regional information. In addition to regular collections of information around the look and feel of the neighbourhood (reported by the interviewer, see data file w_hhsamp), neighbourhood social cohesion and services provided in the local area (reported by the respondents, see w_indresp), Understanding Society contains variables that indicate the area in which survey respondents live at the time of the (household) interview. The values of these variables refer to official geographical units used by administrative bodies such as ONS, local authorities and Royal Mail. Examples of geographical identifiers at higher level include country and Government Office Region; at medium level Local Authority Districts and Travel to Work Areas; at lower level Lower Layer Super Output Areas and Output Areas. On the basis of these unique identifiers it is possible, in principle, to match Understanding Society data with official, scientific or commercial macro-data at these levels. There are plenty of such published data tables available, see, for example, The availability of official geographical units in Understanding Society depends on their spatial scale and on the location of the site of access. Indicators at larger scales such as country or region of residence are distributed with the general release (the variables w_country and w_gor_dv are included in all Understanding Society files) 1. Official geographical units at medium-level scales are available to UK and non-uk users applying for a Special Licence, whereas lower-level identifiers are only available to UK users holding a Special Licence. Finally, the grid reference of postcodes is available through Secure Data Access. For more information on the access conditions visit SPECIAL LICENCE OFFICIAL GEOGRAPHICAL UNIT LOOK-UP FILES The structure of the data files that users can access under Special Licence follows the same principle: The file is a look-up file between wave-specific Understanding Society household identifier (w_hidp) and the specific official geographical unit identifier that you have been granted access to. The look-up file covers all sampled households in that year for which a valid postcode was available. The Royal Mail s postcode look-up file is used to create higher level geographical unit codes. The study currently provides official codes for the 2001 and 2011 Census geographies. Note that you can only use the look-up file for the research that you have described in your application for the Special Licence (or Secure Data Access). If you are in breach of the agreement you and your employer could face enormous penalties such as no further access to research council funding for a specified number of years. You really want to be careful! Important restrictions placed on Special Licence data include, but are not limited to, the following. You are not allowed to: link to macro-indicators or at spatial scales that you have not specified share the look-up file or linked data with co-workers who are not mentioned, or have not been approved, in the Special Licence 1 Due to a processing error the variable w_country is not included in the Understanding Society: Waves 1-4, , release. The country variable can easily be generated from w_gor_dv as a quick fix. The variable will be added back in in the next release. Apologies for any inconvenience caused. 2 E x a m p l e 10

3 keep the linked data or the look-up file for future projects or beyond the specified project deadline store the data in an unprotected area (special data storage restrictions apply!) You may, of course, keep the do-files used to prepare and analyse the linked data. This allows you to re-apply for the Special Licence, and to re-do the analysis if you need to this beyond the original project deadline. You must not include in your do-file comments which allow unauthorised people access to Special Licence information. For instance, the comment recode postcode CO43SQ to CO4 3SQ for w_hidp=xyz is not permitted and can easily be written neutrally add space after first three digits of postcode for w_hidp==xyz ; rather than putting the specific digits of the postcode into a replace command, you can use the substr() function. We will learn more about this below DISCLAIMER ABOUT THE NEIGHBOURHOOD-LEVEL DATA USED IN THIS EXAMPLE The example is based on the Understanding Society-look-up file with Lower Layer Super Output Area (LSOA) identifiers and linkage to a specific published data series. We cannot share the official Special Licence data with you and we must make sure that you cannot link any Understanding Society data back to published tables used in this example. You will, therefore, work with fictitious neighbourhood data that have been created merely to illustrate the steps involved in linking external data using geographical identifiers in Understanding Society. We have issued each household with a randomly picked fake LSOA. Across waves, we allowed some proportion of households to continue living in the same fake LSOA and within waves some households share the same fake LSOA. Whilst this pattern will hold also in the non-scrambled Understanding Society -LSOA look-up file, the particular people who appear to be living in the same fake LSOA over time or with members of other households in the same wave is randomly determined. Fake LSOAs share relevant features with the official LSOAs. Most importantly, unlike all variables we have used in the previous examples, LSOA codes are complex string variables and the first digit provides information about the country of residence. Official codes for areas in England, Wales and Scotland start on the letters E, W and S, respectively (fake LSOA: A, B and C, respectively), followed by a 8-digit number. Official codes for Northern Ireland are 8-digits long starting on 95 and followed by a mix of letters and numbers (fake LSOA: 9-digits long, starting on 95AX followed by four numbers). Questions: Where in the UK do the households listed in Table 1 live? Do households 2, 3 & 8 live in the same neighbourhood? And how about households 6 & 7? Table 1 Fictitious examples of fake LSOA (w_fakelsoa01) and official LSOA (w_lsoa01) string variables w_hidp w_fakelsoa01 w_lsoa AX HH09W1 2 A E A E B W E x a m p l e 10

4 PART II 5 B W C S C S A E SETTING UP THE DIRECTORIES FOR WORKING WITH SPECIAL LICENCE DATA You have been granted access to the Special Licence (SL) data file w_fakelsoa01.dta (NB. In this example we add the word FAKE to make it absolutely clear that this is fictitious data). Your SL agreement stipulates that you will keep this file in a specially protected area, hence, as a first step, define a path linking to the directory where you keep the sensitive projectspecific data. global access "\\isernfs1\conferencedata\ukhls\sl\" SL data and output should be stored in a designated folder so you can more easily manage the data (and results) appropriately. Create a new directory in your home area (M:\), which you call example10 where you can store data files you create for this project. You can do this interactively in Stata using the command mkdir. Alternatively, you can include this command in your do file. However, if you do, remember to add a capture command before this. Otherwise, when you run the do file a second time round, it will give an error message saying that a folder with that name already exists. capture mkdir M:\example10 Make example10 your working directory. cd M:\example FAMILIARIZING WITH THE UNDERSTANDING SOCIETY-LSOA LOOK-UP FILE You can now load the SL data into Stata s memory and inspect it. use ${access}\a_fakelsoa01.dta, clear describe The file contains just two variables, the household identifier as used in the study (w_hidp) and a variable called w_fakelsoa01. The data file contains information for all households issued to field for whom a valid postcode in the UK was available. The variable w_fakelsoa01 reports the official LSOA codes for all four countries of the UK. Note that only England, Wales and Northern Ireland actually use the term LSOA and the population thresholds and boundary construction for the units differ across countries. The comparable official geographical unit for Scotland is called Datazone. In total, the area of the UK is split into 41,775 such areas: There are 32,484 LSOAs in England, 1,896 LSOAs in Wales, 890 LSOAs in Northern Ireland and 6,505 Datazones in Scotland; households in Understanding Society will live in many of them but not in all of them. 4 E x a m p l e 10

5 The variable w_fakelsoa01 is stored as a string with the length of 9 characters. You need to be aware that a very large number of commands in Stata do not recognize string variables. A very useful function for string variables is substr(). substr(s,n1,n2) returns the substring of s, starting at column n1, for a length of n2. The first digit of the LSOA variable indicates the country. Hence, you can use this to derive country of residence variables. To generate a dummy variable for households living in England you can write a conditional statement that evaluates whether or not the first digit of the LSOA string equals the letter A. Remember that you need to write the letter into quotation marks so Stata knows it is referring to a string (and not a number): gen england=(substr(a_fakelsoa01,1,1)=="a") lab var england "lives in england" lab def england 1"yes" 0"no" lab val england england Remove the wave prefix, generate a wave indicator and store the data file. renpfix a_ gen wave=1 lab var wave "survey wave" compress save lsoa01.dta, replace Repeat the same procedure for the Wave 2 LSOA data file, append the Wave 1 file and store it. use ${access}b_fakelsoa01.dta, clear renpfix b_ gen england=(substr(fakelsoa01,1,1)=="a") lab var england "lives in england" lab def england 1"yes" 0"no" lab val england england gen wave=2 lab var wave "survey wave" append using lsoa01.dta save lsoa01_long, replace Note that this long format data file identifies observations uniquely via hidp and wave. It is not a panel data set, however. This is because hidp is not a unique identifier of households over time. Understanding Society does not provide longitudinal household identifiers. To use the information in a longitudinal analysis, you ll have to distribute the information to an individual-level data file in long format using hidp and wave FAMILIARIZING WITH THE EXTERNAL MACRO-DATA AT LSOA-LEVEL Having familiarised yourself with the LSOA look-up file, you now can prepare relevant published tables at the LSOA level. Macro-level data tables can come in all shapes and forms. You may have downloaded data files from the Census website, or you may have seen a listing of characteristics in a publication and copied & pasted them into an Excel spreadsheet. In this example, we will use the Accessibility Statistics series. The statistics have been produced for 2005, and then annually from by the Department for Transport (DfT), and are available for download in CVS format from the following URL: 5 E x a m p l e 10

6 csv.zip Accessibility Statistics relate to eight domains of services: Employment Centres, Primary Schools, Secondary Schools, Further Education, General Practitioners, Hospitals, Supermarkets, and Town Centres. Data is presented for all official LSOA 2001 in England (N=32,484) in separate CVS files for each year and domain (i.e., 6x8 data tables!). For each domain and year there are up to 108 variables reporting how many services are available to what number and proportion of users within certain travel times and by different modes of transportation. This wealth of information offers great opportunities for longitudinal research! IMPORTANT NOTE When using external macro-level information in a longitudinal analysis, you have to check whether there were changes over time in definitions. Unlike in Understanding Society, information that appears to be the same across years because it has the same variable name may not have been produced in the same way. For example, in the Accessibility Statistics series, the data sources for each service (other than town centres) have changed over time. In addition, LSOA are not necessarily longitudinal identifiers of a specific place over time. This means that any change in travel time might be as a result of an increase /decrease of the number of destinations in the dataset; an actual increase /decrease in the number of destinations in England; a change in public transport timetables, road layout, congestion or cycle routes; a change in the underlying definition of the LSOA; or a combination of all five. You can still go ahead using the data in your longitudinal analysis but should consider how these data limitations affect the interpretation of results. Whenever possible, you should include some robustness checks in your analysis to rule out some sources of change PREPARING ACCESSIBILITY STATISTICS FOR MERGE WITH UNDERSTANDING SOCIETY- LSOA LOOK-UP FILE To use external macro-level data files in your Stata-based longitudinal analysis of Understanding Society you (typically) need to go through all of the following steps: 1. convert the external file to Stata format 2. rename the variables so they are consistent across waves 3. rename variables so they are unique across domains 4. generate unique identifiers that allow matching with the Understanding Society lookup file 5. merge multiple waves of neighbourhood data to create markers of neighbourhood change In this example, we will not be able to look at all Accessibility Statistics tables and we will also not have the time to work with data from different services domains. We will restrict ourselves to the Employment Centre domain of the Accessibility Statistics, and focus on only two years of data, i.e., 2009 and We have already downloaded the respective CSV files and replaced the official LSOA indicator with our fake LSOA codes. We also removed other geographical identifiers from the table. In addition, we made some random changes to values in the data table. The data 6 E x a m p l e 10

7 are, therefore, not identical to published Accessibility Statistics but share the relevant features with the data. Task: Open and inspect the file called fakeacs0501_2009.xls in Excel. The data table includes a header and also some table notes. The substantive information that you would like to convert to Stata format is stored in columns A to DD and rows 8 to Variable names are stored in rows number 6 and 7. Either of these rows could be used to create variable names for the new Stata file. In this example we will use row 7 and we have made some changes to those original variable names that Stata would fail to read properly. In particular we added a c in front of variable names starting on numbers and replaced the symbol % with p Convert the external file to Stata format Stata includes a whole range of commands that allow you to import data files in different formats, see help import. Excel files can be imported using the command import excel using filename. If you want to import the entire worksheet, you only need to specify the name and directory of the file. If you want to specify only some cells you need to specify the cellrange() option. You can also tell Stata to treat the first row of the imported data as the variable name. import excel using "${access}fakeacs xls", /// cellrange(a7:dd32489) firstrow clear describe The file description indicates that all variables in this file are string variables. Most analytic commands in Stata only work, however, with numeric variables. When you list out a couple of cases you will see that many of the cells do in fact contain numbers. list emplpttime2 emplptfrequency2 emplcycletime2 emplcartime2 in 1/30 Some commands allow you to flexibly convert numeric variables to string variables (tostring) and string variables to numeric (destring). The respective commands options replace and force will tell Stata to produce missing values for each cell for which the conversion is not possible. See help tostring or help destring. Turn the variable emplpttime2 into a numeric format: destring emplpttime2, replace The command failed to produce the desired results; there is a note emplpttime2 contains nonnumeric characters; no replace. Try again, this time using the force option and generating the new variable ectimept instead of overwriting the existing variable. destring emplpttime2, gen(ectimept) force The command has now produced the desired result. It also reports that 1 missing value has been produced. This is curious! Browsing the observation for which ectimept is [.] suggests that the problem is driven by the characteristics of one FAKElsoa01: All characteristics are set to the string.. and Stata does not know to interpret this as a numeric [.] missing value. Turn all apparently numeric variables into numeric using the force option: destring Empl* Pop* empl* c* All* RISK* p*, replace force 7 E x a m p l e 10

8 Rename the variables so that they are consistent across waves (see next section) Rename variables so that they are unique across domains The variable names and labels are not intuitive (and quite long, and a mix of capital and small letters) and it will be difficult to recognise them later on. Moreover, if you were to take a look at Accessibility Statistics for other services you would see that some variables have identical names but report different information. A simple trick to fix this problem is to add a domain prefix to all variables from one domain. But there are obviously some other desirable renames to be made. To relieve you of the task to come up with consistent variable names and labels, we have prepared an extra do file that renames the variables of interest so they all start with the letters "ec". You can execute the do-file from within your current do file: do ${access}example10_lb.do Task: Browse the data file to see which variables have been renamed. Keep only the variables of interest, i.e., the LSOA identifier and all variables starting on ec! You may want to sort the data on the unique identifiers in the data file and put them to the top of the variable list (but it is not necessary) GENERATE UNIQUE IDENTIFIERS THAT ALLOW MATCHING WITH THE UNDERSTANDING SOCIETY LOOK-UP FILE Finally, rename the LSOA variable so that it has the same name as the LSOA in the Understanding Society-LSOA look-up file that you have created earlier. You also need to generate a wave identifier. We will treat Accessibility Statistics 2009 as the relevant neighbourhood context for wave 1 data (and Accessibility Statistics for 2010 for wave 2). Then save the new data set: rename LSOA_code FAKElsoa01 gen wave=1 lab var wave "survey wave" save acs0501_2009, replace Repeat the same process for the 2010 Accessibility Statistics table. This time around, you can be more efficient by reducing the number of variables in the data file before converting them to numeric. import excel using "${access}fakeacs xls", /// cellrange(a7:dd32489) firstrow clear do ${access}example10_lb.do gen FAKElsoa01=LSOA_code gen wave=2 lab var wave "survey wave" keep FAKElsoa01 wave ec* destring ec*, replace force save acs0501_2010, replace MERGE MULTIPLE WAVES OF NEIGHBOURHOOD DATA TO CREATE MARKERS OF NEIGHBOURHOOD CHANGE Last, but not least, append the Accessibility data for wave 1 to those of wave 2 in memory. 8 E x a m p l e 10

9 append using acs0501_2009 You now have a longitudinal data set of neighbourhood characteristics and you can use it to generate markers of neighbourhood change. An easy way to do this is by using Stata s inbuilt longitudinal functions (such as L. and F. to create lags and leads). Stata will not recognize string variables as the unit identifier, so you cannot use FAKElsoa01 in the xtset command. Create a unique numeric identifier for each FAKElsoa01: bysort FAKElsoa01: egen index=min(_n) lab var index "LSOA ID (scrambled)" list FAKElsoa01 wave index in 1/20 Use index and wave to tell Stata the panel structure of your data set, then create the lag, lead and change of the number of people in the LSOA who receive Job Seeker s Allowance (Hint: use ecrisk) using Stata s L., F. and D. functions.: xtset index wave gen ecrisk_ly=l.ecrisk lab var ecrisk_ly "# of people in this LSOA receiving JSA last year" gen ecrisk_ny=f.ecrisk lab var ecrisk_ny "# of people in this LSOA receiving JSA next year" list FAKElsoa01 wave ecrisk ecrisk_ly ecrisk_ny in 1/20 Finally, sort the file on the merging variables and save the data file under the name acs0501_long. sort FAKElsoa01 wave lab data "LSOA ID (scrambled)" save acs0501_long, replace Question: We argued that there are no longitudinal household identifiers in Understanding Society because the composition of a household may change over time. You can, therefore, not use hidp as the unique unit identifier in panel data analysis. Clearly, the composition of the LSOA may change over time. Nevertheless, we have used FAKElsoa01 as the unique unit identifier in panel data analysis. What assumptions does this make, and how may this affect the interpretation of your results? Is there anything you could do to alleviate some of these concerns? 10.8 MERGING THE DATAFILES ON THE GEOGRAPHICAL INDICATOR After all this (tedious) data preparation work, merging the Accessibility data file with the Understanding Society data file is really easy. You just merge the files on FAKElsoa01 and wave. merge m:m FAKElsoa01 wave using lsoa01_long 9 E x a m p l e 10

10 ta _merge Question: Take a little time to think about why there are so many non-matching cases. Who are the households for whom we do not have Accessibility Statistics, and why are there Accessibility Statistics that could not be linked to Understanding Society households? Keep data only for Understanding Society households living in England, and remove the official geographical locator variable. Remember that you now have the anonymised LSOA ID index which will help you disentangle whether a household still lives in the same or in a different LSOA, and who lives in the same neighbourhood. keep if _merge==3 drop _merge FAKElsoa01 drop _merge Question: Which two variables do uniquely identify the rows in this data set? Can you use this data set in your longitudinal analysis? Task: Sort the data set on the unique identifiers and save the data as hh_access_long.dta in your working memory. You can now link the file to other Understanding Society data files in long format merging on hidp and wave. 10 E x a m p l e 10