Use data on individual respondents from the first 17 waves of the British Household

Transcription

1 Applications of Data Analysis (EC969) Simonetta Longhi and Alita Nandi (ISER) Contact: slonghi and Week 1 Lecture 2: Data Management Use data on individual respondents from the first 17 waves of the British Household Panel Survey (BHPS). qindresp.dta The files are called aindresp.dta, bindresp.dta, 1. Merge files from the first two waves (aindresp.dta, and bindresp.dta) into wide format 2. Merge files from the first two waves (aindresp.dta, and bindresp.dta) into long format 3. Use loops to generalise the merging into long format to multiple waves 4. Declare the data to be a panel and compute transitions New Stata commands: - Combining waves: merge, append, renpfix - Inspecting the data: xtdes, xtsum - Panel data: tsset - Data manipulation: foreach, index(), bysort, L. - In-built Stata variable _N - Changing format from wide to long and from long to wide: reshape Merge Two Files into Wide Format Start your do file with the usual housekeeping (see the previous exercises). Upload the data for the second wave; sort the file for later matching on pid, save the file to be combined with wave 1, and type to reduce its size. use bhid pid bsex bmastat bage bqfachi blknbrd bpaygu /// using "$dirdata/bindresp", clear sort pid save bind_junk, replace The file is now ready to be merged with the file for the first wave. Repeat the same steps with the new file, and merge: use ahid pid asex amastat aage aqfachi alknbrd apaygu /// using "$dirdata/aindresp", clear sort pid merge 1:1 pid using bind_junk 1

2 There are different versions of the command merge, depending on the structure of the two files to be merged: One-to-one merge on specified key variables: merge 1:1 varlist using filename [, options] Many-to-one merge on specified key variables merge m:1 varlist using filename [, options] One-to-many merge on specified key variables merge 1:m varlist using filename [, options] Many-to-many merge on specified key variables merge m:m varlist using filename [, options] One-to-one merge by observation merge 1:1 _n using filename [, options] The merge command has generated a new variable. What is it? Look at the variable (tab _merge). It can assume three values: _merge == 1 these obs. were only in the master data (the file currently in memory, i.e. aindresp.dta) _merge == 2 these obs. were only in the using data (file being merged in, i.e. bind_junk.dta) _merge == 3 these obs. were in both master and using data (i.e. these people are in both waves) If we are interested in transitions across the two waves we can keep only those people present in both waves. How can we use this new variable for this purpose? Write the commands in your do file (and compare with the one at the end of this worksheet) Save the file after reducing its size. Remember to drop the variable _merge; a variable with the same name is created after each merge command. If another variable with the same name is already present, we would get an error message next time we want to combine files: tab _merge keep if _merge == 3 drop _merge save abind_wide, replace 2

3 Inspect the New Dataset and Compute Wage Changes Now describe the data to have a better look at the variables that are contained in the dataset; summarise the data to get information on means and sample sizes. Recode missing values in the variables apaygu and bpaygu Write the commands in your do file (and compare with the one at the end of this worksheet) Compute change in wages across the two waves; call the variable paych and label it. generate paych = bpaygu apaygu label var paych "Change in pay" How does the change in pay vary by sex? Save the file for later analyses. save abind_wide, replace Merge Two Files into Long Format Open wave 2 data file bindresp. Before being able to combine it with wave 1 we need some extra steps: 1. Generate a variable (wave) to identify which wave the observations are from. This is needed to distinguish between waves once the data are combined. 2. Rename all the variables by dropping the prefix b (there is no need to rename variable by variable, use the command renpfix instead). This is necessary so that each variable has the same name across waves and can be appended into a single column. (Note that this is not the case for all datasets. For example, in the German Socio-Economic Panel GSOEP the entire variable names, not only the prefixes, often change across waves.) use bhid pid bsex bmastat bage bqfachi blknbrd bpaygu /// using "$dirdata/bindresp", clear generate wave = 2 renpfix b 3

4 The command renpfix renames all the variables by dropping the prefix. For example, renpfix a would rename all the variables in the dataset whose name starts with a. Note that the command has no effect on the variable pid, since this does not have the wave prefix a : Original variable name (before renpfix a) ahid pid asex apaygu New variable name (after renpfix a) hid pid sex paygu Now save the file to be merged with wave 1. Repeat the same steps with the new file, before merging the two files. Write the commands in your do file (and compare with the one at the end of this worksheet) To combine the files into long format, use the command append. append using bind_junk Note that when there are variables with value labels, the labels retained are those of the master file. That is, the file that is in the working memory before appending, in this case aindresp. NB: In some cases labels vary across waves; consult Volume B of the BHPS user documentation for more information: We can now save the file after reducing its size: save abind_long, replace Inspect the New Dataset and Compute Wage Changes Again, describe the data to have a better look at the variables that are contained in the dataset; summarise the data to get information on means and sample sizes. Recode missing values in the variable paygu Write the commands in your do file (and compare with the one at the end of this worksheet) We can now declare the data to be a panel using the command tsset panelvar timevar. Where panelvar is the cross-wave identifier, in our case pid, while timevar is the time-series identifier, in our case the variable wave. The tsset 4

5 command tells Stata that the data has a panel structure, and allows us to use specific commands for time-series and panel data (e.g. all those starting with xt); it also allows proper treatment of gaps in panel and missing cases generally. tsset pid wave We can now describe (xtdes) and summarise (xtsum) the panel data. Compare different commands. What is the difference between describe and xtdes? What is the difference between sum and xtsum? NOTE: the command xtsum shows descriptive statistics for overall, between and within. The overall statistics refer to the whole sample; while the within statistics refer to each individual and to the variation from each individual s average (T-bar). If a variable does not change over time, its within standard deviation will be zero. The between statistics refer to individuals means and their variation. Now we can decide to keep only those present in both waves. The in-built Stata variable _N counts the number of observations in the bygroup. The command bysort repeats the Stata command on subsets of the data; in this case it repeats the command (_N) separately by pid. Hence, bysort pid: _N counts the number of observations with the same pid (here the maximum number is 2). Describe the data now: bysort pid: keep if _N == 2 xtdes xtsum Now we can compute changes in pay by using the operator L. which takes the lag of the variable following the dot. Hence, L.paygu means paygu t-1. If we want a twoperiods lag we can use L2.; there are also operators for computing leads (F.), first differences (D.) and seasonal differences (S.), which work in a similar way. Look at the Stata help for more details on the time series operators. gen paych = paygu L.paygu label var paych "Change in pay" How does the change in pay vary by sex? 5

6 Save the file for later analyses. save abind_long, replace Merge N Files into Long Format Generalisation to Multiple Waves The basic operations are the same as when we merge two waves. For each wave: Open the file; Rename all the variables by dropping the wave prefix; Generate a variable (wave) to identify the wave of this data; Save as a temporary file. To avoid rewriting the same commands for each wave, we can use loops: foreach wave in a b c d e f g h i j k l m n o p q { use `wave'hid pid `wave'sex `wave'mastat `wave'age /// `wave'qfachi `wave'paygu `wave'jbft /// `wave'region `wave'jbsoc `wave'jbbgy4 /// `wave'jbsemp `wave'jbstat /// `wave'cjsten /// using "$dirdata/`wave'indresp", clear renpfix `wave' // deal with wave p capture rename id pid gen wave = index("abcdefghijklmnopq","`wave'") save ind_junk`wave', replace } We want to repeat the series of commands for each of the 17 waves, each time substituting a different letter for `wave'. We use the command foreach to generate a loop which repeats the commands between {} for each element in the foreach list (a b... q). Note that the opening curly bracket must be on the same line as foreach; while the closing curly bracket must be in a line on its own. foreach creates a local macro, which you can name as you wish. We have called it `wave'. The content of the local macro is specified by the foreach list, which can be specified in different ways. In this specific case the list is a general one. Elements are separated from each other by one or more blanks: a b c d e f g h i j k l m n o p q. The contents of the local macro changes each time we go through the loop. The first time through, the content equals the first element in the foreach list (a). The second time through it contains the second element (b), and so on. To refer to the content of the local macro, we enclose its name in a left quote and apostrophe: `wave'. Note that pid never has a wave prefix. This generates a problem for wave p where renpfix would rename pid to id. If we simply specified rename id pid, we would get an error message which would terminate the execution of the do-file, for all waves in which Stata would not find a variable called id (i.e. all waves other than wave p). 6

7 Using capture tells Stata to swallow the error message and continue executing the do-file. Alternatively, we could use the command index("p","`wave'") to identify when the local variable `wave' has value p. The command identifies the position in the string p that is equal to the value in `wave'. Because the string is of only one character, the position will be 1. In this case the logical statement has value 1 when wave = p and 0 otherwise. When the return code is 1, we rename the variable id back to pid. Alternatively, we can reach the same result using an if statement: if "`wave'" == "p" rename id pid] We also use the index() function to generate the numerical variable identifying the wave: gen wave = index("abcdefghijklmnopq","`wave'"). The new variable wave will have value 1 when the local macro `wave' is equal to a; value 3 when it is equal to c, and so on. Note that some sets of questions are asked intermittently, hence, some variables might not be present in all waves. The easiest way to solve the problem consists in uploading the whole dataset (i.e. upload all variables), append the different waves, and then drop the variables you are not going to use for the analysis. Once the data for each wave have been extracted, prepared and saved, use a similar loop to append all waves. Note that the wave q file is still open. Append all other files to that: foreach wave in a b c d e f g h i j k l m n o p { append using ind_junk`wave'... } We should now recode all BHPS missing (i.e. all negative values) into Stata missing. We can either recode each variable separately, or use a similar version of the foreach loop: foreach var of varlist sex age region qfachi jbft jbstat /// paygu jbbgy4 jbsoc jbsemp mastat { recode `var' -9/-1 =. } Once again, we want to repeat the same commands for each variable in the foreach list. In this case the list contains the name of the variables for which we want to repeat the same command (sex paygu mastat...). Note that in this case we have chosen to call the local macro var. Note the difference between the previous and the current loop. In the first case we type foreach wave in; in the second case we type foreach var of varlist. foreach.. in is a general list type, which can be used with a list of variables, numbers or other types of elements. The advantage of using a specified list type, in this case a variable list, is that we can use shorthand conventions. For example, we could type foreach var of var1 var10, and Stata would understand this as including all variables between the two, in the order in which they appear in the data 7

8 set. Alternatively, we could type foreach var of Var*, which would include all variables starting with the three letters Var. Similarly, we could use the command forvalues with shorthand conventions for numerical lists (see help forvalues). Inspect the data, sort the file and save it: sum describe sort pid wave save ind_long, replace The creation of the data set is now complete. Inspect the New Dataset Now we can declare the data to be a panel, and inspect it using the appropriate commands (xtdes, xtsum). Also try the commands: iis and tis. Write the commands in your do file (and compare with the one at the end of this worksheet) iis // Reminds you which is // the variable with the person identifier Tis // Reminds you which is the variable // with the wave/time identifier When merging data from two files, we kept only those respondents who were present in both waves (we used a balanced panel). Here we could do the same: bysort pid: keep if _N == 17. However, in this case only a small proportion of people would remain in the dataset, and these are likely to be a selected sample. Every year there are new entrants in the BHPS, plus there have been extension samples (e.g. additional households from Scotland and Wales in 1999, and from Northern Ireland in 2001). Every year people drop out of the survey for various reasons, a phenomenon called panel attrition. Hence, it is better to retain all observations and select cases at the analysis stage (the variable creation commands usually are ok since transition variables are set to missing if a person is not present in at least one of the two consecutive waves). Compute Changes in Marital Status, and Wage Changes Inspect the variables mastat: a small number of cases are aged < 16 (code 0) and need to be excluded. For simplicity, also recode marital status into smaller number of categories (married, living as a couple or civil partnership = 1; widowed, divorced or separated = 2; never married = 3), generating the new variable ma. 8

9 Write the commands in your do file (and compare with the one at the end of this worksheet) Now generate a new variable which summarizes different transitions across marital statuses (below we call this variable mach) by creating a 2-digit summary of marital change. gen mach = (10*ma) + L.ma tab mach gen mach = (10*ma) + L.ma is a useful trick that we often use when we want to concatenate two numeric variables. Attach a short description to the variable using the command label var, then make the variable more accessible by labelling its values using the command label value label var mach "marital change" label define machlab 11 "stayed in couple" /// 12 "partnership ended" /// 13 "partnered -> never married" /// 21 "ex-partner -> partnership" /// 22 "stayed ex-partner" /// 23 "ex-partner -> never married" /// 31 "never married -> partnership" /// 32 "never married -> ex-partner" /// 33 "stayed never married" label value mach machlab tab mach list mach in 1/20, sepby(ahid) Note the difference between labelling a variable: label var mach "marital change" and labelling its values, after having defined the labels of each value: label define machlab 11 "stayed in couple" label value mach machlab We can now analyse transitions using the command xttrans. How does marriage vary over time and by sex? Now generate the variable paych, which measures the change in a person s earnings from one wave to the next. How does the change in pay vary by sex? 9

10 How does the change in pay vary over time by type of marital change and sex? 10

11 OPTIONAL From Wide to Long Format, and from Long to Wide Format Usually it is best to derive the data set in the format we need it in, but it is possible to convert files between long and wide format. For this part of the exercise we go back to the two files including only waves a and b. We can convert from long to wide format and vice versa by using the command reshape. Open the long file abind_long you saved previously, and reshape it into side format: use abind_long, clear reshape wide hid lknbrd sex mastat age qfachi paygu paych, /// i(pid) j(wave) sum save abind_converted, replace What happens to those variables with missing values in one wave? Convert from wide to long format by using a slightly different version of the command reshape. (Open the file in wide format if not already open) use abind_wide, clear capture noisily reshape long hid sex pagyu mastat lknbrd /// ma mach paych, i(pid) j(wave) The command is not successful (capture prevents the program from stopping; noisily requests that the error message which would otherwise be swallowed by capture be displayed). In this specific case reshaping does not work because the identifiers of waves are letters rather than numbers. To make the reshape command work we would need to first rename all the variables, e.g. ahid hid1; bhid hid2 Alternatively, try converting the file abind_converted from wide back to long format. Write the commands in your do file (and compare with the one at the end of this worksheet) Clean up delete temporary file no longer needed and close the log file, and close the log file: clear erase bind_junk.dta erase abind_long.dta erase abind_converted.dta log close 11

12 Do File version 11 clear all set more off capture log close set memory 20m global dir1 "\\iserhome\conferencedata\final" global dirresults "M:" log using "$dirresults\exercise2.log", replace ******************************************************************* * Merge files from the first two waves (aindresp.dta, bindresp.dta) * into wide format ******************************************************************* * Prepare file for wave 2 use bhid pid bsex bmastat bage bqfachi blknbrd bpaygu /// using "$dirdata\bindresp", clear sort pid save "$dirresults\bind_junk", replace * Merge with file for wave 1 use ahid pid asex amastat aage aqfachi alknbrd apaygu /// using "$dirdata\aindresp", clear sort pid merge 1:1 pid using "$dirresults\bind_junk" tab _merge * If we are interested in transitions, keep only those present * in both waves keep if _merge == 3 drop _merge * Need to drop the "_merge" variable because a variable * with the same name is created after each 'merge' command save "$dirresults\abind_wide", replace * Describe the data describe sum * Recode missing values and compute wage changes sum apaygu bpaygu recode apaygu -9/-1 =. recode bpaygu -9/-1 =. generate paych = bpaygu - apaygu label var paych "Change in pay" sum paych * How does the change in pay vary by sex? tab asex, sum(paych) 12

13 save "$dirresults\abind_wide", replace ******************************************************************* * Merge files from the first two waves (aindresp.dta, bindresp.dta) * into long format ******************************************************************* use bhid pid bsex bmastat bage bqfachi blknbrd bpaygu /// using "$dirdata\bindresp", clear generate wave = 2 renpfix b save "$dirresults\bind_junk", replace use ahid pid asex amastat aage aqfachi alknbrd apaygu /// using "$dirdata\aindresp", clear generate wave = 1 renpfix a append using "$dirresults\bind_junk" save "$dirresults\abind_long", replace * Describe the data describe sum * Recode missing values and compute wage changes sum paygu recode paygu -9/-1 =. * Declare the data to be panel tsset pid wave describe xtdes sum xtsum * Compute wage changes gen paych = paygu - L.paygu label var paych "Change in pay" * How does the change in pay vary by sex? tab sex, sum(paych) * Analysis results should be the same as for WIDE case save "$dirresults\abind_long", replace ******************************************************************* * Merge files into long format, generalisation to multiple waves ******************************************************************* foreach wave in a b c d e f g h i j k l m n o p q { use `wave'hid pid `wave'sex `wave'mastat `wave'age /// 13

14 `wave'qfachi `wave'paygu /// `wave'jbft `wave'region `wave'jbsoc `wave'jbbgy4 /// `wave'jbsemp `wave'jbstat `wave'cjsten /// using "$dirdata/`wave'indresp", clear // note that pid does not have the `wave' prefix renpfix `wave' // In wave "p" renpfix `wave' renames "pid" as "id" and a new variable called "id" is generated // Solution: rename the variable before the append. Alternatively combine id // and pid later if index("p","`wave'") rename id pid gen wave = index("abcdefghijklmnopq","`wave'") save "$dirresults\ind_junk`wave'", replace } foreach wave in a b c d e f g h i j k l m n o p { append using "$dirresults\ind_junk`wave'" } * Recode missing values foreach var of varlist sex age region qfachi jbft jbstat paygu /// jbbgy4 jbsoc jbsemp mastat { recode `var' -9/-1 =. } sum describe sort pid wave save "$dirresults\finalfile", replace tsset pid wave xtsum xtdes iis * Reminds you which is the variable with the person identifier tis * Reminds you which is the variable with the wave/time identifier * Compute Changes in Marital Status, and Wage Changes * tab mastat recode mastat (0 98 =.) (1 2 7 = 1) (3/5 = 2) (6 = 3), gen(ma) gen mach = (10*ma) + L.ma tab mach label var mach "marital change" label define machlab 11 "stayed in couple" /// 12 "partnership ended" /// 13 "partnered -> never married" /// 21 "ex-partner -> partnership" /// 22 "stayed ex-partner" /// 23 "ex-partner -> never married" /// 31 "never married -> partnership" /// 32 "never married -> ex-partner" /// 33 "stayed never married" label value mach machlab 14

15 tab mach list mach in 1/20, sepby(hid) xttrans mach * change in pay * recode paygu -9/-1 =. gen paych = paygu - L.paygu label var paych "Change in pay" * How does the change in income vary by sex? tab sex, sum(paych) * Income change over time, by type of marital change and sex table mach sex, contents(mean paych n paych) format(%9.2f) **************************************** * From Wide to Long Format, and * from Long to Wide Format **************************************** * Converting from LONG to WIDE format use "$dirresults\abind_long", clear reshape wide hid lknbrd sex mastat age qfachi paygu paych, /// i(pid) j(wave) sum save "$dirresults\abind_converted", replace * Converting from WIDE to LONG format use "$dirresults\abind_wide", clear capture noisily reshape long hid sex pagyu mastat lknbrd /// ma mach paych, i(pid) j(wave) // This command does not work in this specific case // because the identifiers of waves are letters rather than numbers // Need to first rename the variables, // e.g. ahid --> hid1; bhid --> hid2 *... but we can reconvert the abind_converted file from * wide back into long format; use "$dirresults\abind_converted", clear reshape long hid sex paygu lknbrd mastat age qfachi paych, /// i(pid) j(wave) * Clean up: delete temporary file no longer needed ********************************************* clear erase "$dirresults\abind_converted.dta" erase "$dirresults\abind_long.dta" erase "$dirresults\abind_wide.dta" 15

16 erase "$dirresults\bind_junk.dta" foreach wave in a b c d e f g h i j k l m n o p q { erase "$dirresults\ind_junk`wave'.dta" } log close 16