Exploring Maps with ggplot2 1 Draft: Do not Cite

Transcription

1 Exploring Maps with ggplot2 1 Draft: Do not Cite John Kane and William H. Rogers May 12, We thank Josh Ulrich for his helpful comments and corrections. All remaining errors are the those of the authors. Introduction In this Exploring Series I demonstrate how to use and present spatial data with ggplot2. You will need a copy of R, and a copy of the Saint Louis County zip code shape files. Before you start, unzip the shape files and place them in a folder, then set your R directory to that folder using the getwd() function. Please let me know if you have any questions or comments, because this document is a work in progress. You can find extensive documentation at Hadley Wickham s ggplot2 webpage. Keep in mind that there are two ways in which maps (or any twodimensional data) can be presented: vector or raster. Vector maps are a collection of points connected by lines that make up each shape. Raster maps are a collection of pixels (colored boxes) in which the information is stored. This Exploring Series focuses on vector maps. Example 1 Plotting Maps: States Preliminary In the first example you will learn how to plot a simple map from the maps package using the ggplot2 package. You will need the following packages for this example: maps, mapproj, and ggplot2. Click here and here for two related examples. Map First you need to load the map. We will use a map of the United States from the maps package. To load the map, type: > MyMap <- map_data("state") which will install the state map in a map format. The function map_data() is a ggplot2 function that converts map objects into data frames. Type?map_data() after you have called ggplot2, library(ggplot2), to see the documentation. To look at the data structure use the str() function. 2 (Not shown.) > str(mymap) Notice the number of observations. There are only 49 locations in this map, the lower 48 plus the District of Columbia, but this data 2 str() is a function that provides basic information about the structure of an object - any object. When working on a project you will make extensive use of str(); however, by the end of you project the structure command should not be located in the script, becuase the computer doesn t need it to run.

2 exploring maps with ggplot2 draft: do not cite 2 frame has 15,537 observations. Each observation refers to a point in space that makes up the shape of each state. Use the head() function to view the first 6 rows. 3 > head(mymap) long lat group order region subregion alabama <NA> alabama <NA> alabama <NA> alabama <NA> alabama <NA> alabama <NA> Notice that Alabama is in group 1 and each observation has a different combination of long and lat (longitude and latitude). Now look at the number of points per state for the first 5 states. 4 > head(aggregate(mymap[, 1:2], list(mymap[, 5]), length)) Group.1 long lat 1 alabama arizona arkansas california colorado connecticut head(mymap) is equivalent to MyMap[1:6, ]. The sister function is called tail() and it lists the last six rows. 4 I m using the aggregate() function with the first two columns of data, MyMap[, 1:2], grouped by the fifth column which lists the state names, aggregated with the length function. Colorado has only 79 points because it is a simple shape, while California, a complicated shape, has 516 points. To plot the map you could use the plot() function, (not shown) > plot(mymap$long, MyMap$lat) but it s ugly and the United States doesn t look wide enough. So, you could change the plot options from points to lines as follows. (Not shown.) > plot(mymap$long, MyMap$lat, type = "l") That s even worse! The lines are drawn correctly but there are a bunch of additional lines. The additional lines are there because the computer plays connectthe-dots without knowledge about the shapes we want. Notice the line from the bottom of Alabama that goes to the western part of Arizona, and then to the northwestern part of Arkansas? What we need to do is draw each state individually based on the group variable. More on this point below. In the meantime just use the following code. > map("state")

3 exploring maps with ggplot2 draft: do not cite 3 Figure 1: State Map Data Now that we have a map, let s add data to it by collecting the 1977 state data from state.x77. The dataset state.x77 is pre-loaded on every version of R. To call up the data, type the following: > head(state.x77) Population Income Illiteracy Life Exp Murder HS Grad Frost Area Alabama Alaska Arizona Arkansas California Colorado Notice that there is no column name above the state names. That s because the state names are not in a column, they re actually row names of the matrix called state.x77. A matrix can only have one atomic variable type (e.g all numbers, all characters, etc.) In fact, the variable names are actually column names. We want the state.x77 data to be in the object class called a data frame. Data frames allow users to have multiple atomic types in the same object. A data frame is useful in our case because we will add the state, region, and division names to the map. To convert the matrix to a data frame use the as.data.frame() function. > MyState <- as.data.frame(state.x77)

4 exploring maps with ggplot2 draft: do not cite 4 Now add three new columns for the state, region, and division names, then view the results (not shown here). > MyState$State <- rownames(mystate) > MyState$Region <- state.region > MyState$Division <- state.division > head(mystate) The objects state.region and state.division are vectors with the names of the Census regions and division in the appropriate order. In other words, the computer does not know that Alabama is in the South Region. The names in the vector state.region are ordered appropriately such that South is in the first position. You need to read the state documentation to know that everything is ordered correclty. To view the documentation type >?state. Merge Everything is where we want it. Now we just need to combine the MyState data with MyMap. We cannot simply use $ or cbind() to combine the data because MyStateand MyMap have a different number of rows, plus MyMap includes D.C. and MyState includes Alaska and Hawaii. We need to use the function merge() (or ggplot2 s join() function), with some identifier that is common to both data frames. State name will work, but R is case sensitive and MyMap s state names are all lowercase, while MyState s state names are uppercase in the first letter. Not to worry. You can use the tolower() function to make all the MyState s state names lower case. 5 However, I don t like lowercase names. So, let s change the first letter of every name in MyMap to uppercase. To do this you need the gsub() function plus a little knowledge of Perl expressions. The gsub() function stands for global substitution, and it will replace every selected pattern with another: very handy when working with string data. (No addition sofeware is needed.) Perl, on the other hand, expressions are cryptic but useful. Type the following. 5 When you add the state, region and division names, all you d need to do is add a single line: >MyState$State <- tolower(mystate$state) > names(mymap) <- gsub("\\b(\\w)", "\\U\\1", names(mymap), perl = T) The names() function calls all the names in MyMap. The first term in gsub() is Perl language for select the first character at the beginning of each word. The second term says change the first backreferenced character to uppercase. The third term, names(mymap), lists the names to be changed; I want them all changed. The final term tells R that you are using Perl expressions. 6 6 You can find Perl documentation here. Don t panic! You don t need to be fluent in Perl to work with it, just as you don t need to be fluent in Spanish to visit a Tex-Mex restaurant.

5 exploring maps with ggplot2 draft: do not cite 5 Now do the same to the state names, but first change the variable name Region to State. You can use the rename() function provided by the package reshape, which is loaded every time you call ggplot2 (i.e. it s already loaded). Type the following. > MyMap <- rename(mymap, c(region = "State")) > MyMap$State <- gsub("\\b(\\w)", "\\U\\1", MyMap$State, perl = T) Now we can merge. > MyMap <- merge(mymap, MyState, by = "State") > MyMap <- MyMap[order(MyMap$Order), ] The merge() function reorders the rows, so you need to resort the order. The second action above reorders the coordinates. Alternatively, you could use join(), which does not change the order, so the second step is not necessary. 7 Plot Now that the data and map are arranged, it s time to plot using ggplot2. Try the following code: 7 This isn t actually all that different from merge. Just use the following line in place of the two above: >MyMap <- join(mymap, MyState, by= State ) > p <- ggplot(mymap, aes(long, Lat, group = Group)) > p <- p + geom_polygon(aes(fill = Division), colour = alpha("white", + 1/2), size = 0.2) > p <- p + coord_map(project = "lagrange") > p <- p + scale_x_continuous(breaks = NA) + scale_y_continuous(breaks = NA) > p <- p + labs(x = "", y = "", fill = "Census Division") > print(p) Here s what each step means by line of code: 1. The ggplot() function is called using MyMap as a data source. The aes component identifies the x and y coordinates (Long, Lat), and all the points are grouped by Group. (Remember, each state must be plotted separately.) 2. geom_polygon() determines that the graph is a set of polygons. The aes component fills each polygon with a color that refers to the variable Division. Everything outside of the aes component deals with the polygon borders. The first term (colour) says the border will be white and slightly transparent. The second term (size) says the border will be slightly larger than the default. Play around with these two options. 3. coord_map() converts the Long and Lat to values that can be plotted on a flat plane. If you skip this step the map will look to thin

6 exploring maps with ggplot2 draft: do not cite 6 Figure 2: Census Divisions Census Division New England Middle Atlantic South Atlantic East South Central West South Central East North Central West North Central Mountain Pacific (try it). There are other conversion options - review the documentation. 4. scale_x_continuous() changes the x scale, but using the breaks= NA option mean no x ticks or values will be plotted. The x and y values are not useful when looking at the states. 5. labs() changes the labels for everything. I removed the x and y labels, and changed the fill name from Division to Census Division. 6. print(p) does just that. Try another example. Here the only major change is in the fill and scale. The scale_fill_continuous() breaks the colors into levels useful for a continuous variable like income per capita. Notice how I picked the high and low color and value. And Notice the legend. The \n term is expression that means new line, which is common to most non-windows based systems. If you remove the \n term, then the legend will say Income Per Capita all in one line, which will make the legend wider. > p <- ggplot(mymap, aes(long, Lat, group = Group)) > p <- p + geom_polygon(aes(fill = Income), colour = alpha("white", + 1/2), size = 0.2) > p <- p + scale_fill_continuous(low = "black", high = "green", + limits = c(min(mymap$income), max(mymap$income))) > p <- p + coord_map(project = "lagrange")

7 exploring maps with ggplot2 draft: do not cite 7 > p <- p + scale_x_continuous(breaks = NA) + scale_y_continuous(breaks = NA) > p <- p + labs(x = "", y = "", fill = "Income \nper Capita") > print(p) Figure 3: 1977 Income Per Capita by State Income Per Capita Example 2 Plotting Maps: County Preliminary In this example you will learn how to plot a simple map from the maps package in ggplot2, like the last example, but this example focuses on more complicated data manipulation. The goal is to map the per capita income by county as a percent of national income for the year The biggest challenge will be merging the map data with the BEA data. You may find the data processing work very time comsuming, but that s mearly the nature of emerpical work. Remember, after any step you can use the str() function to view what has been created. Click here and here for two related examples. The following code will largely follow Google s R Style Guide. Style guides are important in maintaining the readability of your code. Many open source programs are used by Google programmers including R and Python, so they have developed many useful conventions. These conventions are not ironclad, just useful. Think consistency. You will need the following libraries for this example: maps, mapproj, and ggplot2. 8 You can download the data from the Bureau of Economic Analysis by selecting from the Single Line data. Then choose the CA1 3 - Personal Income Summary serices and the Per capita personal income, percent of US line.

8 exploring maps with ggplot2 draft: do not cite 8 Map You will use a map of the United States from the maps package as follows. > MyMap <- map_data("county") > MyMap2 <- map_data("state") You can view the data in MyMap using the str() function, where you ll notice the state and county names are in two columns and all lower case. To start, change the last two variable names to something more useful using the rename() function from the reshape package. > MyMap <- rename(mymap, c(region = "state.name", subregion = "county.name")) Next, you need to add state abbreviations in anticipation of the BEA data, which includes county names and state abbreviations. Instead of typing in the abbreviations you can use the state.x77 data from the prevoius example. The state.x77 data is loaded on the factory fresh version of R, and therefore convenient. Furthermore, there is nothing you need to load. Simply call state.abb for the abbreviaions and state.name for the full state name. To match the state abbreviation with the state name you ll use the match() function, and type the following. > MyMap$state <- state.abb[match(mymap$state.name, tolower(state.name))] Data The BEA data is a little complicated because it has notes in the last four or five lines of every downloaded dataset, and the county and state abbreviations are in the same column. To deal with the first problem you could open the BEA data in Excel, find the first row number that contains the notes, and use the nrows option in read.csv. 9 This approach is quick and easy if you plan to only work with two or three BEA datasets. However, it pays to have R do most of this work if you have many datasets. This example on requires one BEA file, but this is a teaching document so here is one approach. 1. Load the data and set the colclasses option to character. By doing so, R will read everything as text. I inlcuded the extra step of creating an object called bea.file. I do this because I keep my data in a separate directory from my scripts. If you data is in the same directory, then you can skip the extra step and type Bea <- read.csv( BeaPctIncPcCnty2008.csv, colclasses= character ). 9 For example you could open the BEA file and notice that the notes start on line 3136 (not counting the header line). Thus you could import as follows: Bea <- read.csv( BeaPctIncPcCnty2008.csv, nrows= 3135).

9 exploring maps with ggplot2 draft: do not cite 9 > bea.file <- "K:/WHR/Work/Research/Learning/Data/BeaPctIncPcCnty2008.csv" > Bea <- read.csv(bea.file, colclasses = "character") > Bea <- rename(bea, c(x2008 = "Pct.Pc.Inc")) You should have all character rows. Type tail(bea.dat) to see the last five rows. Notice the BEA notes. You want to remove the last rows with all the notes. I think all the BEA files start their comments with Source: so you can take advantage of that. 2. Use the function grep() to identify the row number with a word that has Source in it. > drop.row <- grep("source", Bea[, 1]) > Bea <- Bea[-(drop.row:nrow(Bea)), -1] The last above command uses the minus sign to remove the last few rows and the first column, which all contain information we don t need. 3. Next you want to remove the state names from the county names. You can use the strplit() function to split the text by,. > bea.name.split <- strsplit(bea$areaname, split = ", ") Type head(bea.name.split) to see the first five items in the list. 4. You want two columns with state abbreviation and county names to match the map data frame you created above. To do this use the lapply() and unlist() functions. lappy() applies a function over each vector in a list. You can use an anonymous fucntion that selects the first element out of each vector in the list. Finally, the unlist() function converts a list into a vector. > Bea$county.name <- unlist(lapply(bea.name.split, function(x) x[1])) > Bea$county.name <- tolower(bea$county.name) > Bea$state <- unlist(lapply(bea.name.split, function(x) x[2])) 5. Finally, you want to change the numeric from character to integer. The (suppresswarnings()) function is present to remove an error that we aren t worried about in this instance: when converting data from one type to another, these functions place an entry of NA where they are unable to convert. We don t need to actually see that each time it happens in this example. > Bea$Pct.Pc.Inc <- suppresswarnings(as.integer(bea$pct.pc.inc)) Merge > MyMap <- join(mymap, Bea, by = c("state", "county.name"))

10 exploring maps with ggplot2 draft: do not cite 10 Plot > p <- ggplot(mymap, aes(long, lat, group = group)) + geom_polygon(aes(fill = Pct.Pc.Inc)) + + geom_polygon(data = MyMap2, colour = "white", fill = NA, + size = 0.2) + coord_map(project = "lagrange") + scale_x_continuous(breaks = NA) + + scale_y_continuous(breaks = NA) + labs(x = "", y = "", fill = "Percent of US") > print(p) Percent of US Figure 4: County Per Capita Income as a Percent of U.S. Mean Example 3 Plotting Point Data Preliminary In this example you will learn how to plot point data in ggplot2. You will need the following libraries for this example: ggplot2, hexbin, MASS. Map There is no map for this example. Data You will need a file called SaleSampA.csv. In the future you will be able to download this data from my website. For now, send me an if you don t have it. (I used a similar dataset in my Econ 4900 and 5900 intercession course.)

11 exploring maps with ggplot2 draft: do not cite 11 The file SaleSampA.csv is a comma delimited file containing information about housing sales in northern Saint Louis County for I created an object called file to locate this file, but I you have your copy in the R working directory, then you don t need to call the full address. The funtion read.csv() has defalts to quickly deal with csv files as they are normally formated. All options on read.csv() can be changed as in read.table(). Try the following code. > file <- "K:/WHR/Work/Research/Learning/Data/SalesSampA.csv" > stl.hsg <- read.csv(file) > stl.hsg <- transform(stl.hsg, sale.dt = as.date(sale.dt)) After read.csv() the functions transform(), subset(), and rename() are used. Their purpose is to convert the SaleDt column to a Date class, limit the dataset to sales in 2001, and rename the coordinate columns to something more meaningful. Merge No merging is nessary in this example. Plot Since the data has coordinates, it is quite easy to plot the points. Try the following code. (Plot not shown.) > ggplot(data = stl.hsg, aes(long, lat)) + geom_point(colour = "cornflowerblue") The plot is fine, but it includes the coordinate values, which are not useful when presenting a graph. To clean up the plot, you can use scale_x_continuous(), and labs(). To add a little more information to the points use aes(colour= ). (Plot not shown.) > ggplot(data = stl.hsg, aes(long, lat)) + geom_point(aes(colour = log(price/sq.ft))) + + scale_x_continuous(breaks = NA) + scale_y_continuous(breaks = NA) + + labs(x = "", y = "", colour = "Log of Price\nper Sq.Ft.") Now the plot looks a little cleaner, and it provides more useful data. It is still missing some refrence information (e.g. county boundaries, interstates, rivers, etc.) We will cover reference material in the next example. For now, you may be interested in summarizing the spatial data. One way to visualize the sales density in an area is to use the geom_hex() function, which requires hexbin. The function is easy to add; see below. (Plot not shown.) > ggplot(data = stl.hsg, aes(long, lat)) + geom_hex() + scale_x_continuous(breaks = NA) + + scale_y_continuous(breaks = NA) + labs(x = "", y = "")

12 exploring maps with ggplot2 draft: do not cite 12 The function geom_density2d() is a useful alternative to geom_hex(). geom_density2d() can be used to create hot-spot or heat grachics as well as data contours. Below are two examples that demonstrate the contour graphic without and with the point data. (Plot not shown.) > ggplot(data = stl.hsg, aes(i(long/5280), I(lat/5280))) + geom_density2d(aes(colour =..level..), + size = 1) + scale_colour_gradient2(low = "darkblue", mid = "orange", + high = "red", midpoint = 0.015, limits = c(0.005, 0.025)) + + scale_x_continuous(breaks = NA) + scale_y_continuous(breaks = NA) + + labs(x = "", y = "") > ggplot(data = stl.hsg, aes(i(long/5280), I(lat/5280))) + geom_point() + + geom_density2d(aes(colour =..level..), size = 1) + scale_colour_gradient2(low = "darkblue", + mid = "orange", high = "red", midpoint = 0.015, limits = c(0.005, )) + scale_x_continuous(breaks = NA) + scale_y_continuous(breaks = NA) + + labs(x = "", y = "") In the above two code chunks, I have rescaled the coordinates in order to make the output values more readable in the key. I have also added an adjustment to the color scheme that is more pleseant to my eye. Of course, you can change it as you wish. The aesthetic value colour=..level.. colors the contour curves based on the value of the level. If you removed that part of the code so that the line read geom_density2d() your contour curves would be black, or red if you wish: try geom_density2d(colour= "red") and remove the scale_fill_gradient2() line. In order to visualize a hot-spot image you only need to change the aesthetic input and two other geom_density2d() options. > ggplot(data = stl.hsg, aes(i(long/5280), I(lat/5280))) + stat_density2d(aes(fill =..density..), + geom = "tile", contour = F) + scale_fill_gradient2(low = "darkblue", + mid = "orange", high = "red", midpoint = , limits = c(0.005, )) + scale_x_continuous(breaks = NA) + scale_y_continuous(breaks = NA) + + labs(x = "", y = "") Example 4 Plotting Shapefiles Preliminary Download Tiger files. Install packages. > library(rgdal) > library(maptools)

13 exploring maps with ggplot2 draft: do not cite 13 Figure 5: North Saint Louis County Real Estate Sales Hot Spots density

14 exploring maps with ggplot2 draft: do not cite 14 Map Read the county shapefiles. > shp.loc <- "K:/WHR/Work/Research/Shape/Tiger10" > shp.name <- "tl_2010_29_county10" > map.mo <- readogr(paste(shp.loc, shp.name, sep = "/"), shp.name) OGR data source with driver: ESRI Shapefile Source: "K:/WHR/Work/Research/Shape/Tiger10/tl_2010_29_county10", layer: "tl_2010_29_county10" with 115 features and 17 fields Feature type: wkbpolygon with 2 dimensions > summary(map.mo) > str(map.mo) > plot(map.mo) > keep.vars <- c("geoid10", "NAMELSAD10", "CSAFP10", "CBSAFP10") > map.mo <- map.mo[, keep.vars] Data Merge Plot Example 5 Plotting Spatial Data on Google Maps Preliminary Map Data Merge Plot