Alyssa Grieco. Data Wrangling Final Project Report Fall 2016 Dangerous Dogs and Off-leash Areas in Austin Housing Market Zip Codes.

Size: px

Start display at page:

Download "Alyssa Grieco. Data Wrangling Final Project Report Fall 2016 Dangerous Dogs and Off-leash Areas in Austin Housing Market Zip Codes."

Milton Potter
5 years ago
Views:

1 Alyssa Grieco Data Wrangling Final Project Report Fall 2016 Dangerous Dogs and Off-leash Areas in Austin Housing Market Zip Codes Workflow

2 Datasets Data was taken from three sources on data.austintexas.gov. One was data about the housing market in Austin, Texas in 2014 incorporating median household income, median rent and median home value for every zip code across the city:

3 Another dataset showed the address, zip code and owner of dangerous dogs throughout Austin:

could all be exported as CSVs, which I was able to easily export to the server, I created a

4 The third dataset had the name and address, along with a lot of extraneous detail of off-leash areas across Austin: Database After finding and looking at these datasets and seeing that they could all be exported as CSVs, which I was able to easily export to the server, I created a database ER diagram for a database linking all of these datasets and created that database on phpmyadmin.

5 Importing Data to Database Once the data was created and all three tables were ready to recieve data, I wrote python code to clean up the data in the housing market csv so that dollar signs and commas were no longer in the median income, rent and home value data, wrote python code to pull out the zip code from the table full of description of the off leash areas, and wrote python code to import the cleaned up data from all three datasets to their respective tables in the database using a mysql query within the code. The code is as follows: (and is also on the server, here sftp://agrieco:@holden.ischool.utexas.edu//export/home/u16/agrieco/project/project_code.py) import pymysql #allows me to use mysql within the python code import pprint import csv import re connection = pymysql.connect(host="localhost", # your host, usually localhost user="grieco_a", # your username passwd="arg985", # your password db="grieco_a_project_dangerous_dogs", # name of the db autocommit=true, # removes a step in queries cursorclass=pymysql.cursors.dictcursor) cursor = connection.cursor() cursor.execute("truncate dangerous_dogs; TRUNCATE housing_market; TRUNCATE off_leash_area") with open('housing_market.csv') as csvfile: mycsvreader = csv.dictreader(csvfile, delimiter=",", quotechar='"') for row in mycsvreader: raw_income = row["median_household_income"] no_sign_income = re.sub('\$', '', raw_income) #gets rid of dollar signs no_comma_income = re.sub(',', '', no_sign_income) #gets rid of commas raw_rent = row["median_rent"] no_sign_rent = re.sub('\$', '', raw_rent) #gets rid of dollar signs

6 no_comma_rent = re.sub(',', '', no_sign_rent) #gets rid of commas raw_value = row["median_home_value"] no_sign_value = re.sub('\$', '', raw_value) #gets rid of dollar signs no_comma_value = re.sub(',', '', no_sign_value) #gets rid of commas # pprint.pprint(row) sql_placeholder = """ INSERT INTO housing_market(zip_code,median_household_income,median_rent,median_home_value) VALUE (%(Zip_Code)s,%(Median_household_income)s,%(Median_rent)s,%(Median_home_value)s) """ #mysql to insert the columns from the housing market csv into the appropriate #columns in the correct table in the database param_dict = {'Zip_Code': row['zip_code'], 'Median_household_income': no_comma_income, 'Median_rent': no_comma_rent, 'Median_home_value': no_comma_value} #allows me to import the cleaned data #print(param_dict) cursor.execute(sql_placeholder, param_dict) import pymysql import pprint import csv connection = pymysql.connect(host="localhost", # your host, usually localhost user="grieco_a", # your username passwd="arg985", # your password db="grieco_a_project_dangerous_dogs", # name of the db autocommit=true, # removes a step in queries cursorclass=pymysql.cursors.dictcursor) cursor = connection.cursor()

7 with open('map_of_declared_dangerous_dogs.csv') as csvfile: mycsvreader = csv.dictreader(csvfile, delimiter=",", quotechar='"') for row in mycsvreader: # pprint.pprint(row) sql_placeholder = "INSERT INTO dangerous_dogs (First_Name,Address,Zip_Code) VALUES (%(First_Name)s,%(Address)s,%(Zip_Code)s)"; #this one needed no cleaning so I could insert the data straight into the table without creating a paramdict cursor.execute(sql_placeholder, row) import pymysql import pprint import csv import re connection = pymysql.connect(host="localhost", # your host, usually localhost user="grieco_a", # your username passwd="arg985", # your password db="grieco_a_project_dangerous_dogs", # name of the db autocommit=true, # removes a step in queries cursorclass=pymysql.cursors.dictcursor) cursor = connection.cursor() with open('off_leash_areas.csv') as csvfile: mycsvreader = csv.dictreader(csvfile, delimiter=",", quotechar='"') for row in mycsvreader:

8 #pprint.pprint(row) print("-"*20) raw_address = row["address"] print(raw_address) #remove tags <.+> no_tags_address = re.sub('<[^>]+>',' ', raw_address) no_tags_address = re.sub('<.*$','', no_tags_address) print(no_tags_address) matches = re.search('(\d\d\d\d\d)',no_tags_address) #finds only 5 numbers in a row to pull out just the zip codes myzip = matches.group(1) #tells it to only print the matches of 5 numbers in a row print(myzip) param_dict = {'Name': row['name'], 'Address': no_tags_address, 'Zip_Code': myzip} sql_placeholder = "INSERT INTO off_leash_area (Name,Address,Zip_Code) VALUES (%(Name)s,%(Address)s,%(Zip_Code)s)" #print(param_dict) cursor.execute(sql_placeholder, param_dict) #inserts cleaned up data Analysis After all the data was successfully cleaned up and put into the database, I needed to find a way to link it and analyze something about it. I wanted to see if the number of dangerous dogs was in any way linked to off-leash areas or median income across Austin zip codes. Therefore, I created a mysql query that counted all the dangerous dogs in each zip code and all the off leash areas in each zip code and put those zip codes in order from lowest to highest median income. This involved COUNT(DISTINCT), grouping by zip code and ordering by income but when I did this with a regular inner join of all the tables it would not give me values where the count was zero. Therefore, I was only getting a portion of Austin zip codes, not all Austin zip codes. I then had to do an outer join of the off leash area table to the housing market table in order to get all values. This resulted in the following mysql query:

9 SELECT housing_market.zip_code, housing_market.median_household_income, COUNT ( DISTINCT dangerous_dogs.id ) AS dangerous_dogs_in_zip, COUNT ( DISTINCT off_leash_area.id ) AS off_leash_area_in_zip FROM housing_market LEFT OUTER JOIN off_leash_area ON housing_market.zip_code = off_leash_area.zip_code LEFT OUTER JOIN dangerous_dogs ON housing_market.zip_code = dangerous_dogs.zip_code GROUP BY housing_market.zip_code ORDER BY housing_market.median_household_income ASC I then wrote python script to make this query in the database and then write the results into a csv on the server that I could use to create a visual analysis of the results. This code is also on the server in the same file as the import code (sftp://agrieco:@holden.ischool.utexas.edu//export/home/u16/agrieco/project/project_code.py) import pymysql import csv import pprint # First set up the connection to the server connection = pymysql.connect(host="localhost", # your host, usually localhost user="grieco_a", # your username passwd="arg985", # your password db="grieco_a_project_dangerous_dogs", # name of the data base autocommit=true, cursorclass=pymysql.cursors.dictcursor) # as with opening a file we can use with to open the connection # the cursor is the object through which we talk to the sql server. with connection.cursor() as cursor: sql = "SELECT housing_market.zip_code, housing_market.median_household_income, COUNT( DISTINCT dangerous_dogs.id ) AS dangerous_dogs_in_zip, COUNT( DISTINCT off_leash_area.id ) AS off_leash_area_in_zip FROM housing_market LEFT OUTER JOIN off_leash_area ON housing_market.zip_code = off_leash_area.zip_code LEFT OUTER JOIN dangerous_dogs ON housing_market.zip_code = dangerous_dogs.zip_code GROUP BY housing_market.zip_code ORDER BY housing_market.median_household_income ASC " # SQL queries are just a string. #mysql query for my analysis cursor.execute(sql)

10 results = cursor.fetchall() # here we get the column names from the keys of the first item. csv_column_order = list(results[0].keys()) with open('project_query.csv', 'w', newline='') as csvfile: mycsvwriter = csv.dictwriter(csvfile, delimiter=',', quotechar='"', fieldnames = csv_column_order) # write the header row (it gets those from the fieldnames) mycsvwriter.writeheader() # and then each of the other results, row by row. for row in results: mycsvwriter.writerow(row) #writes results to the csv specified above: project_query.csv

11 Results The results of this query were written to the following csv:

12 I then decided to create a clustered column chart in excel with columns for count of dangerous dogs and off leash areas for each zip code and have those columns and zip codes be in order from lowest to highest median income for each zip code. From this analysis, it would seem that there is no correlation between number of dangerous dogs and median income because the count of dangerous dogs is higher and lower across the zip codes for all income levels. There are also a scattered number of off leash areas throughout zip codes of all income levels. Also, the number of dangerous dogs does not seem to increase or decrease depending on number of off leash areas in the same zip code. Therefore, it would seem that none of these datasets have any sort of immediate ties to each other. Dangerous dog count must be tied to something other than income or number of off-leash areas. Analysis Tool Learning To create my analytical bar graph from my query results, I chose to use Excel. I have used Excel many times for simple tables and for using functions on certain columns, rows and cells, such as SUM, but I have not created charts using it and not charts as complicated as the one I wanted for this project. My analysis got me a chart with every Austin zip code in order of lowest to highest median household income and the count of dangerous dogs and off leash areas for each zip code. I wanted the count to be in the form of two bars for each zip code and I wanted those zip codes to be in the median income order. I opened the csv file with these results in Excel and tried to highlight and create a bar graph from all the data. However, the bar graph made the

13 bars horizontal and I wanted them vertical, so I changed it to a clustered column graph. However, by simply highlighting all the data and putting it into a chart, it put everything into the bars instead of putting the zip code and median income along the x-axis. Because of this problem, I thought that I needed to create a pivot table with the data because I know you can designate columns and rows there. So, I tried that and made all the data come up as product so that I would get the same data and it wouldn t be a sum. However, there was no way to get zip code into median income order or even have them really be together on the same axis on the clustered column chart. Therefore, I went back to the original data in the original results table and found that in the select data section of the chart formatting menu, you can choose which data you want on the x-axis. I put all the data from the zip code and median household value columns into this and it came up exactly the way that I wanted with every zip code and count in median income order. I then deleted the zip code and median income bars and added labels to the axis. All of this I figured out from trial and error and didn t have to look anything up online. Challenge When working on this project, a challenge that I faced was getting the database design correct and cleaning up the CSVs enough to import them to the database. In class, we learned how to join tables through foreign keys and ids, so I tried to create the database tables with id foreign keys. However, that wasn t really working, so I realized that because the zip code for the housing market table is unique, I could use that to join the other tables through their zip codes. The csv files that I got for the housing market data and the off-leash area data had a lot of issues with them that made them difficult to import to the database. The housing market csv had dollar signs and commas in the columns relating to median household income, median rent and median home value that the database would not let me import into tables. I had to use re.sub() in python in order to get the import without the dollar signs and commas. The off-leash area csv column for address had a bunch of description and symbols that were not necessary and I needed to pull out just the address and zip code to but into column in the table on the database. I used re.sub() to get rid of the symbols and then had the code find five numbers together and count that as the zip code since that was the only item in the column with five numbers together. I used \d\d\d\d\d for this. Also with both of these, I had to create a param dict and have the param dict import into the fields in the database instead of rows since I messed around with the data in the rows.

OVERVIEW OF RELATIONAL DATABASES: KEYS

OVERVIEW OF RELATIONAL DATABASES: KEYS Keys (typically called ID s in the Sierra Database) come in two varieties, and they define the relationship between tables. Primary Key Foreign Key OVERVIEW OF DATABASE