Data Marting Crime Correlations Using San Francisco Crime Open Data

Transcription

1 Data Marting Crime Correlations Using San Francisco Crime Open Data Kiel Gordon Matt Pymm John Tuazon California State University Sacramento CSC 177 Data Warehousing and Data Mining Dr. Lu May 16, 2016

2 Abstract: San Francisco has open data sets online that are published by the City and County of San Francisco for the purpose of transparency and data analysis. Using crime data from SF OpenData we will explore data marting techniques with the purpose of data collection for knowledge discovery on the crimes of San Francisco. We parsed through 1,904,652 crime entries to ETL information into our database on Athena and created a web page where user can query information on data in San Francisco. Using data discovery with an online database we hope to improve safety in the city with awareness. Overview and Background: Data marts are small slices of data warehouses that allow users to use, manipulate and develop data away from enterprise databases. Data marts are organized to cater to end users and give them access to a specific type of data. Data warehouses for the purpose of OLAP are used for data reporting and data mining. We built our data mart for the purpose of exploring data on crime in San Francisco to examine the trends in crime and the effectiveness of the SF police force. We found our data on SF OpenData. It contains records of all crimes reported to the SFPD from It was a 1.9 million rows CSV with 13 dimensions. Dimensions include GPS, type of crime, the resolution, address of the crime, date, time, police district, and day of week. The data are Incidents derived from SFPD Crime Incident Reporting system and they are updated daily, showing data from 1/1/2003 up until two weeks ago from current date. The dataset allows for exporting data in many formats, with the choices from CSV, to JSON, to XML. The socrata platform that the data is housed has the ability shows all 1,907,487 rows. We housed our data so we can manipulate, sort, count and filter the data using queries we chose. For our data mart we chose to use one data set. Originally we wanted to compare Sacramento s crime data to Sf s crime data but the Sacramento Crime data set differed from Sf s as they classified crime differently and that was the main criteria that we wanted to mine the data over. Sf s open data was clean and well maintained, and had included all of the information that we required to mine the data. We loaded housed all rows but removed extraneous dimensions when running queries to improve access. We were limited by privileges granted on the Athena server but we focused our queries to allow users to explore what trends in crime occur in San Francisco. What we now have is a data mart that can show entries in our data set, query stats on the data set, and show the resolution of crimes, all for the purpose of educating the user on crimes in the city. We wanted to show the trends and correlations of crime in order to see what areas are safe and if police departments are effective with the purpose of reducing crime and making SF safer. We believe that solutions can be made with easy access to data on crime and an understanding of patterns and correlations. 1

3 Design and Methodology: We chose to use a the LAMP stack to house our datamart. We chose to make a website to allow anyone the ability examine San Francisco crime data. LAMP is linux, apache, mysql, and php and we chose that implementation as it was readily available through ECS. CSV You can get Our design started with our data set. It was provided online by CSV, a file that stores tabular data in plain text with each field separated by commas. Here s an example of what we were working with. It's a few lines from our ~1,900,000 row csv file. SCHEMA Our original project was to compare crime data between San Francisco and Sacramento. We created a schema based on that project. Unfortunately due to scope creep, time constraints, and incompatible data categories we chose to focus on San Francisco with the abundance of information and the categories that the cities data provided. 2

4 ETL In computing, Extract, Transform and Load (ETL) refers to a process in database usage and especially in data warehousing that: Extracts data from homogeneous or heterogeneous data sources. Transforms the data for storing it in the proper format or structure for the purposes of querying and analysis. By narrowing our data sources we made the ETL process easier. The data provided was clean and consistent, with no rows missing attributes or any noticeable outliers. We used visualization software Tableau to look for any outliers and made sure inserts to our SQL require information so any missing data would be caught by errors made by our insert. Our data set is also constantly updated with support from the author so we assumed any errors would be reported and documented. We had to transform and load our data from csv, a file with all the entries separated by commas, to our database on Athena that uses MySql. We used a sql script that took the file from our directory and parsed the data into inserts into our table. SQL SCRIPT LOAD DATA LOCAL INFILE '/home/physh/downloads/177data/sac_data.csv' INTO TABLE SacData FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 ROWS (RecordID, Offense_Code, Offense_Ext, Offense_Category, Description, District, Beat, Grid,@Occurrence_Date, Occurrence_Time) SET Occurrence_Date = STR_TO_DATE(@Occurrence_Date, '%m/%d/%y'); Lamp Stack After we loaded our SQL database with all of the information from 2003 we explored it using views and counts, grouping interesting categories to understand the data. We wanted to automate queries and make it easier to make calls on our database so we choose to make a web interface. We wanted to allow anyone to access our data so making a client server system allowed that. By using html forms we can customize what queries can be made by the user and by using drop downs we can pick and choose what parameters to view the data by. Our html front end cues what information to generate on the back end. Our back end php files returns 3

5 tables relating to the categories of our search. The php files connected to our Athena database and had embedded sql queries. It returns html formatted tables that are stylized using datatables, a jquery script that makes html tables sortable and searchable and paginated. The embedded sql commands we chose were meant to lead the user to find trends or correlations to see if crime is increasing or decreasing or if crime are being effectively resolved. Here s an example of our HTML, the options that you can choose for our first query and how it sends request to our php files that make up our back end. 4

6 Here is an example of our php files. Highlighted is the prepared query and the parameter inserted from the html post method Queries and Organization On our website we give the user four categories that the user can extract information from our database. Show Crimes With this category users can view full entries in the database and see full details on crimes reported either by police district or category of crime committed. We had to limit the domain of this call by year, PD, and category as Athena would return HTTP 500 errors (Internal Server Error) when calling large data sets. Number of Crimes This category gives counts occurrences within our data, allowing users to see how many crimes occurred. Our database can be queried to see how many crime happened within a police district, the counts of resolutions of crimes, and the amount of times certain crime occurred, either by type of crime or description of crime. The description gives more specification than category allowing more detail of incidents. 5

7 Crimes Over Time Here you can see the how many crimes occurred over chosen time interval. See how many crimes occurred over a span of year or months or see the times of day when certain crimes are committed. Resolution of Crimes This query can show you the results of law enforcement by police department or by crime. We chose to use these category to really drill down on information and to see the changes in crime over time throughout San Francisco. It was meant to inform the user and show if more or less crimes occurred and if they were being resolved. Results: An online queryable database. You can drill down on information on exact crimes, the amount of crime, the occurrences of crime over time, and the resolutions of the crimes. Url for our web interface: ml Here is a screenshot our website with our first category of queries. 6

8 Our returned table from the first category Running a query returns a table that shows our data according to the year and the category or police department that the user specified. We had to queries to grab data year at a time as time out errors occurred when accessing too many entries. The category returns the most detailed view of our data, returning actual entries of crimes reported. Interesting findings from the data: Using our DataMart we found a lot of interesting statistics about crime in San Francisco such as: About half of the reported Kidnappings go unsolved. More than half of the reported crimes go unsolved. Larceny and Theft are the most committed crimes since Although there isn t a huge change between days, most crimes occurred on a Friday. The majority of crimes occurred in the Southern Police District. Grand Theft from Locked Auto is the most common description of a crime. Significantly more Narcotics related crimes are reported in January than are reported in December. Significantly more Prostitution related crimes are reported in January than are reported in December. In 15 cases of Bribery the District attorney refused to prosecute. In 264 cases there was no resolution to an incident involving bribery. Were they bribed? Learning experience of the project lessons learned, tips, contribution of each team member, pointers to useful resources: We had to learn new languages and strategies in order to make our data mart. To make the interface we had to learn html and php and jquery. The back end we had to use sql in the context of a data mart so challenges that we ve never dealt with arose. We now can populate database with csv and can confidently manipulate a table with over millions of rows. We now 7

9 know the ETL process after loading that much data and we are now aware of the difficulties of preprocessing data and we recommend using clean and consistent data. We learned frontend and backend development using html, php, and jquery, which are useful tools for web development and to learn a good resource is W3Schools.com as well as Derek Banas from Youtube. Team Members Kiel Gordon Matt Pymm John Tuazon Responsibilities Contributed to the development of the PHP code which queried the data mart, and bootstrapped the user interface allowing a responsive design for different screen sizes and devices. The data mart is mobile friendly and all functionality should display and function the same on any device. Designed and created the database scheme. Created queries to answer specific questions. Developed PHP to execute queries and create HTML tables. Structured the HTML format and the the categories. Made the front end work on the backend and made the queries for the datamart. Summary: We made a data mart that is accessible through a responsive mobile friendly web interface and it houses entires on crimes in the city dating from References: [1] Textbook: Jiawei Han, Micheline Kambe, "Data Mining", 2nd Edition, [2] Prof. Lu, CSc 177 Lecture Notes, Spring Website: [3] SF OpenData, SFPD Incidents from 1 January Website: Safety/SFPD Incidents from 1 January 2003/tmnf yvry [4] Sf Police District Image, Activerain.com store.s3.amazonaws.com/image_store/uploads/agents/scottkeys/files/police%20 districts%20sf.jpg 8

10 Appendix Here is a map of San Francisco divided into the police districts in the data 9