Session 46: Developing Practical Infrastructure for Analysis from a Crash Records System. Chris Hughes ITIS Corporation

Transcription

1 Session 46: Developing Practical Infrastructure for Analysis from a Crash Records System 2006 Chris Hughes ITIS Corporation

2 Intro / Overview Presentation Goal: to provide a simple overview or primer of the steps to create an infrastructure for useful crash data analysis Crash Analysis Infrastructure and Process Collection Repository Warehouse Analysis Questions

3 ITIS Intro Project Management for Nevada DOT and DPS developing a comprehensive statewide map database and crash repository History Principals: Jerry Dildine, Simon Lewis - long history of professionalism in Transportation and GIS Competencies Client-side Project Management Transportation Safety Permitting Spatial Data Architectures Housecleaning Item: Drawing at the ITIS booth for USB Data drive: 1GB data drive to Richard Conard

4 Why the Need for this discussion Growing crash databases NDOT has 560K from 1997 to k from 2003 to 2006 Growing need/ability to ask specific questions of the crash data Choice for DOT/DPS: Plan for growth or build apps now Differentiate between Repository and Warehouse Repository is for maintenance and transactions (OLTP) Warehouse is for querying and analysis (OLAP) OLTP is messy for analysts and engineers let alone managers

5 Crash Analysis System Infrastructure ETL ETL ETL Fatal Injury PDO 0% 20% 40% 60% 80% 100% Collection Repository Warehouse Analysis Delivery

6

7 Data Collection The process of collecting the crash information Distributed Real time Digital Paper form /Data Entry Need to design a process/etl to Retrieve distributed data sets Verify data compliance Extract, Transform, and Load to the Repository

8 Data Repository Normalized Data structure for data maintenance and transactions Data is inserted daily Users may edit/update Highly normalized reuse of code tables Complex for users Need to design a process/etl to Determine key data elements for query aka dimensions Extract, Transform, and Load to the Warehouse Provide maintenance rules

9 Data Warehouse Data Warehouses are designed for analysis De-normalize the data Remove the code/lookups Develop dimensions of the data to be queried Data Marts can be built as pieces of the Data Warehouses dimension or domain specific Different technologies and methods to optimize query performance OLAP cubes Binary indexing Star/Snowflake schemas

10 Data Warehouse Logical Design Star Schema Conditions Location Vehicle Fact Table: Collision Time Driver/Party Agency

11 Data Warehouse Snowflake Schema Sub Segment Sub Route Sub Intersection Location Sub County MPO Fact Table: Collision

12 Data Analysis and Presentation Develop an Analytical Platform to answer the questions the decision makers need to know Components to consider Graphs and Charts Data Grids Reports Map/Spatial Decide on a platform/methodology COTS vs OpenSource Full Platform vs Components Personalization / Web Portal

13 Data Analysis and Presentation Always gather requirements from your users/customers Develop similar and repeatable BI and Reporting widgets in a dashboard or web top Start small and add to the successes with repeated improvements

14 Basic Recommendations Design of the Warehouse is paramount and can be accomplished in steps Extra dimensions can be added easily Start with a star schema rather than snowflake ETL is complex and is a discipline in and of itself Spatial/map processing should use open standards OGC Recommend doing research There are Multiple vendors and solutions Book: O Reilly - Data Warehousing in Oracle 10g Open GIS Consortium

15 Questions Contact Info: Chris Hughes Solutions Architect ITIS Corporation 5514 NE 38th Ave Portland OR Phone: (503)