Portal: Applications of New Technology to Transportation Data Archiving. Kristin Tufte & the Portal Team NATMEC, July 1, 2014, Chicago, IL

Transcription

1 + Portal: Applications of New Technology to Transportation Data Archiving Kristin Tufte & the Portal Team NATMEC, July, 24, Chicago, IL

2 + Who is Kristin? 2 years Data Management System Design and Implementation Paradise (99-996) - sold to NCR/Teradata NiagaraST Data Stream System (996-24) S-Store Streams + OLTP (23- ) years Transportation Data Management Portal Transportation Data Archive (24- ) Portland Observatory (23- )

3 + New Technologies Big Data, Cloud, Data Streams

4 + Big Data Why all the interest? An increased number and variety of data sources that generate large quantities of data Sensors (e.g. location, acoustical, ); Web 2. Realization that data was too valuable to delete ADUS - ITS Program Plan addendum 998 Dramatic decline in the cost of hardware, especially storage If storage was still $/GB there would be no big data revolution underway Slide credit: David DeWitt, Microsoft/UW-Madison, SQL Server PASS Talk 2

5 + Is Transportation Data Big Data? Amount of Stored Data By Sector (in Petabytes, 29) Petabytes Sources: "Big Data: The Next Frontier for Innovation, Competition and Productivity." US Bureau of Labor Statistics McKinsley Global Institute Analysis zettabyte? = million petabytes = billion terabytes = trillion gigabytes Figures credit: David DeWitt, Microsoft/UW-Madison, SQL Server PASS Talk 2

6 + Volume, Velocity and Variety Variety is the most difficult and potentially most critical axis of Big Data Transportation data has Variety in spades Big Data s Implications for Transportation Operations: An Exploration White Paper March 24; Prepared by Volpe Center; Prepared for US DOT ITS JPO Radar Bluetooth Signal loops Ace Sensor Other: Incident reports Crossing demands Bus priority requests SCATS AQ Sensor Bus AVL, boardings Slide credit: Adam Moore, Miguel Figliozzi, PSU

7 + The Cloud: NoSQL vs. NewSQL vs. RDBMS Cloud Computing: Many definitions, many promises Reduced time to insight Saving resources

8 SQL: Data Arrives Sometimes termed Schema First Cleanse the data Derive a schema Transform the data Load the data RDBMS SQL Queries 6 NoSQL: Data Arrives Sometimes termed Schema Later " No cleansing " No ETL " No load NOSQL System Application Program 2 " Analyze data where it lands Slide credit: David DeWitt, Microsoft/UW-Madison, SQL Server PASS Talk 2 8

9 + Save Resources 9 Data center in the cloud Resources Capacity Resources Capacity Demand Demand Time Time Unused resources Slide Credit: RAD Lab, UC Berkeley

10 + But, Don t be Fooled NoSQL systems make lots of promises But they don t work for everything

11 + Portal An Update

12 + Portal Where is it at today? Happy th Birthday Portal (April ) Publically funded (Thanks to NSF, FHWA, OTREC, Metro & Transport, RTC) Focus on open-source software (PostgreSQL, PostGIS, OpenLayers, HighCharts) Focus on open data (Thanks to all our wonderful collaborators) New Transit Load and Performance Map (GTFS + PAX data) Lots of new (local) data feeds

13 + Freeway Data Flow Portal OR-WA Archive # XML Feed # 2 second granularity # being phased out ODOT DAQ # XML Feed # 2 second granularity # automated station inventory file # XML Feed # 2 second granularity WSDOT - Loop Detectors - Wavetronix ODOT - Loop Detectors - Wavetronix - Bluetooth Travel Time Lane County - Wavetronix

14 + Transit Data Flow # Portal Archive import processing combines PAX and GTFS data Portal OR-WA Archive # Quarterly PAX data exported TriMet Enterprise Database # PAX data inserted in Enterprise Database # Data is cleaned and aggregated TriMet - AVL/APC (Init) - GTFS Data # GTFS data published publically # No enterprise database (yet) # Process to be determined C-Tran - AVL/APC (Init) - GTFS

15 + Arterial Data Flow - Current City of Portland - Bluetooth # Bluetooth data gathered from devices by scripts on CoP servers # Data uploaded to Portal weekly # Processing scripts calculate travel times Portal OR-WA Archive # Hourly data feed created by TransSuite # Data uploaded to PSU hourly (sftp) City of Portland - Signal System, including MOE Logs (TransSuite) - Bike/Ped Counts # Central Signal Server is Shared Washington & Clackamas County - Signal System (TransSuite) Clark County - Wavetronix Near Future: City of Vancouver - Bluetooth Clark County - Bluetooth City of Vancouver - Wavetronix - Signal System (ATMS.Now) # Data generated using Wavetronix report-generation system # Data uploaded to PSU nightly

16 + Transit Map Combines GTFS (General Transit Feed Specification) with AVL/APC (Automatic Vehicle Location/Automatic Passenger Counter) data Lots of GIS processing involved GIS process is complex freeway data too Metrics include: Stop activity Ons/offf; On-time performance Segment activity Load and Utilized capacity Segment Load

17 + Stop-Level performance

18 + S-Store Big Data Bikeshare Demo Demo applying Data Stream technology to a Bikeshare scenario Collaboration between PSU, Intel, MIT, Brown Univ. To be shown at VLDB 24 (Hangzhou, China)

19 + Portland Observatory: Urban Informatics Variety Testbed Challenges the third V of Big Data: variety Observations and context For researchers, planners, managers, public Building on experience from Portal transportation archive Increasing data touches

20 + THANK YOU