CyberGIS Big Data. Dawn Wright Esri Chief Scientist. GIScience 2012, September 20, 2012, Columbus, OH

Transcription

1 CyberGIS Big Data Dawn Wright Esri Chief Scientist GIScience 2012, September 20, 2012, Columbus, OH

2 What are key characteristics of big data and cybergis? What new fundamental problems does big data pose to GIScience?

3 Before: Federated Model Portalpalooza HydroPortal ArcGIS Online Data.gov NOAA Geoplatform OER YouTube NOAA YouTube NCDC NGDC NCDDC OER NOS A CSC SECART B = A searches B NODC IOCM CMGDS USGS.gov

4 Now: An Open Platform Model Discovery, Access and Use of Information Products Scientists Govt Agencies Managers Environmental Professionals Policy Makers ArcGIS Online Citizens

5 Volume 1.6 Petabytes of data in geo-distant data server nodes.

6 Velocity

7 Variety

8 Tenets of Big Data VERACITY understanding data provenance, confidence in resulting information VALUES people-centric platform, building communities, empowering communities The presence of any V factors require a departure from traditional SQL databases and traditional data processing workflows. Distributed processing technology multi-node architectures Stream-processing engines for real-time analysis Spatially-enabled NoSQL database, document (XML) stores and search engines

9 Big Data Technologies Stream processing - Storm Map reduce framework - Hadoop NoSQL databases Hbase and Cassandra or MongoDb Hive and Pig for HDFS

10 Big Data Prototype Projects Air traffic feeds - Detecting clusters/neighbors in 3D space Twitter feeds AIS ship feeds Classified DoD and intelligence

11 CyberGIS and Big Data Deploy successful services Improve collaboration Simplify end user experience

12

13 Project Metrics Initial scope = 50 million sq km Select best of 2 million+ Ikonos scene archives 302,765 scenes acquired, processed 152 Tb total raw data 325 Tb total derived data (pan sharpened images, orthophotos, tiles, caches)

14 King Kong Amazon Cloud Architecture Image Processing Farm Image Processing Farm Selected Scenes Instances + Lustre + JPS Instances + Lustre + JPS Cache Processing Farm Ikonos Source Simple Storage Service Bucket Instances + Lustre + AGS ArcGIS Online Serial processing likely would have taken years

15 Sharing Analyses as Packages and Services Package GIScientists Analysis Share as Service Everyone

16 World Water Online A framework for sharing data, workflows, and services GIS Analysis Maps Global to Local Hydro Models Observations

17

18 Toward a Knowledge Community Playground? Connecting CyberGIS with its communities Integrating and synthesizing information from many sources Facilitating communication and collaboration Breaking down barriers between institutions, disciplines and cultures

19 King Kong Future Technical Challenges Caching Dramatically Improved At Extend Stability & Scalability - Evaluate Correctness of Cache Tiles - General Cache Validation Tools Deploy Tie-point & Bundle Capability Improve Color Balance/Stretching Methods For Better Overall Color Quality Workflow Validation For Other Sensors / Sensor Models Integrate Workflow Manager for Raster Data

20 Path Forward on King Kong Technology Production Workflow Ingest QA Tie Point & Bundle QA Orthophoto Pan Sharpen Color Balance Seamline Process Tile Cache 10.1 Commercial Release Workflow (2012) Ingest Tie Point & Bundle QA Mosaic Dataset Raster Functions Map or Image Service Cache PCI GXL/Geomatica ArcGIS

21 Dawn Wright Steve Kopp