Big Data Computing for GIS Data Discovery

Transcription

1 Big Data Computing for GIS Data Discovery Solutions for Today Options for Tomorrow Vic Baker 1,2, Jennifer Bauer 1, Kelly Rose 1,Devin Justman 1,3 1 National Energy Technology Laboratory, 2 MATRIC, 3 AECOM ESRI User Conference, July 2017

2 Data Discovery Challenges Data is often unstructured, mixed Spatial, contextual FTP, WWW, local filesystems, storage area networks, etc. Convoluted ways to search for and identify data Hard to identify all the data, i.e., see the whole Elephant, without falling down the rabbit hole Image from G. Renee Guzlas (available at 2

3 Data Discovery Needs Need tools to assist with / automate aspects of data discovery Parse data silos Improve how we use search engines Utilize machine learning to correlate relevant information Search for data in new ways (e.g., html source) Need infrastructure capable of processing millions+ of assets to: Extract valuable information Understand complex data relationships on a scale previously not possible Perform more robust spatio-temporal analyses Error 404 s from Izeas and modified from GitHub 3

4 NETL s Big Data Discovery Ecosystem (To Date) Data Mining Clients Data Collection: FTP Recursion WWW Crawl Data Analysis: Phrase Generation Relevance Analysis Geoprocessing Metastore (Hive, HBase) 4

5 Using a Big Data Ecosystem for Data Discovery! NETL s data driven research requires: Lots of data Incorporating different data types & formats, Integrating data from multiple locations (web, local, databases) Traditional Search methods impede our efforts: Search engine limits context to a few terms Labor intensive to conduct data searching Even more difficult to find relevant spatial data 5

6 Harnessing Big Data for Assistive Discovery: Approach Ingest Seed Corpus of Representative Documents / Web Sites Parse library of documents that are related to topic of interest Assistive Identification of Search Terms Generate phrases and sort by # occurrences found Manual filtering of resultant terms and categorization Review phrases and categorization for topic specific context Query search engine with desired terms Make lots of queries to Bing and snag links from Bing result pages Perform Crawl (Web and/or FTP) of results and store in Hadoop database Crawl the Bing result links and each link found within the crawled content Post processing / Data Mining: Solr Search and/or Contextual Cataloging Do useful work with the results! search the html for millions of sites for a map tag Identify new search terms based on catalog of crawled content Discover relevant documents and spatial information 6

7 Harnessing Big Data for Assistive Discovery: Approach Assistive Identification of Search Terms: Run Spark job to: Ingest and parse seed corpus text (library of document(s), web sites) identify phrases (1..n words in length) Sort results by # of occurrences 1 word: (methane,724) (gas,684) (emissions,455) (natural,285) (coal,279) 2 word: (natural gas,230) (methane emissions,149) (slip events,84) (greenhouse gas,67) (gas emissions,49) 3 word: (greenhouse gas emissions,42) (oil and gas,40) (renewable and sustainable,33) (coal bed methane,31) (fossil fuel subsidies,27) 7

8 Harnessing Big Data for Assistive Discovery: Approach Manual filtering of resultant terms and categorization Review generated terms, keeping desired discovered terms Create JSON based categorization schema [ { "Term_type": "Contextual", "Category": "Transmission", "Subcategories": "Pipeline, transmission", "terms": "Transmission, midstream, aboveground, belowground, interstate, intrastate, water crossing, interconnects" }, 8

9 Harnessing Big Data for Assistive Discovery: Approach Sample Search Engine Results Page for oil and gas We utilize Spark and Tika to automate data mining from Bing Crawl Bing pages for specific content Extract links from discovered content Recursively mine Bing s /search?q= links to populate initial crawler queue 9

10 Harnessing Big Data for Assistive Discovery: Approach Query search engine with desired terms to initialize crawler queue Ingest Categorized Terms Generate Bing URLs Execute Bing queries Parse Result Pages Crawl Bing relative search links Crawler Table sourceurl, redirectedurl, header, html, http_links, relative_links, ftp_links, textcontents 10

11 Harnessing Big Data for Assistive Discovery: Approach Perform Crawl (Web and/or FTP) of queue and store in Hadoop database Aggregate unique http, relative, and ftp links from crawler queue Select, crawl, parse links not previously crawled Optionally restrict crawler to specific domains Repeat process until threshold (# rows, queue empty, etc) Crawler Table Select unique links from aggregation of http, relative, ftp links (Optionally) Filter to restrict to specific domains Open links and parse with Apache Tika 11

12 Harnessing Big Data for Assistive Discovery: Approach Post processing / Data Mining: Solr Search and/or Contextual Cataloging 12

13 Use Case: Data Discovery for a Global Oil & Gas Database Big Data Machine Learning Tool Training Resources Advantages to using Big Data Computing: Semi-automated Repeatable Rapid Trainable Learns from each iteration Used to validate & augment manual search results Worldwide Web 13

14 Use Case: Data Discovery for a Global Oil & Gas Database Result: Hotspots of Global Oil & Gas Infrastructure Features Number of Features per 111km 2 Low (1) High (> 60,000) 14

15 Use Case: FTP Data Mining: Hadoop + ESRI Problem: How to search data in FTP data silos (millions of files, spatial and contextual) Solution: Index FTP silos using Hadoop and query using ESRI ArcMap FTP Sites Middleware Client USGS WVGISTC 15

16 Use Case: FTP Data Mining: Hadoop + ESRI Demonstration 16

17 Next Steps - Improving Discoverability to Drive Analytics & Insights Begin integrating offshore data, tools & models developed into an online, common operating platform, serving: Dynamic data Web-based tools & applications Big Data computational tools & analytics Find ways to incorporate additional data! Big Data search algorithms to identify additional data sources & sets Potentially integrate data from electronic forms, obtain equipment location information via sensors 17

18 Thank you Vic Baker Mid-Atlantic Technology, Research & Innovation Center (MATRIC), National Energy Technology Laboratory, Morgantown, West Virginia, USA For more information on data and tools visit: Kelly Rose U.S. Dept. of Energy, National Energy Technology Laboratory, Albany, Oregon, USA Jennifer Bauer ) U.S. Dept. of Energy, National Energy Technology Laboratory, Albany, Oregon, USA Devin Justman ) AECOM for U.S. Dept. of Energy, National Energy Technology Laboratory, Albany, Oregon, USA Acknowledgment: This technical effort was funded in support of the National Energy Technology Laboratory s ongoing research under the RES contract DE-FE Disclaimer: This project was funded by the Department of Energy, National Energy Technology Laboratory, an agency of the United States Government, through a support contract with AECOM. Neither the United States Government nor any agency thereof, nor any of their employees, nor AECOM, nor any of their employees, makes any warranty, expressed or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof. 18