Starting small to go Big: Building a Living Database

Transcription

1 Starting small to go Big: Building a Living Database Michael Sabbatino 1,2, Baker, D.V. Vic 3,4, Rose, K. 1, Romeo, L. 1,2, Bauer, J. 1, and Barkhurst, A. 3,4 1 US Department of Energy, National Energy Technology Laboratory, Albany, OR; 2 AECOM, Albany, OR; 3 Mid-Atlantic Technology Research & Innovation Center (MATRIC), Morgantown WV; 4 US Department of Energy, National Energy Technology Laboratory, Morgantown, WV Jennifer Bauer, Jenny DiGiulio, Devin Justman, Lucy Romeo, Kelly Rose, Patrick Wingo Solutions for Today Options for Tomorrow POC: Michael.Sabbatino@netl.doe.gov

2 Challenges & Needs of Scientific Data 80% Dark Data Data Access ~80% loss of published data after 20 years Data Discovery 20% public data versus 80% private Date Interoperability Variety of data makes it difficult to create, exchange, & use data across different applications and systems Date Analytics & Visualization Requires advanced computational capabilities, algorithms, & large data stores to analyze these data Learn & Optimize Aggregate & Label Explore & Transform Move & Store Collect AI & Deep Learning A/B testing, experimentation, simple ML algorithms Analytic, metrics, segments, aggregates, features, training data Cleaning, anomaly detection, prep Reliable data flow, infrastructure, pipelines, structures and unstructured data storage Instrumentation, logging, sensors, external data, user generated content working up the Data Science Hierarchy of Needs 2

3 Collect >700 datasets >4 million features Machine Learning Automated Approach- A tool that scans seed resources and identifies relevant keywords, then crawls the web and parses the data for integration small Beginnings: Developing a Global Oil & Gas Database Discovered & integrated open data sources of information related to oil & gas infrastructure across the globe /dataset/global-oil-gas-features-database 3

4 Move & Store EDX - A Virtual Library & Laboratory for Energy Science Virtualizing team analytics Continuing innovations to connect researchers to online Earth-Energy system resources Increasing number of tools & apps for use in team workspaces Data Workflows & Structure 80% Dark Data EDX vs Dark Data Custom smart search tool in development Digital spatial team notebook Auto-indexing algorithm, provides analysis of your search and helps recommend other items EDX Smart Search - A machine learning, big data tool for rapid, online,.zip, & FTP spatial & non-spatial data mining with Hadoop + Bing + ESRI 4

5 Explore & Transform Storing Databases with different data types, formats, & resolutions Includes Data workflow, infrastructure, pipelines, structured & unstructured data External Apps Research The Living Database Store & Share Data in a Structured Secure Database Environment Reduce Redundant Acquisition Direct Data Access (not file based storage) Consistent Data with Staff Turnover Enhance Collaboration Curation of data and knowledge Allows Direct Analysis from Database Apps Data Lifecycle People 5

6 Aggregate & Label Improve joint analysis of multiple datasets focus on advancing Big Data mining, machine learning, and advanced geoprocessing computing Combine data and tools to Select relevant datasets Tools, Analytics, & Metrics Developing tools & approaches to manage multiple heterogeneous datasets Develop a probabilistic approach to assess scientific data using big data analyses Develop stochastic approaches to reduce uncertainty Evaluate correlations and spatio-temporal trends Highlight resultant data and analysis and reuse for in further research 6

7 GOGI Learn & Optimize EDX continues to evolve in response to the needs of its users and NETL s knowledge transfer goals Future Big Data Development & Analysis Employing smart search tools to include open resources Billions of Records Carbon Storage Data & Resources Millions of Records EDX Cloud Services AWS Development Living Database Common Operating Platform for Data Analytics Geocube Spatial Data Viewer Fuzzy Logic Analytics (SIMPA) Integration with decades of DOE R&D Federating Open Source data EDX & GeoCube (search and location) ID Data gaps in subsurface puzzle Oil & Gas, Geothermal Data & Resources Millions of Records *These attributes data sources are evolving quickly with implementation of new tools and engagement of key stakeholders Millions of Records 7

8 Advanced computer science & research Developing Schema Matching AI Variety of data sources with diverse data schemas Manual schema matching is time consuming & inefficient Plan to develop and use existing machine leaning algorithms to match disparate data schemas: Schema level Element Level Structure Level Linguistic Matching Syntactic Techniques ~ Thank you! ~ Michael Sabbatino Michael.Sabbatino@netl.doe.gov 8

9 Questions? Come check out this awesome poster! 9