BIG DATA CHALLENGES A NOAA PERSPECTIVE

Size: px

Start display at page:

Download "BIG DATA CHALLENGES A NOAA PERSPECTIVE"

Dale Patterson
5 years ago
Views:

1 BIG DATA CHALLENGES A NOAA PERSPECTIVE Dr. Edward J. Kearns NASA Examiner, Science and Space Branch, OMB/EOP and Chief (acting), Remote Sensing and Applications Division National Climatic Data Center National Oceanic and Atmospheric Administration, US DOC Ed.kearns@noaa.gov

2 Data Received from Many Sources 2

3 Storage Volume and Ingest Rate Total Archive Volume: 9.9 Petabytes at the end of FY 2012 End of Fiscal Year 3

4 Demand for NCDC Data Continues to Grow Types of Data Delivered: 4

Today s situation NOAA s Data Centers hold about ~10 PB of Earth Observing data, derived products, model/predictive results from ~1,000 observing and modeling systems.

5 Today s situation NOAA s Data Centers hold about ~10 PB of Earth Observing data, derived products, model/predictive results from ~1,000 observing and modeling systems. Access to most of these data is possible and is free and open -- but requires a difficult search and an even more difficult task understanding the format and content of the data sought. Result Tailored Data Delivery, which is expensive and limited in scope/use. This is limiting Government s use, the public s use, industry s use and thus the Nation s return on its investment.

6 A Story

7 A General and Noble Goal Achieve the maximum return on investment for the Government s environmental observing and modeling systems. How? By making the data collected by these systems easily discoverable and useable by industry, academia, decision-makers, and the public. This goes beyond open and transparent and addresses utilization via ease-of-discovery/use

Cloud Access Pilot Phase I: consolidated Storage Service 1/31/2013 Costs Technology Challenges Observations Security Conceptual

NOAA dataset is retained internally at a NOAA Data Center, while a public copy is sent via one-way push to a publicly accessible

perhaps invoke additional services (subsetting, visualization, transformation, etc) Non-real-Time Processing: climate product

The resulting products are also disseminated via the Cloud.

8 Cloud Access Pilot Phase I: consolidated Storage Service 1/31/2013 Costs Technology Challenges Observations Security Conceptual Cloud Architecture Processing Services * Possible Cloud deployment scenarios for environmental data include: The master copy of a NOAA dataset is retained internally at a NOAA Data Center, while a public copy is sent via one-way push to a publicly accessible commercial Cloud where external customers (the private sector, the general public, foreign governments) can obtain data and perhaps invoke additional services (subsetting, visualization, transformation, etc) Non-real-Time Processing: climate product generation, satellite data reprocessing, and other non-real-time computation are performed on commercial cloud resources. The resulting products are also disseminated via the Cloud. * Storage Services Commercial CSP CLASS NODC NGDC NGDC Other *Note: This image was adapted from NOAA Environmental Data Management Framework Draft v0.3 Appendix C - Dr. Jeff de La Beaujardière, NOAA Data Management Architect

9 Google/Bing vs current Data.gov Search Engines: Massively indexed search Free text search Users presented with likely top hits based upon factors such as popularity, relevance, etc. Portal: List of URLs Have to examine and investigate each URL Repeat to find those most relevant to the search

10 What we want Free text-based searches for data Similar to Images discovery and access now Not a Government server-side tool (see Data.Gov and its many extensions) Open up government holdings to be crawled and indexed efficiently and effectively Enable web search engines to include government data in their search indices Enable access and interpretation of these Big Data

11 Example: search for ocean color data related to harmful algal blooms in Key West Search result: we get a list of stuff to investigate, including portals and papers, but no actual data

12 Close analogue: Images search Images are now searchable via free text Why? How? Metadata about the images have been made available for crawling and indexing Images are in only a limited number of formats Standardization (but not homogenization) Visualization tools/libraries built into standard browsers We are attempting to make the metadata for earth science datasets adhere to standards (e.g. FGDC, ISO ) Migrating to self-describing data formats like CDF, netcdf, HDF Are NOAA s earth science data ripe for crawling?

13 Data

14 But we ll need Discovery Find the data using an indexed text search Access Have the data present themselves for use Interpretation Data must be independently understandable Manipulation? Ability for subsetting, combination, visualization, etc.

15 So If Earth Scientists have been working towards getting their data organized and have Metadata in a standard format Data in a standard format Practical vocabularies, Unique IDs, etc. etc. etc. Populating catalogs (GeoPortal), Applying access tools (OPeNDAP), Modern transport/management systems (irods) Why don t we have this capability today? Having trouble identifying a general target that will enable such easy search and discovery Need integrated tools for interpretation, visualization, display Need Big Data help!

16 We need NSF Big Data s help We need help in defining a reasonable target for this effort. Big Data Researchers and Information Industry need to understand more about what NOAA has to offer. NOAA needs to understand what Big Data Researchers and the Information Industry have to offer. Need to jumpstart this dialog ideas?

US NODC Archival Management Practices and the OAIS Reference Model Donald W. Collins

US NODC Archival Management Practices and the OAIS Reference Model Donald W. Collins National Oceanographic Data Center SSMC3 4 TH Floor, 1315 East West Hwy, Silver Spring MD 20910 Donald.Collins@noaa.gov