Assessing Applications of Cloud Computing to NASA s Earth Observing System Data and Information System (EOSDIS)

Size: px

Start display at page:

Download "Assessing Applications of Cloud Computing to NASA s Earth Observing System Data and Information System (EOSDIS)"

Millicent Thornton
5 years ago
Views:

1 Assessing Applications of Cloud Computing to NASA s Earth Observing System Data and Information System (EOSDIS) Chris Lynnes, Katie Baynes, Mark McInerney NASA/GSFC ESDIS

2 Earth Observing System Data and Information System (EOSDIS) EOSDIS distribute Research Applications Education data downlink subset Users archive capture and clean process 2

3 Earth Observing System Data and Information System (EOSDIS) EOSDIS distribute Research Applications Education data downlink subset Users archive capture and clean process 3

4 Opportunities from Cloud Computing 1. Data Volume a. Challenge for Archives b. Challenge for Users 2. Performance a. Load balancing and scaling b. Resiliency 3. Cost Reduction(?) 4

5 Overall Approach Program of prototypes Demonstration prototypes: will this work? Operational prototypes: how does this change the way we work? Focus on public clouds for max cost savings Leverage existing software and efforts...but refactor software to be cloud-native not forklift migrations 5

6 EOSDIS Cloud Prototypes Key Archive Mgmt Analytics Support Application Hosting

7 Petabytes Data Volume Challenge for Archives Archive Volume Distribution Volume 7

8 Archive Cloud Prototypes Benefits from Archive in the Cloud Cost savings for storage of Big Data? Avoid data downloading and local data mgmt Key

Alaska Satellite Facility *EDC = EROS (Earth Resources Observation and Science) Data Center Global Imagery Browse

9 Archive Cloud Prototypes Alaska Satellite Facility Web Object Storage prototype Distribute Sentinel radar data from Amazon storage ESA data hub EDGE Server in cloud NASA Sentinel Gateway Data cannon EDGE Server cluster at EDC* Alaska Satellite Facility *EDC = EROS (Earth Resources Observation and Science) Data Center Global Imagery Browse Service in the Cloud (AWS Key Lambda) Ingest and Archive management prototype (AWS Lambda) NISAR Mission Preparatory Prototype

10 Data Volume Challenge for the Users One Solution: Move more analysis closer to the data 10

11 Analysis More Complexity Subset Data Variable Spatial Area Reprojection Quality Filter Mosaicking Transform Simple Stats Analyze Complex Stats End User s Algorithm Seasonal Time Series 11

12 Cloud Analytics Prototypes Benefits from Cloud Analytics Analyze data at scale Analyze datasets together easily Avoid data downloading and local mgmt

13 Users work with the data in place : Avoid tedious cycles of download-preprocess-store-analyze Analysis Today Archive A Archive B Local Computer Archive C A New Paradigm Analysis Cloud

14 Distributing the data enables distributed, parallel computing. Results are computed for chunks of the problem simultaneously 14 and then recombined and reduced

accelerate processing Spark + Cassandra DB Community open

15 Cloud Analytics Prototypes Analysis support toolbox to attract users to cloud analytics NEXUS prototype to accelerate processing Spark + Cassandra DB Community open source tools DAAC-developed tools Cloud analytics examples and recipes

16 Analytics in the Cloud Long Term Paradigm Shift: How Scientists Work on Data Scientists will work on data in place instead of downloading High-value data will (also) be in databases Pre-existing toolsets will be easy to find and use 16

17 Application-Hosting & Processing Prototypes Cloud Application Deployments Port existing applications to public cloud Leverage NASA-Compliant General Application Platform 17

18 NGAP: Compliance-as-a-Service Software-as-a-Service Compliance-as-a-Service security controls, Authorization to Operate governance procurement and accounting reliability and availability Platform-as-a-Service Infrastructure-as-a-Service 18

19 Application-Hosting & Processing in the Cloud Paradigm Shift: How We Implement Systems Off-the-shelf systems reduce effort for hardware procurement and deployment Automate everything: testing, deployment, scaling, failover,... Benefits Science users get: New capabilities sooner NASA gets: More reuse Lower hardware costs 19

20 EOSDIS Cloud Progress so far... Archive Prototypes Serving data to the public from Alaska Satellite Facility Web Object Storage prototype in AWS Simple Scalable Storage End-to-end lambda workflow has been demonstrated for the Ingest / Archive management prototype Global Imagery Browse Service is undergoing system testing 20

21 EOSDIS Cloud Progress so far... Analytics Prototypes NEXUS analytics algorithms benchmarked vs. Giovanni Roughly order-of-magnitude speedup achieved on a cluster Now porting to cloud-native architecture 21

22 EOSDIS Cloud Progress so far... Application Hosting Prototypes NASA-Compliant General Application Platform is now authorized to operate publicly Earthdata Search client is operational and accessible to the public in AWS Modified processes to account for new costing mechanisms 22

23 Terra Incognita 1. Vendor Lock-in 2. Future storage costs 3. Uncapped egress costs 4. Security Restrictions and Network trust

Earthdata Cloud Analytics Project

Earthdata Cloud Analytics Project Chris Lynnes* and Rahul Ramachandran* NASA *U.S. Civil Servant 2 Earth Observing System and Information System (EOSDIS) EOSDIS distribute Research Applications data downlink