EARTHCUBE CONCEPTUAL DESIGN

Transcription

1 EARTHCUBE CONCEPTUAL DESIGN A Scalable Community Driven Architecture Overview PI: G. Djorgovski (Caltech) CO-I: D. Pilone, T. Pilone (Element 84), D. Crichton, E. Law (JPL) Other key personnel: S. Caltagirone (E84), S. Hughes (JPL), T. Huang (JPL), A. Mahabal (Caltech) DSWG May 5, /7/16 1

2 NSF EarthCube Enable geoscientists to address the challenges of understanding and predicting a complex and evolving Earth system by fostering a communitygoverned effort to develop a common System Infrastructure

3 Caltech and JPL Funded Projects! Partnership between Caltech and JPL through Joint Initiative on and Technology! Two Projects! Conceptual Design: A Scalable Community-Driven Architecture (7/14-6/16)! Integrative Activity: EarthCube Integration & Test Environment (9/15-8/17)

4 A high level system blueprint for the definition, construction, and deployment of both existing and new components to ensure that they can be unified and integrated into an evolutionary national infrastructure for EarthCube 1/7/16 4

5 Methodology! Identification of stakeholders, concerns and requirements! Identification of architectural use cases and drivers! Selection of an architectural framework! Development of the architectural principles! Development of the architectural models! Capture of the architecture artifacts in a consolidated report! Generation of recommendations for adopting the architecture for the EarthCube program 1/7/16 5

6 1/7/16 6

7 Stakeholders Stakeholder/Actor NSF Program Managers EarthCube Scientists EarthCube Developers Concerns Make decision and provide guidance at the EarthCube program level. Provide suf>icient funding to support the EarthCube mission. Use EarthCube resources and services to conduct scienti>ic research. Publish scienti>ic results & curate data as needed. Develop technologies and services that can be integrated into EarthCube. Establish EarthCube requirements, framework and operational concept. EarthCube Architects External Users Curators Owners External Facility EarthCube Governance Committees EarthCube Of>ice Staff Develop information model (vocabulary, ontology). Establish standards guidelines. Ensure interoperability between EarthCube Building Blocks. Use EarthCube resources and services for research, education, and decision-making. Ensure data is properly captured in EarthCube compliant data repositories. Responsible for producing the data. Concerned about its distribution and use. Responsible for archiving data at other agencies (NASA, NOAA, USGS, etc); interoperability with the EarthCube Cyberinfrastructure. Responsible for generating and monitoring the governance for the system including data curation, access, use case priority, interoperability standards, Responsible for maintaining the community involvement within EarthCube and communicating changes and how to use the system. 1/7/16 7

8 Use Cases! Big Discovery, Comparison, Provenance, Model & visualization! Collaborative! Dark Contribution! Tools Contribution! Documentation! Models Sharing! High Performance Computing and Storage Resources! Real Time! Physical Sample Curation 1/7/16 8

9 Drivers! Transform and accelerate research and discovery by turning data into knowledge and enabling interdisciplinary data integration.! Provide critically needed data, tools, and computational resources and frameworks for cross-domain scientific collaboration, analysis and with long-term geoscience software and data preservation, discovery and use.! Provide a geosicences cyberinfrastructure and architecture that is scalable, extensible and sustainable.! Understand how various EarthCube building blocks integrate. 1/7/16 9

10 Frameworks! Zachman Framework [ ] - For organizing stakeholder concerns and perspectives.! ISO/IEC/IEEE 42010:2011 [ ] - For architectural description guidelines.! Reference Model for Open Distributed Processing (RM-ODP) [ ] For architectural patterns for distributed systems.! Open Group Architecture Framework (TOGAF) [ For managing the architecture.! Federal Enterprise Architecture Framework (FEAF) [ ] For classifying the architecture into architectural elements and viewpoints. 1/7/16 10

11 ! Scalability! Community Driven! Open! Interoperability! Sustainability! Distributed 1/7/16 11

12 Manage Satellite Instrument Systems Manage Airborne EarthCube CI Manage Agency Earth Archives EarthCube Discovery Provider 1/7/16 12

13 Manage Satellite Instrument Systems EarthCube CI Manage Airborne Manage Agency Earth Archives Provider EarthCube Repository Other Systems (e.g. Other Systems (e.g. NOAA) NOAA) Systems (In-Situ, University) EarthCube Discovery 1/7/16 13

14 Teams EarthCube CI Manage Satellite Instrument Systems Infrastructure (, Algorithms, Machines) Manage Airborne Manage Agency Earth Archives Provider EarthCube Repository Other Systems (e.g. Other Systems (e.g. NOAA) NOAA) Systems (In-Situ, University) EarthCube Discovery 1/7/16 14

15 Teams EarthCube CI Manage Satellite Instrument Systems Infrastructure (, Algorithms, Machines) Earthcube Analytics Centers Research Manage Airborne Applica>ons EarthCube Repository EarthCube Discovery Manage Agency Earth Archives Provider Other Systems (e.g. Other Systems (e.g. NOAA) NOAA) Systems (In-Situ, University) Decision Support Analysis 1/7/16 15

16 Benchmark! Earth System Grid Federation (ESGF)! Early Detection Research Network (EDRN) ASF DAAC SAR Products Sea Ice, Polar Processes! NASA s Earth Observing System and Information System (EOSDIS) Key Center SIPSs ECS Sites JPL MLS, TES PO.DAAC Ocean Circulation Air-Sea Interactions NCAR, U of Col. HIRDLS, MOPITT, SORCE NSIDC DAAC Cryosphere, Polar Processes San Diego ACRIM LP DAAC Land Processes & Features ORNL DAAC Biogeochemical Dynamics, EOS Land Validation GHRC Hydrological Cycle & Severe Weather GES DISC Atmos Composition & Dynamics, Global Modeling, Hydrology, Radiance CDDIS Crustal Dynamics Solid Earth GHRC AMSR-E, LIS, AMSR2 LaRC CERES, SAGE III SEDAC Human Interactions in Global Change GSFC GLAS, MODIS, OMI, OBPG OBPG Ocean Biology & Biogeochemistry LAADS/ MODAPS Atmosphere ASDC Radiation Budget, Clouds, Aerosols, Tropo Chemistry 1/7/16 16

17 Architecture Elements EarthCube System Architecture Process Architecture Architecture Technology Architecture Lifecycle Software Lifecycle Research Software Lifecycle Administrative Information Model Crosscutting Domain Generation Curation Transport Technology Planning Software Development Release Software Development Software Versioning Software Archiving Governance Standards Technology Archive Model Query/ Access Formats Distributed Architecture Access IT Security Ingest (Receive, Validate, Accept) Catalog/ Management Storage (Repository) Ingest Management Search Distribution Analytics Visualization Software Search & Distribution Algorithm Storage & Discovery Policies Resource Planning Archive Organization Grammar Dictionary Collaboration Standards Evaluation User Roles, Support and Feedback Use metrics for data, software and site use Processing Search and Discovery Integration Analysis Distribution Visualization Publication 1/7/16 17

18 Lifecycle Original&genera7on&of&data&(from&sensors,&inves7gators,&etc)& $Genera)on$ Prepare&data&for&use&and&submission&into&EarthCube& $Cura)on$ and$prepara)on$ Maximize&informa7on&throughput&against&available&bandwidth& $Transport$ Supports&the&capture&and&valida7on&of&data&into&EarthCube& $$Ingest$ Provides&overall&data&management&services&for&the&data&in&EarthCube&& $ Management$ Enables&discovery,&access&and&distribu7on&of&the&data& Discovery,$Access$ &$Distribu)on$ Enables&the&analysis&of&massive,&distributed&heterogeneous&data& $Analy)cs$ Provides&a&plaAorm&for&integra7ng&analy7cs&with&rendering&and& understanding&the&data& Visualiza)on$ 1/7/16 18

19 Information Model - Definition! An information model in data engineering is a representation of concepts and the relationships, constraints, rules, and operations to specify data semantics for a chosen domain of discourse.! It provides a sharable, stable, and organized structure of information requirements or knowledge for the domain context. [1] [1] Y. Tina Lee (1999). "Information modeling from design to implementation" National Institute of Standards and Technology. 19

20 Information Model - Context 1/7/16 20

21 Model-Driven Strategy Informa(on)System)Architecture Informa(on)Architecture configure) System)Model) Informa(on)Object) Iden(fica(on) Referencing) State) System)Architecture Domain)Model)(governance)levels)) Top)Level) Representa(on/Format) Context,)Provenance,)Integrity) Domain) ) Engineering) Explora(on) Missions/Systems) Satellite/Airborne) Mission)Opera(ons) describe) Configurable)Components) )Management)Model) Configured)) use) Search/Access)Model) drive) System) Analy(cs)Model) produce) ) 21

22 Architecture (Functional) 22

23 Knowledge Framework 23

24 System Framework Information Knowledge Node Analytic Node Providers Sources Images Measurements/ Observations Remote Sensing Text file/ ASCII Spreadsheets Ingest Transfer Validation Harvesting Packaging Management Stewardship Curation RDBMS Postgres Oracle MySQL NoSQL MongoDB Cassandra Array SciDB Storage SAN S3 SSD Distribution Publication Push Access OPeNDAP THREDDS W10N Search OpenSearch Lucene Solr ElasticSearch Abstraction Java Python Ruby Groovy Scala Analysis Platform Hadoop/HDFS MapReduce ZooKeeper Spark Graph DB TitanDB Neo4J Triple Store Virtuoso AllegroGraph Sesame Fuseki Message Passing Interface Virtual Machine Container Single Machine High Performance Computing GPU Query/ Retrieval Lucene OpenSearch SPARQ Analysis Workflow LAS Analytics Machine Learning Pattern Recognition Climatologies Reduction Uncertainty Analysis Visualization OGC (WMS, WMTS, ) TWMS Slices Plots and Coordination Integrated Views Viewer and Interactive Query Search Query Subset Applied Framework 1/7/16 24

25 Sources Images Measurements/ Observations Remote Sensing Text file/ ASCII Sources Images Measurements/ Observations Remote Sensing Text file/ ASCII Sources Images Measurements/ Observations Remote Sensing Text file/ ASCII Spreadsheets Spreadsheets Spreadsheets Ingest API Transfer Validation Harvesting Packaging Ingest API Transfer Validation Harvesting Packaging Ingest API Transfer Validation Harvesting Packaging Management Stewardship Curation RDBMS Postgres Oracle MySQL NoSQL MongoDB Cassandra Array SciDB Storage SAN S3 SSD Framework Management Stewardship Curation RDBMS Postgres Oracle MySQL NoSQL MongoDB Cassandra Array SciDB Storage SAN S3 SSD Framework Management Stewardship Curation RDBMS Postgres Oracle MySQL NoSQL MongoDB Cassandra Array SciDB Storage SAN S3 SSD Framework Distribution Publication Push Access OPeNDAP THREDDS W10N Search OpenSearch Lucene Solr ElasticSearch Abstraction Java Python Ruby Groovy Scala Distribution Publication Push Access OPeNDAP THREDDS W10N Search OpenSearch Lucene Solr ElasticSearch Abstraction Java Python Ruby Groovy Scala Distribution Publication Push Access OPeNDAP THREDDS W10N Search OpenSearch Lucene Solr ElasticSearch Abstraction Java Python Ruby Groovy Scala Sources Images Measurements/ Observations Remote Sensing Text file/ ASCII Spreadsheets Ingest API Transfer Validation Harvesting Packaging Management Stewardship Curation RDBMS Postgres Oracle MySQL NoSQL MongoDB Cassandra Array SciDB Storage SAN S3 SSD Distribution Publication Push Access OPeNDAP THREDDS W10N Search OpenSearch Lucene Solr ElasticSearch Abstraction Java Python Ruby Groovy Scala Framework Analysis Platform Hadoop/HDFS MapReduce ZooKeeper Spark Graph DB TitanDB Neo4J Triple Store Virtuoso AllegroGraph Sesame Fuseki Message Passing Interface Virtual Machine Container Single Machine High Performance Computing GPU Query/ Retrieval API Lucene OpenSearch SPARQ Analysis Workflow LAS Analytics Machine Learning Pattern Recognition Climatologies Reduction Uncertainty Analysis Visualization OGC (WMS, WMTS, ) TWMS Slices Plots and Coordination Integrated Views Viewer and Interactive Query API Search Query Subset Analysis Platform Hadoop/HDFS MapReduce ZooKeeper Spark Graph DB TitanDB Neo4J Triple Store Virtuoso AllegroGraph Sesame Fuseki Message Passing Interface Virtual Machine Container Single Machine High Performance Computing GPU Analysis Platform Hadoop/HDFS MapReduce ZooKeeper Spark Graph DB TitanDB Neo4J Triple Store Virtuoso AllegroGraph Sesame Fuseki Message Passing Interface Virtual Machine Container Single Machine High Performance Computing GPU Analysis Platform Hadoop/HDFS MapReduce ZooKeeper Spark Graph DB TitanDB Neo4J Triple Store Virtuoso AllegroGraph Sesame Fuseki Message Passing Interface Virtual Machine Container Single Machine High Performance Computing GPU Query/ Retrieval API Lucene OpenSearch SPARQ Analysis Workflow LAS Analytics Machine Learning Pattern Recognition Climatologies Reduction Uncertainty Analysis Visualization OGC (WMS, WMTS, ) TWMS Slices Plots and Coordination Integrated Views Framework Query/ Retrieval API Lucene OpenSearch SPARQ Analysis Workflow LAS Analytics Machine Learning Pattern Recognition Climatologies Reduction Uncertainty Analysis Visualization OGC (WMS, WMTS, ) TWMS Slices Plots and Coordination Integrated Views Framework Query/ Retrieval API Lucene OpenSearch SPARQ Analysis Workflow LAS Analytics Machine Learning Pattern Recognition Climatologies Reduction Uncertainty Analysis Visualization OGC (WMS, WMTS, ) TWMS Slices Plots and Coordination Integrated Views Framework Viewer and Interactive Query API Search Query Subset Viewer and Interactive Query API Search Query Subset Viewer and Interactive Query API Search Query Subset Example Instantiation EarthCube Cyberinfrstructure Satellite Information Systems EarthCube Infrastructure EarthCube Repository EarthCube Analytics Centers Research EarthCube Management Node Analytic Node Research EarthCube Repository Airborne Agency arth Archives EarthCube Repository EarthCube Management Node EarthCube Management Node EarthCube Management Node EarthCube Discipline-Specific Management with Analytic Node Analytic Node Analytic Node Applications Applications Decision Suppor Provider Applied Other Systems (In-Situ, University) 1/7/16 25

26 Future Plans and Thoughts! Submitted two new proposals to NSF! Water Analytics! Architecture Operations! Important for JPLers to see that they can participate in data science research with other agencies 1/7/16 26

27 Thank You Questions? 1/7/16 27