Cloud scaling of Visual Weather

Transcription

1 Cloud scaling of Visual Weather Jozef Matula CTO 1 EGOWS Reading, United Kingdom, 15 th -17 th October 2018

2 Cloud - Separation of responsibilities 2

3 3

4 4

5 5

6 NWP in Cloud idea Storage Services ~10TB of gridded NWP data per day Compressed to 6TB in NetCDF 6

7 Cloud - Scalable technologies Amazon AWS Azure Google Cloud Object Storage S3 (Simple Storage Service) Azure Blob Storage Cloud Storage File Storage Elastic Block Storage Elastic File System Premium Storage Account Cloud Storage + FUSE adapter Queue/Messaging Simple Queue Service Simple Notification Service Storage queues Service Bus Task Queue Cloud Pub/Sub Database Aurora DynamoDB Elasticsearch SQL Database DocumentDB Cloud SQL Cloud Datastore Archive/Backup 7 Serverless Code Glacier Azure Backup Cloud Storage Coldline Lambda Azure Functions App Engine

8 Cloud - Scalable technologies Amazon AWS Azure Google Cloud Object Storage S3 (Simple Storage Service) Azure Blob Storage Cloud Storage File Storage Elastic Block Storage Elastic File System Premium Storage Account Cloud Storage + FUSE adapter Queue/Messaging Simple Queue Service Simple Notification Service Storage queues Service Bus Task Queue Cloud Pub/Sub Database Aurora DynamoDB Elasticsearch SQL Database DocumentDB Cloud SQL Cloud Datastore Archive/Backup 8 Serverless Code Glacier Azure Backup Cloud Storage Coldline Lambda Azure Functions App Engine

9 Load balancer (API Gateway) Cloud processing plan WMS WMS Object Store Service Gridded Elastic Database WMS WCS AWS SNS/SQS Metadata Elastic Search AWS S3 WCS Client AWS S3 Client 9

10 Gridded Elastic Database (G.E.D.) AWS SQS ASW SNS AWS DynamoDB Shredder Shredder Client #1 Shredder Client #2.. Shredder Client #X Unique numbers AWS Auto Scaling Group 10 Metadata AWS Elastic Search AWS S3

11 G.E.D. client Metadata AWS Elastic Search AWS S3 Grid Processor Reprojection, Time and spatial interpolation 11 Any kind of existing VW services: WCS, WMS, WPS RIMA

12 G.E.D. client scaling Metadata AWS Elastic Search AWS S3 Grid Processor Grid Processor Grid Processor.. AWS Auto Scaling Group 12

13 GED Auto Scaling Group Peak time examples, 4 shredder clients are running 13

14 GED performance monitoring 14

15 15

16 Cons of Visual Weather in the Cloud: Very big code base (2GB) 16

17 Cons of Visual Weather in the Cloud: Very big code base (2GB) Inflexible and relatively slow deployment (a lot of configurations) 17

18 18 A new product targeting: Cloud deployments, Cloud scalability & API standardisation

19 Open Weather... Open Weather does/is: Based on Visual Weather code base Include GED (database + client) Include web service suite (RIMA, WCS, WMS) Open Weather does not: Include majority of VW server components (native database, ingestion and telecommunication subsystem) Include user interface of VW Include traditional installation and configuration process (replaced by pure git deployment of configuration) 19

20 Open Weather editions Open Weather Community Edition: local GRIB/NetCDF ingestion GED client Includes WCS and RIMA web service (IBL OGC-like grid processing API) Limited support Open Weather Standard Edition: Includes GED database (the Shredder component) Includes full OGC web service suite (WMS, WCS, WPS) Python API SLA support 20

21 Load balancer (API Gateway) Cloud processing plan WMS WMS Object Store Service Gridded Elastic Database WMS WCS AWS SNS/SQS Metadata Elastic Search AWS S3 WCS Client AWS S3 Client 21

22 Load balancer (API Gateway) Cloud processing plan - Serverless λ λ Client Client Object Store Service Gridded Elastic Database λ λ Client Client AWS SNS/SQS Metadata Elastic Search AWS S3 λ Client λ Client AWS S3 λ Client 22

23 Open Weather Lambda Shrank down version of Open Weather allowing serverless grid processing (because of 250MB AWS Lambda limit). Includes GridProcessor capabilities operating over GED client connectivity No Web Service APIs, Lambda is a web service over our Python API. 23

24 Questions? 24