Processing and analysis of Earth Observation data

Transcription

1 Processing and analysis of Earth Observation data Carsten Brockmann, Brockmann Consult GmbH ESA Climate Change Initiative Toolbox Science Lead Big Data Analytics & GIS, Münster September 2017.

2 Earth Observation Managing big EO data is increasingly complex. But not just technically. Collaboration. Culture. Organisation. Structure.

3 Lake MacKay, Australia

4 Tonga, Pacific

5 Arctic Ocean

6 Karavasta Lagoon, Albania

7 Satellite Images = Measurement Data

8 Satellite Images = Measurement Data Generic Tools Data selection & access Visualisation Analysis Turning image Processing data into information Export Instrument specific tools System correction Data processing (L1 > L2) Thematic processing, synergy Programming tools API for python and Java Scripting Graph model builder

9 SNAP Architecture Sentinel-1 Toolbox (S1TBX) SNAP Desktop Sentinel-2 Toolbox (S2TBX) SNAP Engine Sentinel-3 Toolbox (S3TBX) Any combination of toolboxes add-ons is allowed, even none, as SNAP Desktop is a already a useful stand-alone application for EO data exploitation. SNAP layer NetBeans RCP GeoTools JAI NetCDF 3rd-party library layer Java SE 8 Platform Python Programming language layer

10 SNAP Architecture Sentinel-1 Toolbox (S1TBX) SNAP Desktop Sentinel-2 Toolbox (S2TBX) SNAP Engine Sentinel-3 Toolbox (S3TBX) Any combination of toolboxes add-ons is allowed, even none, as SNAP Desktop is a already a useful stand-alone application for EO data exploitation. SNAP layer NetBeans RCP GeoTools JAI NetCDF 3rd-party library layer Java SE 8 Platform Python Programming language layer

11 SNAP Application Modes

12 Golden Age of Earth Observation By the end of 2017, the operational Sentinel-1, -2, -3 and -5p satellites alone will continuously collect a volume of 27 Terabytes per day / 10PB per year. It could take around 2.5 years to download 1 Petabyte of Sentinel-1 data and a staggering 63 years to preprocess it on your own computer (Wagner, 2015)

13 Data Local Processing Optimising data transfer Sharing input data Sharing result Rapid turn-around cycles

14 Data Local Processing Archive-centric approach Network storage Data are transferred over the network Risk of network bottleneck Hadoop approach Concurrent data-local processing Tasks are transferred over the network Good scalability

15 Calvalus Processing System for EO Data Apply Apache Hadoop to earth observation Transfer the algorithm to the data (data-local in a narrower sense) Avoid an archive-centric approach Calvalus On-demand Portal Calvalus Adapters Calvalus Bulk Production Add Calvalus software layers for EO data processing and validation EO processing workflows Data processor plug-in framework Bulk production control SNAP GPF Operators and Graphs SNAP Aggregators Linux Executables Portal Integrate data processors Linux executables SNAP & BEAM GPF operators and aggregators Ppen for other frameworks Calvalus Adapters

16 Data Local Processing - Level 2 Processing Workflow on Calvalus - MERIS RR L1, North Sea, 3 days CoastColour NN L2 processor 6 minutes (22 nodes) output: L2 files Only Mapper Tasks, no reduce step necessary L1 File L2 Processor (Mapper L2 Processor Task) (Mapper L2 Processor Task) (Mapper L2 Processor Task) (Mapper Task) distributed file system HDFS L1 File on local disks of compute nodes L1 File transparent, optimised data-local L1 File access L1 File L2 Processor (Mapper Task) L2 File L2 File L2 File L2 File L2 File

17 Map-Reduce on Calvalus Algorithm defined by Google employees Dean + Ghemawat in 2004 Idea: partition data into chunks, compute chunks locally (map), concatenate intermediate results to final result (reduce) Allows for high degree of parallelisation Fits very well with principle of data locality distributed file system HDFS on local disks of compute nodes transparent, optimised data-local access

18 Map-Reduce on Calvalus - Temporal and Spatial Integration - MERIS RR L1, global, 10-day SNAP C2RCC Water processor 20 mins (100 nodes) Output: 1 L3 product L3 File L3 Formatting L2 Proc. + Spat. L1 File L2 Binning Proc. + Spat. distributed L1 File file system (Mapper HDFS Binning Task) on local L1 File disks of compute (Mapper nodes Binning Task) transparent, L1 Fileoptimised data-local L1 File access L2 Proc. + Spat. L2 Proc. + Spat. (Mapper L2 Binning Task) Proc. + Spat. (Mapper Binning Task) (Mapper Task) Spatial Bins Spatial Bins Spatial Bins Spatial Bins Spatial Bins Temp. Binning (Reducer Temp. Binning Task) (Reducer Task) Temp. Bins Temp. Bins

19 Map-Reduce on Calvalus - Match-up analysis - MERIS RR L1, global, 3 months CoastColour C2W processor NOMAD in-situ dataset 1.5 minutes (100 nodes) Output: scatter-plots and pixel extraction tables L1 File L1 File L1 File L1 File L1 File L2 Proc. & Matcher L2 (Mapper Proc. & Task) Matcher L2 (Mapper Proc. & Task) Matcher L2 (Mapper Proc. & Task) Matcher L2 (Mapper Proc. & Task) Matcher (Mapper Task) Output Records Output Records Output Records Output Records Output Records Matchup Analysis (Reducer Task) Input Records MA Report

20 Streaming on Calvalus - With SNAP - Supported by SNAP Graph Processing Framework Access to data via reader/writer objects instead of files Operator chaining to build processors from modules Tile cache and pull principle for in-memory processing Hadoop MapReduce for partitioning and streaming

21 EO Data & Data-Processing Platforms European Space Agency & national Space Agencies Thematic Exploitation Platforms Mission Exploitation Platforms European Commission: Copernicus Data and Information Access Services (DIAS) Copernicus Collaborative Ground Segments Private offers Google Earth Engine Amazon Web Services

22 The Urban Thematic Exploitation Platform Visualisation & Analysis gateway to... Urban TEP Processing Centres Urban TEP portal + gateway

23 Portal functions Processing request forms and result access Datasets and services Geo-browser

24 Analysis and visualisation Combination of satellite products and socio-economic data Derivation of new criteria

25 Processing request form

26 GUF Istanbul Moscow Sao Paulo Global binary raster mask showing location of human settlements (12m/75m)

27 Urban growth SAR4Urban ( ) Beijing ERS-2 PRI & ASAR IMP VV m spatial resolution 48 scenes

28 Urban growth Beijing S1A IW GRDH VV m spatial resolution 31 scenes

29 Urban TEP is... attractive high-quality datasets that meet space, time and feature dimensions of the domain the capability to generate them the facilities to access and use them,... to generate more of them

30 Processor development model Local test processing Package Upload Deployment VM for download Processing Request Urban TEP portal Concurrent processing Urban TEP processing centres Browser for request submission

31 Systematic or on-demand processing Datasets may be pre-generated, providing access to them as product for long-running processes for global datasets with high complexity/information reduction in order to be able to visualise them Example: GUF Datasets may be processed on-demand, providing a service instead for short-running processes for selected areas in case of user-defined parameterisation to avoid storage of large output datasets Example: Sentinel-2 timescan service (unless generated systematically)

32 Urban TEP processing centres IT4Innovations Brockmann Consult DLR cluster (Salomon HPC) cluster (Calvalus/Hadoop) YARN scheduler virtualised env. (GeoFarm) +cluster(calvalus/hadoop) - Sentinel-2 (urban areas, Africa), OLCI, MERIS Sentinel-1 and other datasets Geoserver WPS + own backend implementation Calvalus WPS + Urban TEP config+extension - Geoserver WMS Geoserver WMS - large-scale global Landsat timescan processing fast internet access, HPC. host of portal and analysis/visualisation GUF subsetting, Sentinel-2 timescan processing distributed data-local processing and concurrent aggregation GUF and other Urban datasets systematic generation of datasets

33 Copernicus Data and Exploitation Platform Deutschland National entry point to the EU Copernicus Sentinel Satellite Systems, their data products and the products of the Copernicus Services Processing facilities on the platform

34 EU DIAS

35 Confusing? YES!

36 Climate Monitoring Data Climate change is a global challenge. Open climate data is crucial.

37 ESA Climate Change Initiative (CCI) The objective of the Climate Change Initiative (CCI) is to realise the full potential of the long-term global Earth Observation archives that ESA together with its Member States have established over the last 30 years, as a significant and timely contribution to the Essential Climate Variable databases required by the United Nations Framework Convention on Climate Change (UNFCCC).

38 ESA Climate Change Initiative (CCI) 16 projects >300 scientists >100 organisations 18 countries Since 2009

39 7 years. X individuals X organisations X projects

40

41 Climate Monitoring Data An Overview of Climate Data Production.

42 Step 1. Deciding what to actually measure. Essential Climate Variables have been defined by the global science community to support the United Nations Framework Convention on Climate Change (UNFCCC).

43 Criteria of Essential Climate Variables (ECV) Relevance. Critical for climate monitoring. Feasibility. Global measurement is feasible. Cost effective. Using proven technology.

44 Satellites can help Global. Observe entire Earth. Uniformity. Same instrument everywhere. Rapid Measurement. Constant watch. Continuity. Long time series to monitor change in climate.

45 Step 2. Get the raw satellite data. Current data. Archived data. Planning for the future.

46 Example

47 Step 3. Process the data. Gridding, Homogenisation, Calibration & Validation, Quality. Scientific processing - application of state-of-the art algorithms distilled from the very latest scientific reasoning.

48 Step 4. Distribution of Climate Data Products. Just give me the data.

49 ESA Climate Change Initiative (CCI) Managing Complexity of Climate Data Production. Open Data Challenges & Approaches

50 Managing Open Data Complexity Meaningfulness & Community.

51 Managing Open Data Complexity Ease of Data Access.

52

53

54 Managing Open Data Complexity Bespoke Open Standards.

55 Managing Open Data Complexity Machine & Human Readable Standards.

56 Managing Open Data Complexity Interoperability & Collaboration.

57 Managing Open Data Complexity Open-source Tooling

58 CATE Climate Analysis Toolbox for ESA A software to facilitate processing and analysis of all the data products generated by the ESA Climate Change Initiative Programme (CCI).

59 ESA UNCLASSIFIED - For Official Use

60 ESA Open Data Portal and other data services Process 2 Process 3 Python Core Lib (API) Plugin 1 Plugin 2 Web Service (WebAPI) { RESTful } Desktop App (GUI) Python Core Lib (API) Plugin 1 Plugin 2 Command-Line Data App sources (CLI) Operations Workflows Process 1 61

61 Cate Desktop 62

62 Browse datasets published by CCI Open Data Portal Download full datasets or just subsets Temporal subset Spatial subset Variable subset Manage also your local data sources 63

63 64

64 65

65 66

66 67

67 Every operation is a new workflow step Workflows can be executed from Python or from the Command-Line Interface (CLI) 68

68 Python Programming & Batch Processing Exported Python Code Exported CLI Calls 69

69 Execution Scenarios Cate Desktop Cate Desktop Cate Desktop 70

70 71

71 Processing and analysis of Earth Observation data Managing big EO data is increasingly complex. But not just technically. Collaboration. Culture. Organisation. Structure. Communication. Learning. Exchange. Sentinel Toolbox SNAP: step.esa.int CCI Toolbox: github.com/cci-tools/cci-tools.github.io ect-core.readthedocs.io cci-tools.github.io Earth System Datacube: earthsystemdatacube.net/