1 ST ESA SNAP HACKATHON

Transcription

1 1 ST ESA SNAP HACKATHON SNAP Architecture SNAP Development Team ESA ESRIN, October 2015

2 SNAP Architecture Dynamic, module-based architecture, with various extension points and extensions Install and update of extension modules Use from Java and Python programs, extend by Java and Python plugins Write a SNAP plugin, use it in all toolboxes High-level architecture comprises two subsystems SNAP Engine, the core and command-line interface SNAP Desktop, the graphical user interface

3 SNAP Architecture Sentinel-1 Toolbox (S1TBX) SNAP Desktop Sentinel-2 Toolbox (S2TBX) SNAP Engine Sentinel-3 Toolbox (S3TBX) Any combination of toolboxes add-ons is allowed, even none, as SNAP Desktop is a already a useful stand-alone application for EO data exploitation. SNAP layer NetBeans RCP GeoTools JAI NetCDF 3rd-party library layer Java SE 8 Platform Python Programming language layer

4 Graph Processing Framework Majority of SNAP functions are implemented as operators Each operator can be invoked from SNAP Desktop and from the command line Processing chains ( graphs ) are configured in XML files Graphical Graph Builder Graph Processing Tool (gpt) for executing of graphs (chains) >gpt -help

5 Generic Functions and Tools Applicable for all toolboxes and wide range of sensors Raster data and vector data tools Visualisation Multi-layer displays, layer editors Image, mask, shapes overlays Colour management, fast navigation Data Analysis Various statistics and plot types Spectrum display (optical) Data processing Reprojection, Collocation, Mosaicing Level-3 processor Graph processing, ürocessing graph builder

6 SNAP Basic Concept SST ECV Reader Impl. v1.2 GeoTIFF Writer Impl. v1.0.1 Gap-Filling Alg. Impl. v2.1 Gap-Filling GUI Impl. v2.2 Reader API Writer API Operator API RCP Product Data Model NetBeans Platform SNAP Engine SNAP Desktop

7 Basic Concepts SNAP has been designed as a EO raster data visualisation, analysis and processing system However vector data is supported through an OGC compatible model (GeoTools) In SNAP, almost all EO data processing is done through a pull-processing approach: If some raster data is required, such as for display or processing, it is obtained from some raster data source The source can be a product reader that reads from a file, or a raster data operator that computes its output from other raster data This way, raster data read/computation requests are made from one raster data node to another forming directed acyclic imaging graphs

8 Basic Concepts Product Data Model Product : has a file-name, type-name has a scene raster size has a GeoCoding has a model coordinate reference system (CRS) has any number of RasterDataNodes has any number of VectorDataNodes has MetadataElements RasterDataNode: has a name, unit, description has a raster data type and size provides raster data can be a TiePointGrid, Band, VirtualBand, FilteredBand, Mask

9 Basic Concepts Product Data Model TiePointGrid: a grid that is up-sampled by linear interpol. to a larger raster size Band: receives its raster data on-demand from a ProductReader or an Operator or a MultiLevelImage VirtualBand: a band which computes its raster data on demand from an band maths expression FilteredBand: a band which computes its raster data on demand from filter kernels Mask: a band with 1-byte Boolean pixels which can be used as ROI

10 Basic Concepts - Behaviour ProductWriter: 1. writes header once (structure + metadata) 2. writes raster data to output on demand 3. requests raster data from sources Operator: 1. One target product, N source products 2. computes raster data for target on demand 3. requests raster data from sources ProductReader: 1. reads product header once (structure + metadata) 2. reads raster data regions from input on demand

11 Programming Language SNAP is programmed entirely in Java. Why? Java is cool and beautiful. Java is also very fast (as C), secure, and truly platform independent. So we have a SNAP Java API SNAP now supports also Python. Why? Python is cool and beatiful, and most of you use it. However, for raster data processing plain Python is very, very slow. Unless you use numpy and the like. Therefore SNAP supports two Python implementations CPython 2.7, 3.3, 3.4 to use with numpy etc ( snappy) Jython 2.7 to develop also SNAP GUI extensions There is no dedicated SNAP Python API Instead we reuse the SNAP Java API from Python

12 SNAP API Use Cases Use case #1: Toolbox reuse SNAP calls into user s code via extension point API (e.g. Operator) User s code calls back into SNAP libraries via SNAP API Use case #2: SNAP extension User s Java Program User s program calls into SNAP libraries via SNAP API e.g. ProductIO.readProduct(..) User s Python Program SNAP Engine - Product - Band - TiePointGrid - - ProductIO - GPF - In-built or user s native Java Extension In-built or user s native Python extension In-built or user s executable extension Native R extension SNAP invokes external user tool with selected ECV file, e.g. a MATLAB program, then ingests generated outputs: stand-alone tools adapter

13 Development Environment SNAP 2.0 beta 8, the installed version Python coding Python 2.7, 3.3 or 3.4 git to clone code from github.com/senbox-org, especially snap-examples PyCharm, probably the best Python IDE Java coding JDK 1.8 update 60 IntelliJ IDEA for sure the best Java IDE Maven to compile and package Java sources

14 Java API Docs Java API Docs SNAP Engine SNAP Desktop Start Python coding by looking at the examples by looking up classes used in the Java API Docs How to interpret as Python developer?

15 Java API Docs - ProductIO

16 Java API Docs Python code From ProductIO: From Product: from snappy import ProductIO p = ProductIO.readProduct( /eodata/test.nc ) b13 = p.getband( radiance_13 )

17 Java API Docs Python (cont.) Java to (C)Python type conversions: Java java.lang.string boolean byte, short, int, long float, double byte[], short[], int[], long[], float[], double[] null value any java.lang.object Python str bool int float Python buffer protocol, Python sequence protocol e.g. pass to numpy array constructors None value Wrapped Java object

18 Product Data Model - Product Product * * VectorDataNode uuid: String filelocation: URI starttime: UTC stoptime: UTC * RasterDataNode

19 Product Data Model Metadata Node name: String 0..1 metadata MetadataElement 0..* children attributes 0..* MetadataAttribute value: Object

20 Product Data Model Node Groups Node 0..n children Group<T extends Node> getnodecount(): int getnode(index: int): T iterator(): Iterator<T> addnode(node: T) insertnode(i: int, node: T) removenode(node: T) 0..1 parent

21 Product Data Model Node Observation Product Data Model (SNAP-Engine) RCP (SNAP Desktop) register listener implementation ProductExplorerController Node NodeListener ProductExplorerNodeListener addnodelistener(l: NodeListener) removenodelistener(l: NodeListener) firenodechanged() firenodedatachanged() * nodelisteners nodechanged(e: NodeChangeEvent) nodedatachanged(e: NodeChangeEvent) nodesadded(e: NodeChangeEvent) nodesremoved(e: NodeChangeEvent) nodechanged(e: NodeChangeEvent) nodedatachanged(e: NodeChangeEvent) nodesadded(e: NodeChangeEvent) nodesremoved(e: NodeChangeEvent)

22 Product Data Model Vector Data Node featuretype 1 SimpleFeatureType Point Polyline Polygon MultiPolygon VectorDataNode 0..* features SimpleFeature Geometry MultiGeometry

23 Product Data Model - Raster Data RasterDataNode MultiLevelImage getlevelcount(): int getlevelimage(level:int): RenderedImage provide RenderedImage gettilewidth(): int gettileheight(): int getwidth(): int getheight(): int getdata(): Raster getdata(r: Rectangle): Raster gettile(tilex:int, tiley:int): Raster 1 sourceimage role: String rasterwidth: int rasterheight: int getgeophysicalimage(): MultiLevelImage getsourceimage(): MultiLevelImage setsourceimage(img: MultiLevelImage) createsourceimage(): MultiLevelImage readpixels(x:int, y:int, w:int, h:int, data:int[]) readpixels(x:int, y:int, w:int, h:int, data:float[]) readpixels(x:int, y:int, w:int, h:int, data:double[]) writepixels(x:int, y:int, w:int, h:int, data:int[]) writepixels(x:int, y:int, w:int, h:int, data:float[]) writepixels(x:int, y:int, w:int, h:int, data:double[]) 0..* ancillarybands Used to represent uncertainty information. TiePointGrid Band 0..1 validpixelmask VirtualBand FilteredBand Mask Used to reflect missing-data, e.g. given by a nodata-value or quality flag combintation, etc.

24 Product Data Model Flags and Index Coding Node MetadataElement SampleCoding rasterwidth: int rasterheight: int rastertype: int RasterDataNode flagcoding indexcoding FlagCoding IndexCoding

25 Product Data Model Geo Coding TiePointGeoCoding RasterDataNode rasterwidth: int rasterheight: int rastertype: int 0..1 geocoding GeoCoding getgeocrs(): CRS getmapcrs(): CRS getimagecrs(): CRS getpixelpos(g: GeoPos): PixelPos getgeopos(p: PixelPos): GeoPos PixelGeoCoding GcpGeoCoding CrsGeoCoding

26 Data Pull Processing SstCciProductReader: ProductReader reader FileReadOperator: Operator SubsetOperator: Operator AggregOperator: Operator GapFillOperator: Operator targetproduct operator targetproduct operator targetproduct operator targetproduct operator <CDM-Instance-1>: Product sourceproduct <CDM-Instance-2>: sourceproduct <CDM-Instance-3>: sourceproduct Product Product <CDM-Instance-4>: Product convert and write ECV data AnimGifWriter: ProductWriter