Spatio-Temporal Big Data: - the rasdaman approach

Transcription

1 Spatio-Temporal Big Data: - the rasdaman approach OGRS 2014 Otaniemi, Espoo, Finland, 10 June 2014 Peter Baumann Jacobs University rasdaman GmbH p.baumann@jacobs-university.de [animation: gamingfeeds.com]

2 Array DB Jacobs U Large-Scale Scientific Information Systems group massive n-d array services Main impact: pioneer Array DBMS, rasdaman Standardization: OGC WCS suite editor, ISO Array SQL ISO: member, SC32 / WG3 SQL; SC32 Big Data Study Group; OGC liaison, TC211 Open Geospatial Consortium: co-chair, BigData.DWG, WCS.SWG, Coverages.DWG; co-founder, Temporal.DWG Research Data Alliance: co-chair, Big Data Interest Group and Geospatial Interest Group member, ERCIM Expert Group Big Data member, Belmont Forum, WP 3 Harmonization of global environmental data infrastructure Charter Member, OSGeo council member, CGI / IUGS founding member and secretary, CODATA Germany...

3 Conference Announcements congrexprojects.com/2014- events/bigdatafromspace

4 Overview Big Data in geo? OGC Big Coverage Data standards Array Databases: the rasdaman approach Wrap-up

5 Big Data (not only) in Geo Volume Velocity Variety Veracity Ocean Science Interoperability Experiment [OGC]

6 Tackling Variety Stock trading: 1-D sequences (i.e., arrays) Social networks: large, homogeneous graphs Ontologies: small, heterogeneous graphs Climate modelling: 4D/5D arrays Satellite imagery: 2D/3D arrays (+irregularity) Genome: long string arrays Particle physics: sets of events analyzing domain standards (like OGC WCS) can help Bio taxonomies: hierarchies (such as XML) Documents: key/value stores: sets of unique identifiers + whatever etc.

7 Overview: Some Relevant OGC WGs WCS.SWG - WCS/WCPS maintenance Coverages.DWG - exchange with all interested domains Temporal.DWG - establishing semantics for time & calendars BigData.DWG - well...you guess it

8 Managed Variety: OGC Coverage Model [OGC r2] Coverage = regular & irregular grids, point clouds, meshes Fully n-d, spatio-temporal Can be served by WMS, WFS, WCS, WPS, SOS,...

9 (Part of) The OGC Standards Quilt data images data data feature coverage meta FE WCPS CQL WFS-T WCS-T CS-T WFS WMS WCS CS-W 9

10 OGC Web Coverage Service (WCS) Core: access & subsetting on spatio-temporal coverages + format transcoding subset = trim slice WCS 1.x grandfathered, WCS 2.0 maintained Extensions add functionality facets Scaling, CRS transformation,..., ad-hoc analytics Application Profiles bundle purpose-specific functionality EO-WCS, MetOcean-WCS 10

11 OGC WCPS OGC Web Coverage Processing Service (WCPS) - adopted 2008 = high-level grid coverage filtering & processing language "From MODIS scenes M1, M2, M3: difference between red & nir, as TIFF" but only those where nir exceeds 127 somewhere for $c in ( M1, M2, M3 ) where some( $c.nir > 127 ) return encode( $c.red - $c.nir, image/tiff ) (tiff A, tiff C ) 11

12 Case Study: Satellite Image Time Cube [Diedrich et al 2001]

13 My Database is Faster Than Your Web Server!

14 Case Study: Time Series Analysis Avg chlorophyll concentration for given area & time period, from x/y/t cube 10, 60,120, 240 days Conclusions: we must minimise data transfer as well as [client] processing standards such as WCPS provide the greatest benefit [Oliver Clements, Plymouth Marine Laboratory, EGU 2014]

15 Semantic Interoperability: WCPS vs WPS WCPS: semantics in query for $c in ( M1, M2, M3 ) return encode abs( $c.red - $c.nir ), image/tiff" ) WPS: semantics in human-readable text

16 OGC Coverage Service Portfolio OGC standards cover full range of coverage services, from data-intensive to processing-intensive WCS WCPS WPS data access ad-hoc analytics predefined process

17 OGC End-to-End Architecture for Coverages SOS SWE, SOS: upstream sensor data capturing & homogenization coverage server WMS WCS WPS WFS... WCS: downstream coverage download & processing & visualization

18 The rasdaman Array Database System raster data manager : SQL + n-d arrays select ls.green[x0:x1,y0:y1] > 130 from LandsatArchive as ls where avg_cells( ls.nir ) < 17 Scalable parallel tile streaming architecture In operational use OGC WCS Core Reference Implementation rasdaman website visitors

19 The rasql Query Language selection & section select c[ *:*, 100:200, *:*, 42 ] from ClimateSimulations as c result processing select img * (img.green > 130) from LandsatArchive as img search & aggregation select mri from MRI as img, masks as am where some_cells( mri > 250 and m ) data format conversion select encode( c[ *:*, *:*, 100, 42 ], image/png ) from ClimateSimulations as c PNG HDF NetCDF rasdaman DB PNG HDF NetCDF

20 Let s Take a Closer Look... t Divergent access patterns for ingest and retrieval Server must mediate between access patterns

21 Tiled Array Storage in rasdaman multidimensional array multidimensional tiles [Baumann 1994] = unit of access Index tiling strategies tuning storage layout language insert into MyCollection values... tiling area of interest [0:20,0:40], [45:80,80:85] tile size

22 Sample Application: Database Visualization select encode( struct { red: (char) s.image.b7[x0:x1,x0:x1], green: (char) s.image.b5[x0:x1,x0:x1], blue: (char) s.image.b0[x0:x1,x0:x1], alpha: (char) scale( d.elev, 20 ) }, "image/png" ) from SatImage as s, DEM as d [JacobsU, Fraunhofer; data courtesy BGS, ESA]

23 From Clouds to Federations ad-hoc federation of data centers, intelligent sensors,... autonomous heterogeneous Dataset D Dataset C Dataset A Dataset B

24 Next: On-Board Query Intelligence ESA Democratize direct data access NASA [imagery courtesy ESA, NASA]

25 Agile analytics on any-size spatio-temporal geo data EU FP7 INFRA, sep 2011 aug 2014, 11 partners, 5m budget 6 Lighthouse Applications covering Earth & Planetary Sciences Established data centers adding EarthServer technology to service portfolio Summer 2014: 260 TB operational Cryospheric Science landcover mapping Airborne Science high-altitude longendurance drones Atmospheric Science climate variables Geology geological models Oceanography marine model runs + in-situ data Planetary Science Mars geology

26 Related Work UDF techniques stay at surface: inconvenient syntax, difficult to scale PostGIS Raster: OR extensions, 2D, limited sizes, tiling is user task Oracle GeoSpatial: 2D, parallelized, no QL integration TeraData: UDT, 5D, limited sizes, tiling is user task; no geo SciDB: PostgreSQL + UDFs/ScaLAPACK SciQL: MonetDB column store rasdaman: nd, dedicated array engine, tile streaming

27 Summary n-d arrays major contributors to Big Data [ISO/IEC SC32 Study Group on Big Data, Amsterdam, 2014] Trend towards query languages Flexibility, scalability, information integration rasdaman: Agile Array Analytics through Array SQL n-d QL + parallel tile streaming architecture [rasdaman screenshots]