Spatio-Temporal Gridded Data Processing on the Semantic Web

Transcription

1 Spatio-Temporal Gridded Data Processing on the Semantic Web Andrej Andrejev, Dimitar Misev*, Peter Baumann*, Tore Risch Department of Information Technology, Uppsala University * Computer Science & Electrical Engineering Department, Jacobs University, Bremen andrej.andrejev@it.uu.se 1/28

2 Introduction They never come alone... 2/28

3 Introduction Big Data.. 3/28

4 Introduction Z X Namely: Y Big Data.. Massive Numeric Datasets typically ordered along a number of orthogonal axes 3/28

5 Introduction Z X Namely: Y Big Data.. Massive Numeric Datasets typically ordered along a number of orthogonal axes 3/28

6 Introduction They come together... 4/28

7 Introduction... with Metadata 5/28

8 Introduction Z X Y... with Metadata 5/28

9 Introduction Z X Domain Y axis names origins low and high bounds... 6/28

10 Introduction Z Range X field descriptions accuracy and noise reserved values... Domain Y 7/28

11 Introduction Z Range X Domain Y Master instance 8/28

12 Introduction Z Range X Domain Y Master instance 8/28 Provenance input parameters tools & methods used...

13 Introduction They stay together... 9/28

14 Introduction They stay together since our data model is RDF with Arrays 9/28

15 RDF with Arrays databases RDF is very suitable for describing properties about scientific experiments (metadata) but: Arrays are represented in a very inefficient way in RDF Our approach: Extend RDF with compact numerical array representation 10/28

16 Scientific SPARQL SPARQL is very suitable for searching scientific RDFbased metadata, but: SPARQL has no support for queries involving array operations Our approach: Extent SPARQL with common array operators => SciSPARQL 11/28

17 Reusing program libraries Often need for using existing program libraries when processing experiments data, but: SPARQL has no standard way of plugging in external program libraries and algorithms Our approach: SciSPARQL provides a general mechanism to call functions in C, Java, Python, or Matlab 12/28

18 Our System Architecture SciSPARQL queries / updates SciSPARQL results SSDM SciSPARQL Database Manager Python, Java, MATLAB,.. engines In-memory database RDF importer functional extensibility interface External functions generic storage backend/wrapper interface RasQL wrapper 13/28

19 Our System Architecture SciSPARQL queries / updates SciSPARQL results SSDM SciSPARQL Database Manager Python, Java, MATLAB,.. engines In-memory database RDF importer functional extensibility interface Retrieving array data generic storage backend/wrapper interface External functions RasQL wrapper 13/28

20 Our System Architecture SciSPARQL queries / updates SciSPARQL results SSDM SciSPARQL Database Manager Python, Java, MATLAB,.. engines In-memory database RDF importer functional extensibility interface Retrieving array data generic storage backend/wrapper interface External functions RasQL wrapper Delegating common array computations 13/28

21 Introducing Geo Coverages 14/28

22 A Coverage... 15/28

23 Coverages A Coverage is an abstract data structure that represents some space/time varying phenomenon 15/28

24 Coverages A Coverage is an abstract data structure that represents some space/time varying phenomenon According to ISO and its implementation model by Open Geospatial Consortium: Coverage subtypes are available for regular and irregular grids, point clouds and meshes 16/28

25 Coverages A Coverage is an abstract data structure that represents some space/time varying phenomenon According to ISO and its implementation model by Open Geospatial Consortium: Coverage subtypes are available for regular and irregular grids, point clouds and meshes 16/28

26 A Grid Coverage 17/28

27 Grid Coverages coverage id A Grid Coverage is identified by domain set, specifying spatio-temporal region of interest geometric grid: integer coordinates rectified grid: grid origin and offset vector in some CRS referenceable grid: CRS coordinates for each position range type: structural description and technical metadata, required of appropriate understanding of a coverage 17/28

28 Grid Coverages coverage id A Grid Coverage is identified by domain set, specifying spatio-temporal region of interest geometric grid: integer coordinates rectified grid: grid origin and offset vector in some CRS referenceable grid: CRS coordinates for each position range type: structural description and technical metadata, required of appropriate understanding of a coverage A Range Type is composed of one or more fields, each having its own: name identifier human-readable description type definition unit of measure list of reserved values... 17/28

29 Grid Coverages Grid Coverage Ontology axisnames dimensionality id value reason GridDomain GridCoverage RangeType NILValue low CRS origin high ReferenceableGridDomain RectifiedGridDomain description Field name Range offset coordinategrid definition isoptional isupdatable IntervalRange EnumeratedRange TypedRange low high values datatype 18/28

30 Grid Coverages Grid Coverage Ontology axisnames dimensionality id value reason GridDomain GridCoverage RangeType NILValue low CRS origin high ReferenceableGridDomain RectifiedGridDomain description Field name Range offset Flexibility of RDF coordinategrid definition isoptional isupdatable low IntervalRange TypedRange EnumeratedRange datatype high values 18/28

31 SciSPARQL query example 19/28

32 Normalized Difference Vegetation Index (NDVI) DEFINE FUNCTION NDVI(?nir?red?x) AS SELECT (255 * xsd:integer(((?nir -?red) / (?nir +?red)) >?x) AS?result) SciSPARQL Example Create 0.5-threshold NDVI map of the coverage identified as "mycoverage", using its :nir and :red properties SELECT (MAP(xsd:integer, NDVI(*, *, 0.5),?nir_array,?red_array) AS?result) WHERE {?c :id "mycoverage" ; :nir?nir_array ; :red?red_array } 20/28

33 Normalized Difference Vegetation Index (NDVI) DEFINE FUNCTION NDVI(?nir?red?x) AS SELECT (255 * xsd:integer(((?nir -?red) / (?nir +?red)) >?x) AS?result) SciSPARQL Example Create 0.5-threshold NDVI map of the coverage identified as "mycoverage", using its :nir and :red properties SELECT (MAP(xsd:integer, NDVI(*, *, 0.5),?nir_array,?red_array) AS?result) WHERE {?c :id "mycoverage" ; :nir?nir_array ; :red?red_array } :real 20/28

34 Normalized Difference Vegetation Index (NDVI) DEFINE FUNCTION NDVI(?nir?red?x) AS SELECT (255 * xsd:integer(((?nir -?red) / (?nir +?red)) >?x) AS?result) SciSPARQL Example :Boolean Create 0.5-threshold NDVI map of the coverage identified as "mycoverage", using its :nir and :red properties SELECT (MAP(xsd:integer, NDVI(*, *, 0.5),?nir_array,?red_array) AS?result) WHERE {?c :id "mycoverage" ; :nir?nir_array ; :red?red_array } 21/28

35 DEFINE FUNCTION NDVI(?nir?red?x) AS SELECT (255 * xsd:integer(((?nir -?red) / (?nir +?red)) >?x) AS?result) SciSPARQL Example Normalized Difference Vegetation Index (NDVI) either 0 or 255 :integer Create 0.5-threshold NDVI map of the coverage identified as "mycoverage", using its :nir and :red properties SELECT (MAP(xsd:integer, NDVI(*, *, 0.5),?nir_array,?red_array) AS?result) WHERE {?c :id "mycoverage" ; :nir?nir_array ; :red?red_array } 22/28

36 "mycoverage" SciSPARQL Example Normalized Difference Vegetation Index (NDVI) DEFINE FUNCTION NDVI(?nir?red?x) AS SELECT (255 * xsd:integer(((?nir -?red) / (?nir +?red)) >?x) AS?result) either 0 or 255 :integer Create 0.5-threshold NDVI map of the coverage identified as "mycoverage", using its :nir and :red properties SELECT (MAP(xsd:integer, NDVI(*, *, 0.5),?nir_array,?red_array) AS?result) WHERE {?c :id "mycoverage" ; :nir?nir_array ; :red?red_array }?nir_array :id?red_array?c :nir :red 22/28

37 "mycoverage" SciSPARQL Example Normalized Difference Vegetation Index (NDVI) DEFINE FUNCTION NDVI(?nir?red?x) AS SELECT (255 * xsd:integer(((?nir -?red) / (?nir +?red)) >?x) AS?result) either 0 or 255 :integer Create 0.5-threshold NDVI map of the coverage identified as "mycoverage", using its :nir and :red properties SELECT (MAP(xsd:integer, NDVI(*, *, 0.5),?nir_array,?red_array) AS?result) WHERE {?c :id "mycoverage" ; lexical closure :nir?nir_array ; :red?red_array }?nir_array :id?red_array?c :nir :red 22/28

38 "mycoverage" SciSPARQL Example Normalized Difference Vegetation Index (NDVI) DEFINE FUNCTION NDVI(?nir?red?x) AS SELECT (255 * xsd:integer(((?nir -?red) / (?nir +?red)) >?x) AS?result) either 0 or 255 :integer Create 0.5-threshold NDVI map of the coverage identified as "mycoverage", using its :nir and :red properties SELECT (MAP(xsd:integer, NDVI(*, *, 0.5),?nir_array,?red_array) AS?result) WHERE {?c :id "mycoverage" ; lexical closure :nir?nir_array ; :red?red_array }?nir_array free arguments - bound by the mapper :id?red_array?c :nir :red 22/28

39 "mycoverage" SciSPARQL Example Normalized Difference Vegetation Index (NDVI) DEFINE FUNCTION NDVI(?nir?red?x) AS SELECT (255 * xsd:integer(((?nir -?red) / (?nir +?red)) >?x) AS?result) either 0 or 255 :integer Create 0.5-threshold NDVI map of the coverage identified as "mycoverage", using its :nir and :red properties SELECT (MAP(xsd:integer, NDVI(*, *, 0.5),?nir_array,?red_array) AS?result) WHERE {?c :id "mycoverage" ; lexical closure :nir?nir_array ; :red?red_array }?nir_array free arguments - bound by the mapper?c :id :nir :red?red_array MAP NDVI(*, *, 0.5)?result 22/28

40 "mycoverage" SciSPARQL Example Normalized Difference Vegetation Index (NDVI) DEFINE FUNCTION NDVI(?nir?red?x) AS SELECT (255 * xsd:integer(((?nir -?red) / (?nir +?red)) >?x) AS?result) either 0 or 255 :integer Create 0.5-threshold NDVI map of the coverage identified as "mycoverage", using its :nir and :red properties SELECT (MAP(xsd:integer, NDVI(*, *, 0.5),?nir_array,?red_array) AS?result) WHERE {?c :id "mycoverage" ; lexical closure :nir?nir_array ; :red?red_array }?nir_array free arguments - bound by the mapper?c :id :nir :red?red_array MAP NDVI(*, *, 0.5)?result 22/28

41 ?c SciSPARQL Example Normalized Difference Vegetation Index (NDVI) DEFINE FUNCTION NDVI(?nir?red?x) AS SELECT (255 * xsd:integer(((?nir -?red) / (?nir +?red)) >?x) :id AS?result) :nir :red either 0 or 255 MAP NDVI(*, *, 0.5) :integer Create 0.5-threshold NDVI map of the coverage identified as "mycoverage", using its :nir and :red properties, and insert it as :ndvimap property INSERT {?c :ndvimap?result } WHERE {?c :id "mycoverage" ; :nir?nir_array ; :red?red_array. BIND (MAP(xsd:integer, NDVI(*, *, 0.5), "mycoverage"?nir_array,?red_array) AS?result) }?nir_array?red_array :ndvimap?result 23/28

42 Comparison Doing the same thing in a programming language of your choice? 24/28

43 Normalized Difference Vegetation Index (NDVI) DEFINE FUNCTION NDVI(?nir?red?x) AS SELECT (255 * xsd:integer(((?nir -?red) / (?nir +?red)) >?x) AS?result) Create 0.5-threshold NDVI map of the coverage identified as "mycoverage", using its :nir and :red properties Find the right instance of a Gridded Coverage and all the associated data needed for the task Comparison "mycoverage"?nir_array?c :id :nir :red?red_array MAP NDVI(*, *, 0.5)?result 25/28

44 "mycoverage" Normalized Difference Vegetation Index (NDVI) DEFINE FUNCTION NDVI(?nir?red?x) AS SELECT (255 * xsd:integer(((?nir -?red) / (?nir +?red)) >?x) AS?result) Create 0.5-threshold NDVI map of the coverage identified as "mycoverage", using its :nir and :red properties Find the right instance of a Gridded Coverage and all the associated data needed for the task Comparison?nir_array Load Load & validate?c :id :nir :red?red_array MAP NDVI(*, *, 0.5)?result 25/28

45 "mycoverage" Normalized Difference Vegetation Index (NDVI) DEFINE FUNCTION NDVI(?nir?red?x) AS SELECT (255 * xsd:integer(((?nir -?red) / (?nir +?red)) >?x) AS?result) Create 0.5-threshold NDVI map of the coverage identified as "mycoverage", using its :nir and :red properties Find the right instance of a Gridded Coverage and all the associated data needed for the task Comparison?nir_array Load Load & validate Allocate?c :id :nir :red?red_array MAP NDVI(*, *, 0.5)?result 25/28

46 "mycoverage" Normalized Difference Vegetation Index (NDVI) Create 0.5-threshold NDVI map of the coverage identified as "mycoverage", using its :nir and :red properties Find the right instance of a Gridded Coverage and all the associated data needed for the task Comparison DEFINE FUNCTION NDVI(?nir?red?x) AS SELECT (255 * xsd:integer(((?nir -?red) / (?nir +?red)) >?x) AS?result)?nir_array Load Load & validate Hardcode a nested loop across all array elements Allocate?c :id :nir :red?red_array MAP NDVI(*, *, 0.5)?result 25/28

47 Benefits of combining data and metadata within a single query: More transparent and self-contained queries are easier to maintain Result selection and postprocessing on the server whenever metadata is needed, it is in the same place Single round-trip to the server instead of retrieving metedata in order to guide the data retrieval and processing More freedom for the optimizer one complex query is better than a series of simple ones 26/28

48 Summary of contributions RDF with Arrrays a flexible way to model Gridded Coverages with any application-relevant metadata Hybrid data store storing RDF graphs in-memory and gridded data on disk (distributed and parallel) SciSPARQL queries combining graph patterns and array operations: metadata conditions and data filtering / postprocessing Mediator architecture performing computations where the data are 27/28

49 The software, documentation, and examples are available at This work is supported by 28/28