Processing Queries on Top of Linked Data and Sensor Data

Transcription

1 Processing Queries on Top of Linked Data and Sensor Data Cry Distribution Marcel Karnstedt Copyright All rights reserved.

2 Linked Data Use URIs as names for things (documents, people, organisations, products, ) Use HTTP URIs so that people can look up those names When someone looks up a URI, provide useful information, typically structured data in RDF Include links to other URIs, so that they can discover more things

3 Accessing Linked Data

4 Linked Data in Practice User Agent HTTP GET RDF Web Server

5 Forward References User Agent HTTP GET 303 HTTP GET RDF Web Server

6 Consumption - Essentials Linked Data provides for a global data-space with a uniform API (due to RDF as the data model) Access methods Dereference URIs via HTTP GET (RDF/XML, RDFa, etc.) SPARQL ( the SQL of RDF ) Data dumps (RDF/XML, etc.) Metadata about LOD datasets void ( Allows to select datasets based on their characteristics (topic, license, interlinking, formats, etc.)

7 Consumption - Technologies Basic Linked Data access mechanisms widely supported in all major platforms and languages (HTTP interface & RDF parsing), such as Java, PHP, C/C++/.NET, etc. Inspect and debug tools Command line tools (curl, rapper, etc.) Online tools (HTTP/low-level) (RDF/data-level) SPARQL endpoints (generic and dataset-specific)

8 Gimme URIs!! Distributed setup need for central point of access (indexer, aggregator) Sindice, an index of the Web of Data Sig.ma, Web of Data aggregator & browser Relationship discovery But where is the DB?! Complex queries, efficient storage, quick access, etc.

9 Ranking Huge data, millions of sources, messiness, RDF model, Requires special ranking TF-IDF style, Graph based (PageRank style), Cardinality based (histogram style), etc. Triple level URI level Document level Source level Domain level

10 Skyline Ranking Objects that are not dominated by other objects Scoring function on multiple attributes, no weighting In contrast to (multidimensional) top-k distance age price dominated objects No straightforward IR operation price time

11 Querying SELECT?f?n WHERE { an:f#ah foaf:knows?f.?f foaf:name?n. } SELECT?x1?x2 WHERE { dblppub:hoganhp08 dc:creator?a1,?a2.?x1 owl:sameas?a1.?x2 owl:sameas?a2.?x1 foaf:knows?x2.?x2 foaf:knows?x1. }?f?n

12 Query Algebra WHERE {?s 1 <p 1 >?o 1.?s 2?p 2?o 2.?s 2 <add>?add. FILTER (edist(?o 1,?o 2 ) < k). FILTER?p 2 =<pred>} ω(?s 2 ;[?s 2?add <add> ]) (edist(?o 1,?o 2 )<k) ξ([?s 1?o 1 <p 1 > ]) σ(?p 2 =<pred>) ξ([?s 2?p 2?o 2 <pred> ]) ξ([?s 2?p 2?o 2 ])

13 Querying Data Across Sources Data warehousing or materialisation-based approaches (MAT) CRAWL INDEX SERVE RDBMS One big table Property tables Vertical storage, column stores Hybrid approaches, such as Virtuoso Native stores, such as YARS Special (simplified) structures, special indexes!

14 Indexing in YARS etc. Index the different parts of a triple Ideally: all combinations <x> name <x> knows <x> seealso <x> sameas Xavier <friend> <link> <y> Optimised for read-only access With prefix support: only 6 (spo, sop, pso, pos, osp, ops) Trade-off: storage vs. performance, read vs. write Optional special indexes (full-text, string similarity, )

15 Live Queries SELECT * FROM R S R ODBC S ODBC Live lookups, on-demand querying SELECT?s WHERE TP TP TP HTTP GET TP HTTP GET 1 5

16 Live Queries: Approaches SELECT?f?n WHERE { an:f#ah foaf:knows?f.?f foaf:name?n. } Direct lookups dereferencing URIs, recursive Data summaries TP (an:f#ah foaf:knows?f) TP (?f foaf:name?n) Select source(s) HTTP GET RDF HTTP GET RDF Select source(s) ?f?n Andreas Harth Data Dan Brickley

17 Federated/Distributed Querying Federated feature in SPARQL1.1 Directly refer to sources Automatically split/copy queries and forward Depends on query capabilities! Simple sources vs. SPARQL end points etc. Similar issues as in central engines Indexing, local stores, New challenges Availability, guarantees, robustness, consistency,

18 Basic federated Queries (time permitting) Will be integrated in Query spec Essentially new pattern SERVICE Similar to GRAPH allows delegate query parts to a specific (remote) endpoint PREFIX foaf: < SELECT?N FROM < WHERE { { < foaf:knows?f.?f foaf:name?n } UNION { SERVICE < { [ foaf:maker < /authors/tim_berners-lee>, [ foaf:name?n ] ]. } } }

19 Total Decentralisation DB Find all reviews for movies made in 1994 in central Europe! RDF: Geonames data RDF: IMDB data RDF: reviews

20 Indexing in UniStore Use distributed hash tables (DHT) Indexing of attributes = key for Hashing Which attributes? All! <x> name <x> knows <x> seealso <x> sameas Xavier <friend> <link> <y> h(s) for subject lookup h(p 1 o 1 ) for?s p i?o. h(p 2 o 2 ) filter (?o v)... (prefix search)...trade-off storage vs. performance

21 Robustness: Parallel Execution Goal: stateless processing push approach Messages containing both plan and intermediate results (based on Mutant Query Plans [Papadimos et al. 02]) Receiver peer is identified by applying the hash function Multiple instances of the plan travel trough the network p 0 σ(a) σ(a) σ(a) B B B {(A,1),(A,2)} p 1 p 2 {(A,3),(A,4)} p 3 {(A,5),(A,6)} {(A,5)} {(A,2)} {(B,2),(C,1),(B,2),(C,4)} p 4 {(A,3),(A,4)} p 5 {(B,5),(B,6)} B B B {(A,2,B,2,C,1), (A,2,B,2,C,4)} p 0 {(A,5),(B, 5)}

22 Less Bandwidth: Sequential Execution All peers can be queried in a sequence Decision at each peer: adaptive query processing σ(a) B {(A,1),(A,2)} p 1 {(A,2)} B {(B,2),(C,1),(B,2),(C,4)} p 4 {(A,2,B,2,C,1),(A,2,B,2,C,4),...} p 0 {(A,2),(A,3),(A,4)} p 2 {(A,3),(A,4)} B p 3 {(A,5),(A,6)} p 5 {(B,5),(B,6)} p 0 {(A,2),(A,3),(A,4),(A,5,B,5)} B {(A,2),(A,3),(A,4),(A,5)} B

23 Mixed Query Execution May result in unpredictable behavior Fire and forget A peer may see the same query multiple times Different data to process Different operators to process {(A,1),(A,2)} σ(a) B p 1 {(A,2)} B {(B,2),(C,1),(B,2),(C,4)} p 4 {(A,2,B,2,C,1), (A,2,B,2,C,4)} p 0 {(A,2),(A,3),(A,4)} p 2 {(A,3),(A,4)} B p 3 {(A,5),(A,6)} p 5 {(B,5),(B,6)} {(A,2),(A,3),(A,4),(A,5)} B p 0 {(A,5),(B, 5)}

24 Reasoning

25 In Principle Machine-interpretable representation of data allows for deductive reasoning Drawing conclusions from axioms and data Web Ontology Language (OWL) provides constructs supporting entity consolidation Same as, inverse-functional properties (mbox_sha1sum) Reasoning can further be used to: Unite fractured data sets Disambiguate entities Check consistency of knowledgebases

26 Example General Idea: Answer Queries with implicit answers Simplified example: :jeff rdfs:type foaf:person & :jeff foaf:knows :aidan query: select?x {?x rdfs:type foaf:agent } foaf:person rdfs:subclassof foaf:agent :jeff as result query: select?x {?x rdfs:type foaf:person } foaf:knows rdfs:range foaf:person :jeff and :aidan as result Inverse-functional properties, sameas, subpropertyoff etc.

27 Problems Usually expensive, huge amount of new facts Potential conflicts due to inconsistencies Potentially infinite results Ontology hijacking e.g. foaf:person subclassof my:person A new statement for each Person in the dataset Non-distinguished variables SELECT?X {?X :hasfather?y } No such triple in the data, but every person has a father?! 08445a31a78661b5c746feff39a9db6e4e2cc5cf sha1-sum of mailto:

28 Implementation Materialise inferred triples Forward chaining Query rewriting, recursive/iterative Backward chaining On-the-fly with data summaries Ongoing research Stateless query expansion Parallel sub-queries

29 Query Expansion Unexpanded query Map operators added First mapping Expanded query

30 Querying Sensor Data

31 Processing Paradigms

32 AnduIN

33 CQL

34 In-Network Query Processing

35 Example Anomalies in sensor networks Sensors deliver measurements y s 8 s 6 s 5 s 25 s 2 s 7 s 26 s 27 s 3 s 1 s 24 s 12 s 28 s 10 s 4 s 13 s 11 s 38 s 37 s 9 s 17 s 16 s 15 s 31 s 29 s 30 s 19 s 23 s 35 s 36 s 39 s 34 s 14 s 18 s 41 s s s 20 s 22 s 32 s 33 x

36 Example Anomalies in sensor networks Identify anomalies from the stream y s 8 s 6 s 5 s 25 s 2 s 7 s 26 s 27 s 3 s 1 s 24 s 12 s 28 s 10 s 4 s 13 s 11 s 38 s 37 s 9 s 17 s 15 s 16 s 31 s 29 s 30 s 19 s 23 s 35 s 36 s 39 s 34 s 14 s 18 s 41 s s s 20 s 22 s 32 s 33 x

37 Example Anomalies in sensor networks Determine anomalous regions y s 8 s 6 s 5 s 25 s 2 s 7 s 26 s 27 s 3 s 1 s 24 s 12 s 28 s 10 s 4 s 13 s 11 s 38 s 37 s 9 s 17 s 15 s 16 s 31 s 29 s 30 s 19 s 23 s 35 s 36 s 39 s 34 s 14 s 18 s 41 s s s 20 s 22 s 32 s 33 x

38 Example Anomalies in sensor networks Respect obstacles y s 8 s 6 s 5 s 25 s 2 s 7 s 26 s 27 s 3 s 1 s 24 s 12 s 28 s 10 s 4 s 13 s 11 s 38 s 37 s 9 s 17 s 15 s 16 s 31 s 29 s 30 s 19 s 23 s 35 s 36 s 39 s 34 s 14 s 18 s 41 s s s 20 s 22 s 32 s 33 x

39 Example Anomalies in sensor networks Obstacles change regions y s 8 s 6 s 5 s 25 s 2 s 7 s 26 s 27 s 3 s 1 s 24 s 12 s 28 s 10 s 4 s 13 s 11 s 38 s 37 s 9 s 17 s 15 s 16 s 31 s 29 s 30 s 19 s 23 s 35 s 36 s 39 s 34 s 14 s 18 s 41 s s s 20 s 22 s 32 s 33 x

40 Example Scenario Storms in California

41 Anomaly Degrees and Regions Regions for different thresholds Triangulated Wireframe Surface (TWS) Degree plane at height 0.25 Degree plane at height 0.4

42 IN Region Detection Focus on energy consumption But cost model supports multiple dimensions

43 Cost Estimation In streams: continuous queries Thus, query planning should be adaptive Not optimise first, execute next any more When and how to re-optimise Forecast, e.g., by exponential smoothing Decide between alternative query plans

44 Eval: Anomalies Anomaly rate and size of sliding window

45 Eval: Anomalies /2 Number of leader nodes

46 Eval: Anomalous Region Anomaly rate

47 Brief Wrap-Up Different approaches for querying SemWeb data Linked Data is inherently distributed...but not inherently dynamic?! Distributed approaches promising: Support dynamic data Scalable Query processing in sensor networks shows similarities Scalability requirements advise to focus on distributed approaches, resource limitations demand it