Stream and Complex Event Processing Discovering Exis7ng Systems:c- sparql

Transcription

1 Stream and Complex Event Processing Discovering Exis7ng Systems:c- sparql G. Cugola E. Della Valle A. Margara Politecnico di Milano Vrije Universiteit Amsterdam

2 Agenda Introduction RDF Stream Analysis Running example C-SPARQL language Query and Stream Registration FROM STREAM Clause TimeStamp Function Accessing background Information Resources Stream & Complex Event Processing - Introduc7on 2

3 Introduc7on C- SPARQL Engine Architecture Simple, modular architecture It relies en7rely on exis7ng technologies Integra7on of DSMSs (Esper) and SPARQL engines (Jena- ARQ) Stream & Complex Event Processing - Introduc7on 3

4 Introduc7on C- SPARQL Engine Features at a glance 1/2 Efficient RDF stream Processing Con7nuous queries, filtering, aggrega7ons, joins, sub- queries via C- SPARQL (a SPARQL 1.1 extension) Push based High throughput, low latency Minimal support for payern detec7on via!mestamp func7on Minimal support for "sta7c" RDF graph access (a.k.a., background informa7on) Alert! using 7mestamp func7on and combining with "sta7c" RDF graphs spoil performances Stream & Complex Event Processing - Introduc7on 4

5 Introduc7on C- SPARQL Engine Features at a glance 2/2 Extensible Middleware Run7me management of RDF streams C- SPARQL query Result listerners API driven Web APIs for data flow management and service layer Quick start available hyp://streamreasoning.org/download Stream & Complex Event Processing - Introduc7on 5

6 Introduc7on C- SPARQL Engine abstrac7ons CsparqlEngine (eu.larkc.csparql.engine.csparqlengine) Contains a stream and a query registry RDF Stream (eu.larkc.csparql.cep.api.rdfstream) A stream of Timestamped Triples (eu.larkc.csparql.cep.api.rdfquadruple) C- SPARQL Queries (eu.larkc.csparql.streams.formats.csparqlquery) Con7nuously executed against one or more RDF Stream May refer to "sta7c" RDF graphs Can be started/stopped Result Proxy (eu.larkc.csparql.engine.csparqlqueryresultproxy) Allows mul7ple clients to observe the results of C- SPARQL queries Result FormaYer (eu.larkc.csparql.core.resultformayer) Allows to process the results of a query (RDFTable) a row (RDFTuble) at a 7me Stream & Complex Event Processing - Introduc7on 6

7 Introduc7on Combining C- SPARQL Engine abstrac7ons The C- SPARQL Engine processing model is con7nuous RDF Streams, C- SPARQL queries, result proxies and result formayers are manually connected to form graphs RDF Stream RDF Stream C- SPARQL query Result Proxy Result FormaYer Result FormaYer Stream & Complex Event Processing - Introduc7on 7

8 Introduc7on Combining C- SPARQL Engine abstrac7ons: code CsparqlEngine engine = new CsparqlEngineImpl(); engine.ini7alize(true); engine.registerstream(<<rdf Stream>>); CsparqlQueryResultProxy c = engine.registerquery(<<query>>); c1.addobserver(<<formayer>>);. engine.unregisterquery(c.getid()); engine.unregisterstream(<<rdf Stream IRI>>); See also HelloWorldCSPARQL.java in the C- SPARQL ready to go pack Stream & Complex Event Processing - Introduc7on 8

9 Introduc7on Chaining C- SPARQL queries A special result formayers (RDFStreamFormaYer) can inject the results of C- SPARQL queries, whose form is STREAM, into another RDF streams. RDFStreamFormaYer MEMO: RDF streams are not typed Stream & Complex Event Processing - Introduc7on 9

10 Introduc7on Chaining C- SPARQL queries: code CsparqlEngine engine = new CsparqlEngineImpl(); engine.ini7alize(true); engine.registerstream(<<rdfstream1>>); RdfStream s2; s2 = new RDFStreamForma5er(<<RDFstream2>>); engine.registerstream(s2); CsparqlQueryResultProxy c1; CsparqlQueryResultProxy c2; c1 = engine.registerquery(<<query on RDFstream1>>); c1.addobserver((rdfstreamforma5er) s2); c2 = engine.registerquery(<<query Down Stream>>); c2.addobserver(<<forma9er>>); See also the COMPOSABILITY case in HelloWorldCSPARQL.java in the C- SPARQL ready to go pack Stream & Complex Event Processing - Introduc7on 10

11 Running Example following TwiYerUser sr:screenname(xsd:string) posts likes Tweet sr:messageid(xsd:string) sr:messagetimestamp(xsd:string) talksabout talksaboutposi7vely Topic Stream & Complex Event Processing - Introduc7on 11

12 Streaming vs. Background Informa7on User related background information following& TwiBerUser& sr:screenname(xsd:string)& posts& Streaming information likes& Tweet& sr:messageid(xsd:string)& sr:messagetimestamp(xsd:string)& talksabout& talksaboutposi>vely& Topic& Topic related background information

13 Background Informa7on - example User related background information, e.g. :Alice a :TwitterUser. :Bob a :TwitterUser. :Bob :follows :Alice. See also hyp://streamreasoning.org/larkc/csparql/lbsma- sta7c- k.rdf Which is used in the STREAMING_AND_EXTERNAL_STATIC_RDF_GRAPH case of HelloWorldCSPARQL.java in the C-SPARQL ready to go pack Stream & Complex Event Processing - Introduc7on 13

14 Streaming Informa7on RDF Stream Data Type Ordered sequence of pairs, where each pair is made of an RDF triple and its timestamp Timestamps are not required to be unique, they must be non- decreasing E.g., (<:Alice :posts :post1 >, T13:34:41) (<:post1 :talksaboutpositively :LaScala>, T13:34:41) (<:Bob :posts :post2 >, T13:36:28) (<:post2 :talksaboutpositively :Duomo>, T13:36:28) See also LBSMARDFStreamTestGenerator.java in the ready to go pack Stream & Complex Event Processing - Introduc7on 14

15 C- SPARQL language C- SPARQL queries derive and aggregate informa7on from one or more RDF streams to join or merge RDF streams can access "sta7c" RDF graphs and to feed results from one RDF stream to subsequent C- SPARQL query Stream & Complex Event Processing - Introduc7on 15

16 MEMO: SPARQL Stream & Complex Event Processing - Introduc7on 16

17 Where C- SPARQL Extends SPARQL 17

18 C- SPARQL language C- SPARQL is an extension of SPARQL 1.1 C- SPARQL queries Adds to SPARQL 1.1 query forms the STREAM form Changes the seman7cs of SPARQL 1.1 query forms from the one- 7me seman7cs to the con7nuous one (i.e., instantaneous) and Adds to SPARQL 1.1 datasets clauses the FROM STREAM one Adds a built- in to access the 7mestamp of a triple Stream & Complex Event Processing - Introduc7on 18

19 Query and Stream Registra7on 19

20 Query and Stream Registra7on All C- SPARQL queries over RDF streams are conhnuous Registered through the REGISTER statement The output of queries is in the form of Instantaneous tables of variable bindings Instantaneous RDF graphs RDF stream Only queries in the the CONSTRUCT form can be registered as generators of RDF streams Composability: Query results registered as streams can feed other registered queries just like every other RDF stream Stream & Complex Event Processing - Introduc7on 20

21 Query registration - Example What objects do TwiYer's users like? REGISTER QUERY WhatUserLikes AS PREFIX ex: < SELECT?u?o FROM STREAM < /twitterstream> [RANGE 1s STEP 1s] WHERE {?u :post?t.?t :talksaboutpositively?o. } GROUP BY?o NOTE: The resul7ng variable bindings has to be interpreted as an instantaneous. It expires as soon as the query is recomputed Stream & Complex Event Processing - Introduc7on 21

22 Stream registration - Example Results of a C- SPARQL query can be stream out for down stream queries REGISTER STREAM WhatUserLikes AS PREFIX ex: < CONSTRUCT {?u ex:likes?o } FROM STREAM < /twitterstream> [RANGE 1s STEP 1s] WHERE {?u :post?t.?t :talksaboutpositively?o. } GROUP BY?o NOTE: all the queries that follow are assumed to consume the results of this query that is streamed in an RDF graph whose IRI is Stream & Complex Event Processing - Introduc7on 22

23 Stream Registra7on - Notes The output is constructed in the format of an RDF stream. Every query execu7on may produce from a minimum of zero triples to a maximum of an en7re graph. The 7mestamp is always dependent on the query execu7on 7me only, and is not taken from the triples that match the payers in the WHERE clause. Stream & Complex Event Processing - Introduc7on 23

24 FROM STREAM Clause 24

25 FROM STREAM Clause FROM STREAM clauses are similar to SPARQL datasets They idenhfy RDF stream data sources They represent windows over a RDF stream They define the RDF triples available for querying and filtering. Stream & Complex Event Processing - Introduc7on 25

26 FROM STREAM Clause - windows physical: a given number of triples logical: a variable number of triples which occur during a given time interval (e.g., 1 hour) Sliding: they are progressively advanced of a given STEP (e.g., 5 minutes) Tumbling: they are advanced of exactly their time interval Stream & Complex Event Processing - Introduc7on 26

27 FROM STREAM Clause - Windows Grammar of the FROM STREAM clause 'AS' Name The op7onal NAMED keyword works exactly like when applied to the standard SPARQL FROM clause for tracking the provenance of triples. It binds the IRI of a stream to a variable which is later accessible through the GRAPH clause. Stream & Complex Event Processing - Introduc7on 27

28 FROM STREAM Clause - Example How many users like the same object? Does the coun7ng on a window of 5 seconds that slides every seconds REGISTER QUERY HowManyUsersLikeTheSameObj AS PREFIX ex: < SELECT?o (count(?s) as?countusers) FROM STREAM < [RANGE 5s STEP 1s] WHERE {?s ex:likes?o } GROUP BY?o See also the HOW_MANY_USERS_LIKE_THE_SAME_OBJ case in HelloWorldCSPARQL.java in the C- SPARQL ready to go pack Stream & Complex Event Processing - Introduc7on 28

29 C- SPARQL reports only snapshots Incoming 7mestamped RDF triples t Time window [RANGE 4s STEP 1s] At t+4 At t+5 At t+6 At t+7 At t+8 Results t+1 t+2 t+3 W 1 t+4 W 1 W 1 W 1 W 1 W 1 W 1 W 2 t+5 W 2 W 2 W 2 W 2 W 1, W 2 t+6 W 3 W 3 W 1, W 2 W 3 t+7 W 1, W 2, W 3 t+8 W 2, W 3 Stream & Complex Event Processing - Introduc7on 29

30 Mul7ple FROM STREAM Clause - Example C- SPARQL queries can combine triples from mul7ple RDF stream, e.g., trendy objects on mul7ple social networks REGISTER QUERY TrendyObjectsOnMultipleSN AS PREFIX ex: < SELECT?o (count(?s) as?countusers) FROM STREAM < [RANGE 5s STEP 1s] FROM STREAM < [RANGE 5s STEP 1s] WHERE {?s ex:likes?o } GROUP BY?o ORDER BY DESC(?countUsers) See also the MULTI_STREAM case in HelloWorldCSPARQL.java in the C- SPARQL ready to go pack Stream & Complex Event Processing - Introduc7on 30

31 Mul7ple Times the same stream - Example Emerging as trendy objects REGISTER QUERY TrendyObjectsOnMultipleSN AS PREFIX ex: < SELECT?o (count(?s1)/count(?s2) as?index) FROM NAMED STREAM < [RANGE 5s STEP 1s] AS SHORT FROM NAMED STREAM < [RANGE 5s STEP 1s] AS LONG WHERE { WINDOW SHORT {?s1 ex:likes?o } WINDOW LONG {?s2 ex:likes?o } } GROUP BY?o HAVING (?index>0.9) ORDER BY DESC(?index) ALERT: this query is not supported by the current C- SPARQL Engine Stream & Complex Event Processing - Introduc7on 31

32 TimeStamp Func7on 32

33 TimeStamp Func7on Syntax and Seman7cs The 7mestamp of a triple can be bound to a variable using a timestamp() func7on Syntax 7mestamp(variable IRI, variable IRI, variable IRI literal) Seman7cs Triple It is not in the window Result of evalutaion Type Error It appears once in the window Timestamp of triple It appears mul7ple 7mes in the window The 7mestamp ofmost recent triple Stream & Complex Event Processing - Introduc7on 33

34 TimeStamp Func7on and CEP The 7mestamp func7on can be used to model 7me rela7onships used in CEP E.g. A -> B in C- SPARQL?a rdf:type :A.?b rdf:type :B. FILTER ( ) timestamp(?a,rdf:type,:a) < timestamp(?b,rdf:type,:b) Stream & Complex Event Processing - Introduc7on 34

35 TimeStamp Func7on - Example Who are the opinion makers? i.e., the users who are likely to influence the behavior of other users REGISTER QUERY FindOpinionMakers AS PREFIX f: < PREFIX ex: < SELECT?opinionMaker?o (COUNT(?follower) AS?n) FROM STREAM < [RANGE 10s STEP 5s] WHERE {?opinionmaker ex:likes?o.?follower ex:likes?o. FILTER(?opinionMaker!=?follower) FILTER (f:timestamp(?follower,ex:likes,?o) > f:timestamp(?opinionmaker,ex:likes,?o)) } GROUP BY?opinionMaker?o HAVING (COUNT(?follower)>3) See also the FIND_OPINION_MAKERS case in HelloWorldCSPARQL.java in the C- SPARQL ready to go pack Stream & Complex Event Processing - Introduc7on 35

36 Accessing background Informa7on Accessing background informa7on in DSMSs and CEPs is problema7c and it can spoil performances C- SPARQL allows for loading external RDF graphs using the SPARQL 1.1 FROM clauses Keep the external RDF files small!!! For a beyer solu7on see: Le- Phuoc, D., Dao- Tran, M.: A na7ve and adap7ve approach for unified processing of linked streams and linked data. In: Interna7onal Seman7c Web Conference (ISWC 2011). Volume 1380., Bonn, Germany, Springer (2011) Stream & Complex Event Processing - Introduc7on 36

37 Accessing background Informa7on - Example Who are the opinion makers? i.e., the users who are likely to influence the behavior of other users who follow them REGISTER QUERY StreamingAndExternalStaticRdfGraph AS PREFIX f: < PREFIX ex: < SELECT?opinionMaker?o (COUNT(?follower) AS?n) FROM STREAM < [RANGE 10s STEP 5s] FROM < WHERE {?opinionmaker ex:likes?o.?follower ex:follows?opinionmaker.?follower ex:likes?o. FILTER(?opinionMaker!=?follower) FILTER (f:timestamp(?follower,ex:likes,?o) > f:timestamp(?opinionmaker,ex:likes,?o)) } GROUP BY?opinionMaker?o HAVING (COUNT(?follower)>3) See also the STREAMING_AND_EXTERNAL_STATIC_RDF_GRAPH case in HelloWorldCSPARQL.java in the C- SPARQL ready to go pack Stream & Complex Event Processing - Introduc7on 37

38 Resources Download the C- SPARQL Ready To Go Pack hyp://streamreasoning.org/download See demos hyp://streamreasoning.org/demos Read out more D. F. Barbieri, D. Braga, S. Ceri, E. Della Valle, M. Grossniklaus, Querying RDF streams with C- SPARQL, SIGMOD Record 39 (1) (2010) Contact point Stream & Complex Event Processing - Introduc7on 38