FedX: Optimization Techniques for Federated Query Processing on Linked Data. ISWC 2011 October 26 th. Presented by: Ziv Dayan

Transcription

1 FedX: Optimization Techniques for Federated Query Processing on Linked Data ISWC 2011 October 26 th Presented by: Ziv Dayan Andreas Schwarte 1, Peter Haase 1, Katja Hose 2, Ralf Schenkel 2, and Michael Schmidt 1 1 fluid Operations AG, Walldorf 2 Max-Planck Institute for Informatics, Saarbrücken

2 Outline Linked Data Introduction Movie Semantic Web Basics Motivation Federated Query Processing FedX Query Processing Model Optimization Techniques Evaluation Conclusion & Outlook 2

3 Semantic Web Technologies RDF subject-predicate-object triples with URIs RDFS and OWL adding ontologies and ontological reasoning to data management (ideas from Knowledge Management) classes and sub-class relationships SPARQL a query language for RDF 3

4 RDF vs. Relational EmployeeID FirstName LastName Salary 7 John Doe Mary Major Richard Miles DeptNo The columns = properties The rows = values of entities The intersection = the value of that property for that entity Employee 17 FirstName Mary 4

5 RDF Triples Employee Subject17 LastName Property FirstName Salary Major Value Mary Alternatively: Subject-Predicate-Object 5

6 URIs Subjects and Properties must be URIs URIs serve as globally unique primary keys /employees/ Major Mary

7 URIs + Namespaces Namespaces a notation for abbreviating URIs = emp = ex emp:id17 employees/id17 ex:lastname ex:firstname Major Mary ex:salary

8 URIs Linking Data EmployeeID FirstName LastName Salary 7 John Doe Mary Major Richard Miles DeptNo DepartmentNo Name ManagerID 1 Sales 17 2 Accounting 8 emp:id7 ex:firstname John ex:department dept:id1 ex:deptname Sales 8

9 emp:id7 ex:firstname John ex:department ex:salary ex:lastname Doe dept:id1 ex:deptname Sales ex:department emp:id17 ex:manager ex:salary ex:firstname ex:lastname Mary Major emp:id777 ex:firstname Richard ex:department ex:salary ex:lastname Miles dept:id ex:deptname Accounting 9

10 Motivation Query processing involving multiple distributed data sources, e.g. Linked Open Data cloud New York Times DBpedia Query both data collections in an integrated way 10

11 SPARQL SPARQL SPARQL Federated Query Processing Federation mediator at the server Virtual integration of (remote) data sources Communication via SPARQL protocol Data Source Query Federation Mediator Data Source Data Source 11

12 Federated Query Processing Example Query Find US presidents and associated news articles SELECT?President?Party?TopicPage WHERE {?President rdf:type dbpedia-yago:presidentsoftheunitedstates.?nytpresident owl:sameas?president.?president dbpedia:party?party.?nytpresident nytimes:topicpage?topicpage. } 12

13 SPARQL SPARQL Federated Query Processing Query: SELECT?President?Party?TopicPage WHERE {?President rdf:type dbpedia-yago:presidentsoftheunitedstates.?nytpresident owl:sameas?president.... } DBpedia Federation Mediator Barack Obama George W. Bush...?President rdf:type dbpedia-yago:presidentsoftheunitedstates. Barack Obama George W. Bush... NYTimes 13

14 SPARQL SPARQL Federated Query Processing Query: SELECT?President?Party?TopicPage WHERE {?President rdf:type dbpedia-yago:presidentsoftheunitedstates.?nytpresident owl:sameas?president.... } DBpedia Federation Mediator yago:obama Input:?nytPresident owl:sameas Barack Obama. Barack Obama George W. Bush... NYTimes Output: Barack Obama, yago:obama Barack Obama, nyt:obama nyt:obama 14

15 SPARQL SPARQL Federated Query Processing Query: SELECT?President?Party?TopicPage WHERE {?President rdf:type dbpedia-yago:presidentsoftheunitedstates. }... DBpedia Federation Mediator?nytPresident owl:sameas George W. Bush. Input: Barack Obama George W. Bush... NYTimes Output: Barack Obama, yago:obama Barack Obama, nyt:obama George W. Bush, nyt:bush nyt:bush... and so on for the other intermediate mappings and triple patterns... 15

16 FedX Query Processing Model Requirments Efficient SPARQL query processing on multiple distributed sources Data sources are known and accessible as SPARQL endpoints FedX is designed to be fully compatible with SPARQL 1.0 No a-priori knowledge about data sources No local preprocessing of the data sources required No need for pre-computed statistics On-demand federation setup 16

17 Challenges to Federated Query Processing 1) Involve only relevant sources in the evaluation Avoid: Subqueries are sent to all sources, although potentially irrelevant 2) Compute joins close to the data Avoid: All joins are executed locally in a nested loop fashion 3) Reduce remote communication Avoid: Nested loop join that causes many remote requests 17

18 Optimization Techniques 1. Source Selection: Triple patterns are annotated with relevant sources Sources that can contribute information for a particular triple pattern SPARQL ASK requests in conjunction with a local cache After a warm-up period the cache learns the capabilities of the data sources During Source Selection remote requests can be avoided 2. Exclusive Groups: Group triple patterns with the same single relevant source Evaluation in a single (remote) subquery Push join to the endpoint 18

19 Optimization Techniques (2) Example: Source Selection + Exclusive Groups SELECT?President?Party?TopicPage WHERE {?President rdf:type dbpedia-yago:presidentsoftheunitedstates.?president dbpedia:party?party.?nytpresident owl:sameas?president.?nytpresident nytimes:topicpage?topicpage. } Source DBpedia Exclusive DBpedia, NYTimes Advantages: Avoid sending subqueries to sources that are not relevant Delegate joins to the endpoint by forming exclusive groups (i.e. executing the respective patterns in a single subquery) 19

20 Optimization Techniques (3) 3. Join Order: Iteratively determine the join order based on count-heuristic: Count free variables of triple patterns and groups Consider "resolved" variable mappings from earlier iteration Example: Find all departments with managers named "Yaron" SELECT?depName WHERE {?department ex:departmentname?depname.?person ex:firstname Yaron.?department ex:manager?person. } Free Vars

21 Optimization Techniques (4) 4. Bound Joins: Compute joins in a block nested loop fashion: Reduce the number of requests by "vectored" evaluation of a set of input bindings Renaming and Post-Processing technique for SPARQL

22 Optimization Techniques (5) Example: Find all employees with salary SELECT?S?O WHERE {?S salary ?S name?o. } Block Input emp1 emp2 emp3 Data Source Triples t1 = (emp1, name, Peter ) t2 = (emp3, name, Andreas ) Expected Results?S?O emp1 Peter emp3 Andreas Bound Join Subquery SELECT?O_1?O_2?O_3 WHERE { { emp1 name?o_1 } UNION { emp2 name?o_2 } UNION { emp3 name?o_3 } } Subquery Results?O_1?O_2?O_3 Peter Andreas 22

23 Optimization Example 1 SPARQL Query 2 Unoptimized Internal Representation Compute Micronutrients using Drugbank and KEGG 4x Local Join SELECT?drug?title WHERE { =?drug drugbank:drugcategory drugbank-cat:micronutrient.?drug drugbank:casregistrynumber?id.?keggdrug rdf:type kegg:drug.?keggdrug bio2rdf:xref?id.?keggdrug dc:title?title. } 4x NLJ 3 Optimized Internal Representation Exlusive Group Remote Join 23

24 Evaluation Based on FedBench benchmark suite 14 queries from the Cross Domain (CD) and Life Science (LS) collections Federation with 5 (CD) and 4 (LS) data sources Real-World Data from the Linked Open Data cloud Queries vary in complexity, size, structure, and sources involved 24

25 Evaluation (2) a) Evaluation times of Cross Domain (CD) and Life Science (LS) queries 25

26 Evaluation (3) b) Number of requests sent to the endpoints Runtimes AliBaba: >600s DARQ: >600s FedX: 0.109s Runtimes AliBaba: >600s DARQ: 133s FedX: 1.4s 26

27 Conclusion FedX: Efficient SPARQL query processing in federated setting Available as open source software: Implementation as a Sesame SAIL Novel join processing strategies, grouping techniques, source selection Significant improvement of query performance compared to state-of-the-art systems On-demand federation setup without any preprocessing of data sources Compatible with existing SPARQL 1.0 endpoints 27

28 Outlook SPARQL 1.1 Federation Extension in Sesame 2.6 Ongoing integration of FedX optimization techniques into Sesame Federation Extensions in FedX SERVICE keyword for optional user input to improve source selection BINDINGS (VALUES) clauses for bound joins (for SPARQL 1.1 endpoints) (Remote) statistics to improve join order 28

29 THANK YOU

30 FedX The Bigger Picture Information Workbench: Integration of Virtualized Data Sources as a Service Application Layer Semantic Wiki Collaboration Reporting & Analytics Visual Exploration Virtualization Layer Transparent & On-Demand Integration of Data Sources Data Layer SPARQL Endpoint SPARQL Endpoint SPARQL Endpoint Data Source Data Source Data Source Metadata Registry Data Registries CKAN, data.gov, etc. + Enterprise Data 30

31 Query Cross Domain 3 (CD3) Find US presidents, their party and associated news SELECT?president?party?page WHERE {?president < < < < < < < } 31

32 Query Life Science 3 (LS3) Find drugs and interaction drugs (+ the effect) SELECT?Drug?IntDrug?IntEffect WHERE {?Drug < < < < < < } 32

33 Benchmark Environment HP Proliant 2GHz 4Core, 32GB RAM 20GB RAM for server (federation mediator) Local copies of the SPARQL endpoint to ensure reproducibility and reliability of the service Provided by the FedBench Framework 33