TriAD: A Distributed Shared-Nothing RDF Engine based on Asynchronous Message Passing

Transcription

1 TriAD: A Distributed Shared-Nothing RDF Engine based on Asynchronous Message Passing Sairam Gurajada, Stephan Seufert, Iris Miliaraki, Martin Theobald Databases & Information Systems Group ADReM Research Group Max-Planck Institute for Informatics University of Antwerp Saarbrücken, Germany Antwerp, Belgium 1 / 19

2 Resource Description Framework (RDF) RDF is a data model for representing information on the Web 2 / 19

3 Resource Description Framework (RDF) RDF is a data model for representing information on the Web RDF triples obtained from IE Barack Obama isa President of USA. Barack Obama bornin Honolulu. Barack Obama Nobel Peace Prize. Barack Obama memberof Democratic Party. 2 / 19

4 Resource Description Framework (RDF) RDF is a data model for representing information on the Web RDF triples obtained from IE Barack Obama isa President of USA. Barack Obama bornin Honolulu. Barack Obama Nobel Peace Prize. Barack Obama memberof Democratic Party. Barack Obama isa Singer isa Lady Gaga bornin memof Honolulu Democratic Party Grammy Award bornin locin USA New York locin locin locin locin Texas New Haven governor bornin Plains Nobel Peace prize George W Bush bornin memof Jimmy Carter Republican Party 2 / 19

5 Resource Description Framework (RDF) RDF is a data model for representing information on the Web RDF triples obtained from IE Barack Obama isa President of USA. Barack Obama bornin Honolulu. Barack Obama Nobel Peace Prize. Example datasets: YAGO (>120 MBarack facts), Obama DBpedia memberof (>400 MDemocratic facts), Party. Freebase,... isa isa Barack Obama Singer Lady Gaga bornin memof Honolulu Democratic Party Grammy Award bornin locin USA New York locin locin locin locin Texas New Haven governor bornin Plains Nobel Peace prize George W Bush bornin memof Jimmy Carter Republican Party 2 / 19

6 TriAD: A Distributed Shared-Nothing RDF Engine based on Asynchronous Message Passing Scale of RDF Data Published Linked Open Data Cloud I As of 2011, there exists more than 30 billion triples in the linked data cloud (from more than 300 sources) I Billion Triple Challenge (BTC) published more than one billion triples in years 2008, 2010, 2011, 2012 Source: 3 / 19

7 Indexing and Querying RDF Data RDF triples are stored and indexed in a relational table (relational approach) SPARQL is the language suggested by W3C for querying RDF data SPARQL has many similarities with standard SQL SELECT-PROJECT-JOIN forms the main building blocks of SPARQL 4 / 19

8 Indexing and Querying RDF Data RDF triples are stored and indexed in a relational table (relational approach) SPARQL is the language suggested by W3C for querying RDF data SPARQL has many similarities with standard SQL SELECT-PROJECT-JOIN forms the main building blocks of SPARQL SPARQL Query: Find persons who are born in USA and a prize.. SELECT?person,?city,?prize WHERE { (R1)?person <bornin>?city. (R2)?city <locatedin> USA. (R3)?person <>?prize. } bornin locatedin?person?city USA?prize RDF Data: Subject Predicate Object Barack Obama bornin Honolulu Barack Obama Nobel Peace Prize Barack Obama Grammy Award Barack Obama memberof Republican Party Honolulu locatedin United States Barack Obama isa Singer John F. Kennedy bornin Brookline John F. Kennedy memberof Republican Party John F. Kennedy diedin Dallas Dallas locatedin United States Brookline locatedin United States / 19

9 Indexing and Querying RDF Data RDF triples are stored and indexed in a relational table (relational approach) SPARQL is the language suggested by W3C for querying RDF data SPARQL has many similarities with standard SQL SELECT-PROJECT-JOIN forms the main building blocks of SPARQL SPARQL Query: Find persons who are born in USA and a prize.. SELECT?person,?city,?prize WHERE { (R1)?person <bornin>?city. (R2)?city <locatedin> USA. (R3)?person <>?prize. }?person bornin locatedin?city USA?prize Results: (R1 R2 R3)?person?city?prize Barack Obama Honolulu Peace Nobel Prize Barack Obama Honolulu Grammy Award... RDF Data: Subject Predicate Object Barack Obama bornin Honolulu Barack Obama Nobel Peace Prize Barack Obama Grammy Award Barack Obama memberof Republican Party Honolulu locatedin United States Barack Obama isa Singer John F. Kennedy bornin Brookline John F. Kennedy memberof Republican Party John F. Kennedy diedin Dallas Dallas locatedin United States Brookline locatedin United States / 19

10 Current Approaches Efficiency thoroughly investigated in single-node setting crucial factors Join-order optimization, Join-ahead pruning, indexing layout, choice of operators Eg. Jena, Sesame, HexaStore, MonetDB-RDF, SW-Store, RDF-3X, TripleBit, BitMat, gstore,... 5 / 19

11 Current Approaches Efficiency thoroughly investigated in single-node setting crucial factors Join-order optimization, Join-ahead pruning, indexing layout, choice of operators Eg. Jena, Sesame, HexaStore, MonetDB-RDF, SW-Store, RDF-3X, TripleBit, BitMat, gstore,... Scalability recently a line of distributed systems has been proposed SHARD, H-RDF-3X, Trinity.RDF,... }{{}}{{} Relational-based Graph-based 5 / 19

12 Current Approaches Efficiency thoroughly investigated in single-node setting crucial factors Join-order optimization, Join-ahead pruning, indexing layout, choice of operators Eg. Jena, Sesame, HexaStore, MonetDB-RDF, SW-Store, RDF-3X, TripleBit, BitMat, gstore,... Scalability recently a line of distributed systems has been proposed SHARD, H-RDF-3X, Trinity.RDF,... }{{}}{{} Relational-based Graph-based Relational-based (Joins) vs Graph-based (Exploration) [distributed setting] SPARQL 1.0 requires a row-oriented output joins are inevitable Relational approaches suffer from large intermediate relations (inaccurate/insufficient statistics resulting in poor query plans) 5 / 19

13 Current Approaches Efficiency thoroughly investigated in single-node setting crucial factors Join-order optimization, Join-ahead pruning, indexing layout, choice of operators Eg. Jena, Sesame, HexaStore, MonetDB-RDF, SW-Store, RDF-3X, TripleBit, BitMat, gstore,... Scalability recently a line of distributed systems has been proposed SHARD, H-RDF-3X, Trinity.RDF,... }{{}}{{} Relational-based Graph-based Relational-based (Joins) vs Graph-based (Exploration) [distributed setting] SPARQL 1.0 requires a row-oriented output joins are inevitable Relational approaches suffer from large intermediate relations (inaccurate/insufficient statistics resulting in poor query plans) Graph-based systems use distributed graph exploration followed by relational joins Effective when graph exploration prunes a lot of bindings (in case of selective queries) 5 / 19

14 Current Approaches Efficiency thoroughly investigated in single-node setting crucial factors Join-order optimization, Join-ahead pruning, indexing layout, choice of operators Eg. Jena, Sesame, HexaStore, MonetDB-RDF, SW-Store, RDF-3X, TripleBit, BitMat, gstore,... Scalability recently a line of distributed systems has been proposed SHARD, H-RDF-3X, Trinity.RDF,... }{{}}{{} Relational-based Graph-based Relational-based (Joins) vs Graph-based (Exploration) [distributed setting] SPARQL 1.0 requires a row-oriented output joins are inevitable Relational approaches suffer from large intermediate relations (inaccurate/insufficient statistics resulting in poor query plans) Graph-based systems use distributed graph exploration followed by relational joins Effective when graph exploration prunes a lot of bindings (in case of selective queries) Problems with existing distributed relational-based RDF engines 1. Synchronous processing of joins 2. Dangling triples occur in intermediate relations but not in final results 5 / 19

15 1. Synchronous Processing of Joins Consider a query with four relations R 1, R 2, R 3, R 4 with join order: (R 1 2 R 2 ) 1 (R 3 3 R 4 ) R 1 R 2 R 4 R 3 6 / 19

16 1. Synchronous Processing of Joins Consider a query with four relations R 1, R 2, R 3, R 4 with join order: (R 1 2 R 2 ) 1 (R 3 3 R 4 ) Synchronous case 1 : MR job1 MR job2 MR job3 MR job3 1 MR job1 2 MR job2 3 R 1 R 2 R 4 R 3 6 / 19

17 1. Synchronous Processing of Joins Consider a query with four relations R 1, R 2, R 3, R 4 with join order: (R 1 2 R 2 ) 1 (R 3 3 R 4 ) Synchronous case 1 : MR job1 MR job2 MR job3 Synchronous case 2 : MR job1 MR job2 MR job1 MR job R 1 R 2 R 4 R 3 6 / 19

18 1. Synchronous Processing of Joins Consider a query with four relations R 1, R 2, R 3, R 4 with join order: (R 1 2 R 2 ) 1 (R 3 3 R 4 ) Synchronous case 1 : MR job1 MR job2 MR job3 Synchronous case 2 : MR job1 MR job2 Synchronous case 3 : (MR job1 MR job2) MR job3 MR job1 MR job3 1 2 MR job2 3 R 1 R 2 R 4 MR job3 Slave 1 1 MR job3 R 3 Slave 2 1 MR job1 2 MR job2 3 MR job1 2 MR job2 3 Wait!! 6 / 19

19 1. Synchronous Processing of Joins Consider a query with four relations R 1, R 2, R 3, R 4 with join order: (R 1 2 R 2 ) 1 (R 3 3 R 4 ) Synchronous case 1 : MR job1 MR job2 MR job3 Synchronous case 2 : MR job1 MR job2 Synchronous case 3 : (MR job1 MR job2) MR job Our approach: Minimize dependancies among join operators R 1 and only synchronize when needed! (using asynchronous communication (MPICH2) protocol) R 2 R 4 Slave 1 1 R 3 Slave Asynchronous Wait!! 6 / 19

20 2. Pruning Dangling Triples Consider a query with four relations R 1, R 2, R 3, R 4 with join order: (R 1 2 R 2 ) 1 (R 3 3 R 4 ) R 1 R 2 R 4 R 3 1 RDF-3X: a RISC-style Engine for RDF, VLDBJ / 19

21 2. Pruning Dangling Triples Consider a query with four relations R 1, R 2, R 3, R 4 with join order: (R 1 2 R 2 ) 1 (R 3 3 R 4 ) Sideways Information Passing (SIP) of RDF-3X 1 Powerful runtime pruning technique Shares information across join operators 1 Requires synchronization 2 3 R 1 R 2 R 4 R 3 1 RDF-3X: a RISC-style Engine for RDF, VLDBJ / 19

22 2. Pruning Dangling Triples Consider a query with four relations R 1, R 2, R 3, R 4 with join order: (R 1 2 R 2 ) 1 (R 3 3 R 4 ) Sideways Information Passing (SIP) of RDF-3X 1 Powerful runtime pruning technique Shares information across join operators 1 Requires synchronization 2 3 Our approach: Join-ahead pruning 1 Pre-partition the triples (into groups) R 1 R 2 R 4 R 3 1 RDF-3X: a RISC-style Engine for RDF, VLDBJ / 19

23 2. Pruning Dangling Triples Consider a query with four relations R 1, R 2, R 3, R 4 with join order: (R 1 2 R 2 ) 1 (R 3 3 R 4 ) Sideways Information Passing (SIP) of RDF-3X 1 Powerful runtime pruning technique Shares information across join operators 1 Requires synchronization 2 3 Our approach: Join-ahead pruning 1 Pre-partition the triples (into groups) 2 Query over groups to find the ones which are relevant to the query R 1 R 2 R 4 R 3 1 RDF-3X: a RISC-style Engine for RDF, VLDBJ / 19

24 2. Pruning Dangling Triples Consider a query with four relations R 1, R 2, R 3, R 4 with join order: (R 1 2 R 2 ) 1 (R 3 3 R 4 ) Sideways Information Passing (SIP) of RDF-3X 1 Powerful runtime pruning technique Shares information across join operators 1 Requires synchronization 2 3 Our approach: Join-ahead pruning 1 Pre-partition the triples (into groups) 2 Query over groups to find the ones which are relevant to the query 3 Scan & Join only triples from the relevant groups R 1 R 2 R 3 R 4 1 RDF-3X: a RISC-style Engine for RDF, VLDBJ / 19

25 2. Join-ahead Pruning via Graph Summarization isa isa Barack Obama Singer Lady Gaga bornin memof Honolulu Democratic Party Grammy Award bornin locin USA New York locin locin locin locin Texas New Haven memof governor bornin Plains Nobel Peace prize George W Bush bornin memof Jimmy Carter Republican Party 8 / 19

26 2. Join-ahead Pruning via Graph Summarization Locality-based grouping (using METIS) isa isa Barack Obama Singer Lady Gaga bornin memof Honolulu Democratic Party Grammy Award bornin locin USA New York locin locin locin locin Texas New Haven memof governor bornin Plains Nobel Peace prize George W Bush bornin memof Jimmy Carter Republican Party 8 / 19

27 2. Join-ahead Pruning via Graph Summarization Locality-based grouping (using METIS) isa isa Barack Obama Singer Lady Gaga bornin memof Honolulu Democratic Party Grammy Award bornin locin USA New York locin locin locin locin Texas New Haven memof governor bornin Plains Nobel Peace prize George W Bush bornin memof Jimmy Carter Republican Party Summary Graph memof governor locin, bornin locin isa,, bornin locin P 1 P 2 isa, locin, memof P 3 P 4 memof,bornin,bornin 8 / 19

28 2. Join-ahead Pruning via Graph Summarization Locality-based grouping (using METIS) isa isa Barack Obama Singer Lady Gaga bornin memof Honolulu Democratic Party Grammy Award bornin locin USA New York locin locin locin locin Texas New Haven memof governor bornin Plains Nobel Peace prize George W Bush bornin memof Jimmy Carter Republican Party Summary Graph memof governor locin, bornin locin isa,, bornin locin P 1 P 2 isa, locin, memof P 3 P 4 memof,bornin,bornin SPARQL Query: SELECT?city,?prize WHERE { Barack Obama <bornin>?city.?city <locatedin> USA. Barack Obama <>?prize. } Supernode Bindings:?city : P 1?prize : P 2, P 4 8 / 19

29 2. Join-ahead Pruning via Graph Summarization Locality-based grouping (using METIS) isa isa Barack Obama Singer Lady Gaga bornin memof Honolulu Democratic Party Grammy Award bornin locin USA New York locin locin locin locin Texas New Haven memof governor bornin Plains Nobel Peace prize George W Bush bornin memof Jimmy Carter Republican Party SPARQL Query: SELECT?city,?prize WHERE { Barack Obama <bornin>?city.?city <locatedin> USA. Barack Obama <governor>?city2. } Summary Graph memof governor locin, bornin locin isa,, bornin locin P 1 P 2 isa, locin, memof P 3 P 4 memof,bornin,bornin Supernode Bindings:!!! Empty Result?city : --?prize : -- 8 / 19

30 2. Join-ahead Pruning via Graph Summarization Locality-based grouping (using METIS) isa isa Barack Obama Singer Lady Gaga bornin memof Honolulu Democratic Party Grammy Award bornin locin USA New York locin locin locin locin Texas New Haven memof governor bornin Plains Nobel Peace prize George W Bush bornin memof Jimmy Carter Republican Party Summary Graph memof governor locin, bornin locin isa,, bornin locin P 1 P 2 isa, locin, memof P 3 P 4 memof,bornin,bornin SPARQL Query: SELECT?city,?prize WHERE { Barack Obama <bornin>?city.?city <locatedin> USA. Barack Obama <governor>?city2. } Supernode Bindings:!!! Empty Result?city : --?prize : -- False postives may occur but no false negatives!!! 8 / 19

31 TriAD Distributed RDF Store TriAD (for Triple Asynchronous and Distributed ) a distributed RDF Store 9 / 19

32 TriAD Distributed RDF Store TriAD (for Triple Asynchronous and Distributed ) a distributed RDF Store Main-memory backed six permutation distributed indexes (with locality-aware sharding and encoded summary information) 9 / 19

33 TriAD Distributed RDF Store TriAD (for Triple Asynchronous and Distributed ) a distributed RDF Store Asynchronous Processing of Joins via MPICH2 communication protocol Main-memory backed six permutation distributed indexes (with locality-aware sharding and encoded summary information) 9 / 19

34 TriAD Distributed RDF Store TriAD (for Triple Asynchronous and Distributed ) a distributed RDF Store Join-ahead pruning via Graph Summarization Asynchronous Processing of Joins via MPICH2 communication protocol Main-memory backed six permutation distributed indexes (with locality-aware sharding and encoded summary information) 9 / 19

35 TriAD Distributed RDF Store TriAD (for Triple Asynchronous and Distributed ) a distributed RDF Store Two-stage Query Optimization (distributed, multi-threading, and join-ahead pruning into account) Join-ahead pruning via Graph Summarization Asynchronous Processing of Joins via MPICH2 communication protocol Main-memory backed six permutation distributed indexes (with locality-aware sharding and encoded summary information) 9 / 19

36 TriAD Architecture RDF Data METIS Partitions Info Master Node Summary Graph SPARQL Query Encoded Triples RDF Parser Bidirectional Dictionaries Query Plan Local Query Processor Local SPO Indexes Partitioner Intermediate Results SPO SOP PSO OSP OPS POS Summary Triples Data Triples Horizontal Partitioning Encoded Triples Global Statistics Local Statistics Query Plan Local Query Processor Local SPO Indexes SPO SOP PSO OSP OPS POS Supernodes MPICH2 Asynchronous Communication Protocol Query Optimizer Global Query Plan Intermediate Encoded Results Triples SPARQL Query Parser Query Graph Query Plan SPARQL Results Results Intermediate Results Local Query Processor Local SPO Indexes SPO SOP PSO OSP OPS POS Slave 1 Slave 2 Slave n 10 / 19

37 TriAD Architecture Indexing 1 Encoded Triples RDF Data RDF Parser Bidirectional Dictionaries Query Plan Local Query Processor Local SPO Indexes Partitioner Intermediate Results SPO SOP PSO OSP OPS POS METIS Partitions Info Master Node Summary Graph Summary Triples Data Triples Horizontal Partitioning Encoded Triples Global Statistics Local Statistics Query Plan Local Query Processor Local SPO Indexes SPO SOP PSO OSP OPS POS Supernodes MPICH2 Asynchronous Communication Protocol Query Optimizer Global Query Plan Intermediate Encoded Results Triples SPARQL Query Parser Query Plan SPARQL Query Query Graph SPARQL Results Results Intermediate Results Local Query Processor Local SPO Indexes SPO SOP PSO OSP OPS POS Slave 1 Slave 2 Slave n 10 / 19

38 TriAD Architecture 2 Indexing 1 Encoded Triples RDF Data RDF Parser Bidirectional Dictionaries Query Plan Local Query Processor Local SPO Indexes Partitioner Intermediate Results SPO SOP PSO OSP OPS POS METIS Partitions Info Master Node Summary Graph Summary Triples Data Triples Horizontal Partitioning Encoded Triples Global Statistics Local Statistics Query Plan Local Query Processor Local SPO Indexes SPO SOP PSO OSP OPS POS Supernodes MPICH2 Asynchronous Communication Protocol Query Optimizer Global Query Plan Intermediate Encoded Results Triples SPARQL Query Parser Query Plan SPARQL Query Query Graph SPARQL Results Results Intermediate Results Local Query Processor Local SPO Indexes SPO SOP PSO OSP OPS POS Slave 1 Slave 2 Slave n 10 / 19

39 TriAD Architecture 2 Indexing 1 Encoded Triples RDF Data RDF Parser Bidirectional Dictionaries Query Plan Local Query Processor Local SPO Indexes Partitioner Intermediate Results SPO SOP PSO OSP OPS POS METIS Partitions Info Master Node Summary Graph Summary Triples Data Triples Horizontal Partitioning Encoded Triples Global Statistics Local Statistics Query Plan Local Query Processor Local SPO Indexes 3 SPO SOP PSO OSP OPS POS Supernodes MPICH2 Asynchronous Communication Protocol Query Optimizer Global Query Plan Intermediate Encoded Results Triples SPARQL Query Parser Query Plan SPARQL Query Query Graph SPARQL Results Results Intermediate Results Local Query Processor Local SPO Indexes SPO SOP PSO OSP OPS POS Slave 1 Slave 2 Slave n 10 / 19

40 TriAD Architecture Encoded Triples RDF Data RDF Parser Bidirectional Dictionaries Query Plan Local Query Processor Local SPO Indexes Partitioner Intermediate Results SPO SOP PSO OSP OPS POS METIS Partitions Info Master Node Summary Graph Summary Triples Data Triples Horizontal Partitioning Encoded Triples Global Statistics Local Statistics Query Plan Local Query Processor Local SPO Indexes SPO SOP PSO OSP OPS POS Supernodes MPICH2 Asynchronous Communication Protocol Intermediate Encoded Results Triples SPARQL Query Processing Query Optimizer Global Query Plan SPARQL Query Parser Query Plan SPARQL Query Query Graph SPARQL Results Results Intermediate Results Local Query Processor Local SPO Indexes SPO SOP PSO OSP OPS POS Stage1 Slave 1 Slave 2 Slave n 10 / 19

41 TriAD Architecture Encoded Triples RDF Data RDF Parser Bidirectional Dictionaries Query Plan Local Query Processor Local SPO Indexes Partitioner Intermediate Results SPO SOP PSO OSP OPS POS METIS Partitions Info Master Node Summary Graph Summary Triples Data Triples Horizontal Partitioning Encoded Triples Global Statistics Local Statistics Query Plan Local Query Processor Local SPO Indexes SPO SOP PSO OSP OPS POS Supernodes MPICH2 Asynchronous Communication Protocol Intermediate Encoded Results Triples SPARQL Query Processing Query Optimizer Global Query Plan SPARQL Query Parser Query Plan SPARQL Query Query Graph SPARQL Results Results Intermediate Results Local Query Processor Local SPO Indexes SPO SOP PSO OSP OPS POS Stage1 Stage2 Slave 1 Slave 2 Slave n 10 / 19

42 Indexing RDF graph partitioning using a locality-based non-overlapping partitioning algorithm i.e Each s (or o) of RDF triple s, p, o mapped to one supernode P s (or P o) 11 / 19

43 Indexing RDF graph partitioning using a locality-based non-overlapping partitioning algorithm i.e Each s (or o) of RDF triple s, p, o mapped to one supernode P s (or P o) Triple encoding Each triple s, p, o is encoded into two triples: data triple, summary triple Dictionary encoding: Each entity is assigned a globally unique id is computed by concatenating (supernode id) P s and a (local id) id s Eg. Barack Obama,, Nobel Prize Data triple: 1 1, 6, 4 3 Summary triple 1, 6, 4 11 / 19

44 Indexing RDF graph partitioning using a locality-based non-overlapping partitioning algorithm i.e Each s (or o) of RDF triple s, p, o mapped to one supernode P s (or P o) Triple encoding Each triple s, p, o is encoded into two triples: data triple, summary triple Dictionary encoding: Each entity is assigned a globally unique id is computed by concatenating (supernode id) P s and a (local id) id s Eg. Barack Obama,, Nobel Prize Data triple: 1 1, 6, 4 3 Summary triple 1, 6, 4 Locality-aware sharding and distributed indexing of data triples Each data triple is hash partitioned onto atmost two slaves and indexed in six permutations (in total) Eg. triple 1 1, 6, 4 3 is hashed on to slaves 1 mod n and 4 mod n (for n-slaves) 11 / 19

45 Indexing RDF graph partitioning using a locality-based non-overlapping partitioning algorithm i.e Each s (or o) of RDF triple s, p, o mapped to one supernode P s (or P o) Triple encoding Each triple s, p, o is encoded into two triples: data triple, summary triple Dictionary encoding: Each entity is assigned a globally unique id is computed by concatenating (supernode id) P s and a (local id) id s Eg. Barack Obama,, Nobel Prize Data triple: 1 1, 6, 4 3 Summary triple 1, 6, 4 Locality-aware sharding and distributed indexing of data triples Each data triple is hash partitioned onto atmost two slaves and indexed in six permutations (in total) Eg. triple 1 1, 6, 4 3 is hashed on to slaves 1 mod n and 4 mod n (for n-slaves) Summary graph index Summary triples are indexed in two-permutation adjacency list for efficient graph exploration 11 / 19

46 Query Processing In TriAD, query processing is performed in two stages Stage 1: Summary graph query processing (join-ahead pruning) Performed using graph exploration at master node to generate supernode bindings 12 / 19

47 Query Processing In TriAD, query processing is performed in two stages Stage 1: Summary graph query processing (join-ahead pruning) Performed using graph exploration at master node to generate supernode bindings Stage 2: Data graph query processing Done using relational joins in a distributed setup Inspired by the RISC style processing, we employ three physical operators: Distributed Index Scan (DIS): Invokes a parallel scan over a permutation list that is sharded across all slaves Distributed Merge Join (DMJ): If both input (or intermediate) relations are sorted according to the join key(s) in the query plan Distributed Hash Join (DHJ): If the input (or intermediate) relations are not sorted according to their join key(s) 12 / 19

48 Global Statistics & Query Optimization Precomputed statistics Computed in parallel at slaves and sent to master node Statistics: Individual cardinalities: s, p, o of SPO triples Pair cardinalities: (s, o), (s, p), (p, o) Join selectivities of predicate pairs (p 1, p 2 ) Similar statistics are computed over summary graph For a given query with patterns Q {R 1, R 2,..R n} Stage 1: Exploration Optimization We compute the exploration plan by using a bottom-up dynamic programming and the following cost model: Cost(R 1,..., R n) Card(R 1 ) + n i=2 ( Card(R i ) i j=1 ) Sel(R i, R j ) (1) 13 / 19

49 Global Statistics & Query Optimization Precomputed statistics Computed in parallel at slaves and sent to master node Statistics: Individual cardinalities: s, p, o of SPO triples Pair cardinalities: (s, o), (s, p), (p, o) Join selectivities of predicate pairs (p 1, p 2 ) Similar statistics are computed over summary graph For a given query with patterns Q {R 1, R 2,..R n} Stage 1: Exploration Optimization We compute the exploration plan by using a bottom-up dynamic programming and the following cost model: Cost(R 1,..., R n) Card(R 1 ) + n i=2 ( Card(R i ) i j=1 ) Sel(R i, R j ) (1) 13 / 19

50 Re-estimating cardinalities of a relations R i New cardinalities are re-estimated from the Stage 1 supernode bindings Card(R i ) := CQ s C s CQ o C o Card(R i) (on the fly computation) where C s, C o are the supernode bindings of R i and C Q s, C Q o are the supernode bindings of Q (R i Q) Stage 2: Global plan optimization A global plan is computed for stage 2 using re-estimated cardinalities and the following cost model Cost(Q) := Cost(Ri k) if R i denotes a DIS over permutation k; max(cost(q left ), Cost(Q right )) +Cost(Q left op Q right ) +Cost(Q left op Q right ) otherwise. The cost model captures the multi-threaded and distributed execution framework 14 / 19

51 Re-estimating cardinalities of a relations R i New cardinalities are re-estimated from the Stage 1 supernode bindings Card(R i ) := CQ s C s CQ o C o Card(R i) (on the fly computation) where C s, C o are the supernode bindings of R i and C Q s, C Q o are the supernode bindings of Q (R i Q) Stage 2: Global plan optimization A global plan is computed for stage 2 using re-estimated cardinalities and the following cost model Cost(Q) := Cost(Ri k) if R i denotes a DIS over permutation k; max(cost(q left ), Cost(Q right )) +Cost(Q left op Q right ) +Cost(Q left op Q right ) otherwise. The cost model captures the multi-threaded and distributed execution framework 14 / 19

52 Querying Optimization & Processing - Example SPARQL Query: Find persons who are born in USA and a prize.. SELECT?person,?city,?prize WHERE { R 1?person <bornin>?city. R 2?city <locatedin> USA. R 3?person <>?prize. R 4?prize <hasname>?name. } 15 / 19

53 Querying Optimization & Processing - Example SPARQL Query: Find persons who are born in USA and a prize.. SELECT?person,?city,?prize WHERE { R 1?person <bornin>?city. R 2?city <locatedin> USA. R 3?person <>?prize. R 4?prize <hasname>?name. } Stage 1 (Optimization) Summary Graph locin, bornin isa,, bornin locin memof governor locin P 1 P 2 isa, locin, memof P 3 P 4 memof,bornin,bornin Exploration Supernode Bindings:?person : P1, P2, P4?city : P1, P2, P4?prize : P2, P4 15 / 19

54 Querying Optimization & Processing - Example SPARQL Query: Find persons who are born in USA and a prize.. SELECT?person,?city,?prize WHERE { R 1?person <bornin>?city. R 2?city <locatedin> USA. R 3?person <>?prize. R 4?prize <hasname>?name. } Stage 1 (Optimization) Summary Graph locin, bornin isa,, bornin locin memof governor locin P 1 P 2 isa, locin, memof P 3 P 4?person DHJ(R 1,2,3,4 ) Cost:max(105,215)+130 Sharding:R 1,2,R 3,4 Cost:max(100,10)+5 DMJ(R 1,2) DMJ(R 3,4) Sharding: R 2?city?prize Cost:max(200,150)+15 Sharding: None Order: Cost: Slaves: Supernodes: Global Plan DIS(R 1) DIS(R 2) DIS(R 3) DIS(R 4) POS POS POS PSO [1,2] [1] [1,2] [1,2] [1,2,4] [1] [1,2,4] [1,2,4] memof,bornin Exploration Supernode Bindings:?person : P1, P2, P4?city : P1, P2, P4 Stage 2?prize : P2, P4 (Optimization),bornIn 15 / 19

55 Query Optimization & Processing Example (2) Slave 1 Slave 2 DHJ(R 1,2,3,4) Partial Results R 1,2 R 3,4 Partial Results DHJ(R 1,2,3,4) DMJ(R 1,2) DMJ(R 3,4) EP1 EP2 EP3 EP4 R 2 DMJ(R 1,2) DMJ(R 3,4) DIS(R 1 ) DIS(R 2 ) DIS(R ) DIS(R ) DIS(R ) DIS(R 3 ) 4 DIS(R 4 ) DIS(R ) (bornin,?o,?s) (locin,usa,?s) (,?o,?s) (hasname,?s,?o) (bornin,?o,?s) (locin,usa,?s) (,?o,?s) (hasname,?s,?o) P O S P O S P O S P S O P O S Stage 2: Distributed query execution P O S P S O 16 / 19

56 Evaluation We compared performance of TriAD with the following state-of-the-art systems Centralized systems RDF-3X, MonetDB-RDF, BitMat Distributed systems H-RDF-3X, Trinity.RDF, 4store, SHARD, Spark Datasets: (Synthetic) LUBM Million triples, 16GB raw data (Synthetic) LUBM Billion triples, 730GB raw data (Real world) BTC Billion triples, 231 GB raw data (Synthetic) WSDTS 109 Million triples, 15 GB raw data Benchmark queries for LUBM, BTC, & WSDTS datasets System Setup: TriAD, TriAD-SG is implemented in C++ Cluster setup: 12-nodes, 48GB RAM, 2 quad-core CPUs of 2.4GHz (HT enabled) 17 / 19

57 Evaluation - Large datasets LUBM Dataset 1.8 Billion triples (Query Performance in milli seconds) Geo.- Mean (in ms) 1e4 1e3 1e2 1e1 Warm/Main-memory Cold 1e0 TriAD TriAD-SG Trinity.RDF H-RDF-3X RDF-3X } {{ } Our Approach Queries Characteristics TriAD TriAD-SG Trinity.RDF H-RDF-3X RDF-3X Q1 Selective (6 joins) 7,631 2,146 12, e6 1.7e5 1.9e6 1.8e6 Q2 Non-Selective(1 join) 1,663 2,025 6, e5 4, e5 1.8e5 Q4 Selective (5 joins) Q7 Selective (6 joins) 14,895 16,863 31, e6 2.1e5 6.5e5 46, / 19

58 Summary TriAD is a fast distributed RDF engine built on top of asynchronous communication layer and multi-threaded execution framework efficient distributed and parallel join executions join-ahead pruning technique via graph summarization helps in pruning dangling triples and making query processing efficient distributed- and join-ahead pruning aware query optimizer so far reported fastest runtimes over three benchmark datasets: LUBM, BTC, WSDTS 19 / 19

59 Summary TriAD is a fast distributed RDF engine built on top of asynchronous communication layer and multi-threaded execution framework efficient distributed and parallel join executions join-ahead pruning technique via graph summarization helps in pruning dangling triples and making query processing efficient distributed- and join-ahead pruning aware query optimizer so far reported fastest runtimes over three benchmark datasets: LUBM, BTC, WSDTS Questions & Thank You!! 19 / 19