Classical DB Questions on New Kinds of Data

Transcription

1 1 Classical DB Questions on New Kinds of Data Marcelo Arenas & Pablo Barceló, Center for Semantic Web Research PUC Chile Universidad de Chile

2 Evolution of data models 80s early 90s Overcome expressiveness limitations of the relational model

3 Evolution of data models 80s early 90s Overcome expressiveness limitations of the relational model Logical data model OODB Active databases Disjunctive databases Temporal databases Constraint databases...

4 Evolution of data models The 90s on: Flexibility, easy integration, topology (sometimes)

5 Evolution of data models The 90s on: Flexibility, easy integration, topology (sometimes) Graphs XML RDF JSON CSV...

6 Classical DB questions on new kind of data

7 Classical DB questions on new kind of data Data design Languages (markup, querying) Optimization Updates

8 Classical questions that gain renewed interest

9 Classical questions that gain renewed interest Uncertainty Distribution Ranking Query workloads

10 New questions on new kind of data

11 New questions on new kind of data Access Schema extraction Trust Variety

12 7 In this talk We talk about Graph DBs, RDF and tabular data (CSV)

13 7 In this talk We talk about Graph DBs, RDF and tabular data (CSV) Bottomline: Despite similarities with old DB problems, new applications require: understanding the nature of their problems refining old/developing new techniques

14 Graph Databases 8

15 9 The need for a standard graph QL in practice Recognized by major graph DB engines

16 9 The need for a standard graph QL in practice Recognized by major graph DB engines Neo4J Cypher

17 9 The need for a standard graph QL in practice Recognized by major graph DB engines Neo4J Cypher Sparksee Graph algebra

18 9 The need for a standard graph QL in practice Recognized by major graph DB engines Neo4J Cypher Sparksee Graph algebra Oracle PGQL

19 9 The need for a standard graph QL in practice Recognized by major graph DB engines Neo4J Cypher Sparksee Graph algebra Oracle PGQL Apache Tinkerpop Gremlin

20 9 The need for a standard graph QL in practice Recognized by major graph DB engines Neo4J Cypher Sparksee Graph algebra Oracle PGQL Apache Tinkerpop Gremlin Not all of them are declarative

21 10 LDBC GraphQL task force Formed by researchers/practitioners (Neo4J, Sparksee, IBM, Oracle, SAP, HP)

22 10 LDBC GraphQL task force Formed by researchers/practitioners (Neo4J, Sparksee, IBM, Oracle, SAP, HP) Objectives:

23 10 LDBC GraphQL task force Formed by researchers/practitioners (Neo4J, Sparksee, IBM, Oracle, SAP, HP) Objectives: 1. Define a graph data model: Property graphs (July 2015)

24 10 LDBC GraphQL task force Formed by researchers/practitioners (Neo4J, Sparksee, IBM, Oracle, SAP, HP) Objectives: 1. Define a graph data model: Property graphs (July 2015) 2. Identify features/limitations of existing QLs (December 2015)

25 10 LDBC GraphQL task force Formed by researchers/practitioners (Neo4J, Sparksee, IBM, Oracle, SAP, HP) Objectives: 1. Define a graph data model: Property graphs (July 2015) 2. Identify features/limitations of existing QLs (December 2015) 3. Identify features needed in a graph QL (ongoing)

26 10 LDBC GraphQL task force Formed by researchers/practitioners (Neo4J, Sparksee, IBM, Oracle, SAP, HP) Objectives: 1. Define a graph data model: Property graphs (July 2015) 2. Identify features/limitations of existing QLs (December 2015) 3. Identify features needed in a graph QL (ongoing) 4. Design a standard declarative graph QL (2017?)

27 11 A property graph by example e 1 : acts in role = R. Kincaid ref = IMDb n 1 : Person name = C. Eastwood gender = male n 2 : Movie title = The bridges of Madison County e 5 : acts in role = H. Callahan ref = IMDb n 5 : Movie title = Dirty Harry e 2 : directs e 6 : acts in n 6 : Person name = A. Robinson gender = male

28 The property graph data model Definition (Property graph) A property graph G is a tuple (V, E, ρ, λ, σ), where:

29 The property graph data model Definition (Property graph) A property graph G is a tuple (V, E, ρ, λ, σ), where: 1. V is a finite set of nodes

30 The property graph data model Definition (Property graph) A property graph G is a tuple (V, E, ρ, λ, σ), where: 1. V is a finite set of nodes 2. E is a finite set of edges

31 The property graph data model Definition (Property graph) A property graph G is a tuple (V, E, ρ, λ, σ), where: 1. V is a finite set of nodes 2. E is a finite set of edges 3. ρ : E (V V ) is a total function ρ(e) = (v 1, v 2 ) means that e is of the form v 1 v 2

32 The property graph data model Definition (Property graph) A property graph G is a tuple (V, E, ρ, λ, σ), where: 1. V is a finite set of nodes 2. E is a finite set of edges 3. ρ : E (V V ) is a total function ρ(e) = (v 1, v 2 ) means that e is of the form v 1 v 2 4. λ : (V E) Lab is a total function with Lab a set of labels ρ(v) = l means that l is the label of node v in G

33 The property graph data model Definition (Property graph) A property graph G is a tuple (V, E, ρ, λ, σ), where: 1. V is a finite set of nodes 2. E is a finite set of edges 3. ρ : E (V V ) is a total function ρ(e) = (v 1, v 2 ) means that e is of the form v 1 v 2 4. λ : (V E) Lab is a total function with Lab a set of labels ρ(v) = l means that l is the label of node v in G 5. σ : (V E) Prop Val is a partial function with Prop a finite set of properties and Val a set of values σ(v, p) = s means that s is the value of property p for v in G

34 In existing query languages Basic unit for querying the structure: Graph patterns, a.k.a. conjunctive queries

35 13 In existing query languages Basic unit for querying the structure: Graph patterns, a.k.a. conjunctive queries Interpretation: Match a small graph pattern into a large property graph

36 13 In existing query languages Basic unit for querying the structure: Graph patterns, a.k.a. conjunctive queries Interpretation: Match a small graph pattern into a large property graph Semantics: homomorphism, isomorphism, edge-injective homomorphism

37 13 In existing query languages Basic unit for querying the structure: Graph patterns, a.k.a. conjunctive queries Interpretation: Match a small graph pattern into a large property graph Semantics: homomorphism, isomorphism, edge-injective homomorphism Property graph Relational table

38 In existing query languages Basic unit for querying the structure: Graph patterns, a.k.a. conjunctive queries Interpretation: Match a small graph pattern into a large property graph Semantics: homomorphism, isomorphism, edge-injective homomorphism Property graph Relational table Example: y (x, directs, y) (x, acts in, y)

39 In existing query languages Basic unit for querying the structure: Graph patterns, a.k.a. conjunctive queries Interpretation: Match a small graph pattern into a large property graph Semantics: homomorphism, isomorphism, edge-injective homomorphism Property graph Relational table Example: y (x, directs, y) (x, acts in, y) MATCH (x) -[directs]-> (y) <-[acts in]- (x) RETURN (x)

40 Dealing with paths is fundamental Queries of the form x a y allowed in Cypher Retrieve pairs of nodes linked by a path of a s

41 Dealing with paths is fundamental Queries of the form x a y allowed in Cypher Retrieve pairs of nodes linked by a path of a s Example: Friend-of-a-friend relationship

42 Dealing with paths is fundamental Queries of the form x a y allowed in Cypher Retrieve pairs of nodes linked by a path of a s Example: Friend-of-a-friend relationship Combined with patterns: patterns with reachability

43 Dealing with paths is fundamental Queries of the form x a y allowed in Cypher Retrieve pairs of nodes linked by a path of a s Example: Friend-of-a-friend relationship Combined with patterns: patterns with reachability Reasonable data complexity so far: NLogspace

44 RPQs not implemented yet... but considered to be desirable

45 15 RPQs not implemented yet... but considered to be desirable Regular Path Query (RPQ): x L y, for L a regex Retrieve pairs of nodes linked by a path in L

46 15 RPQs not implemented yet... but considered to be desirable Regular Path Query (RPQ): x L y, for L a regex Retrieve pairs of nodes linked by a path in L Example: ( acts in acts in ) +

47 15 RPQs not implemented yet... but considered to be desirable Regular Path Query (RPQ): x L y, for L a regex Retrieve pairs of nodes linked by a path in L Example: ( acts in acts in ) + Data complexity still reasonable: NLogspace

48 15 RPQs not implemented yet... but considered to be desirable Regular Path Query (RPQ): x L y, for L a regex Retrieve pairs of nodes linked by a path in L Example: ( acts in acts in ) + Data complexity still reasonable: NLogspace For a semantics based on arbitrary paths

49 Systems tend to favour a simple paths semantics 16

50 16 Systems tend to favour a simple paths semantics Is there a simple path from x to y labeled in (aa)? NP-complete [Lapaugh & Papadimitriou, 1984]

51 Systems tend to favour a simple paths semantics Is there a simple path from x to y labeled in (aa)? NP-complete [Lapaugh & Papadimitriou, 1984] A simple path semantics already tried/discarded in SPARQL 1.1 [Losseman & Martens, 2012; Arenas, Conca & Pérez, 2012]

52 They also return paths...

53 They also return paths... Example MATCH p = (x) -[acts in*0..5]-> (y) RETURN p, length(p)

54 They also return paths... Example MATCH p = (x) -[acts in*0..5]-> (y) RETURN p, length(p) A single path, a simple path, all paths, all simple paths?

55 18 Just imagine this combined with RPQs......an exponential explosion in the size of the output

56 18 Just imagine this combined with RPQs......an exponential explosion in the size of the output What are the use cases for these functionalities?

57 18 Just imagine this combined with RPQs......an exponential explosion in the size of the output What are the use cases for these functionalities? Which paths users are looking for?

58 18 Just imagine this combined with RPQs......an exponential explosion in the size of the output What are the use cases for these functionalities? Which paths users are looking for? Can they be represented compactly?

59 18 Just imagine this combined with RPQs......an exponential explosion in the size of the output What are the use cases for these functionalities? Which paths users are looking for? Can they be represented compactly? Can they be enumerated with reasonable delay?

60 18 Just imagine this combined with RPQs......an exponential explosion in the size of the output What are the use cases for these functionalities? Which paths users are looking for? Can they be represented compactly? Can they be enumerated with reasonable delay? From a theoretical point of view: Return a single shortest path (complexity in NLogspace)

61 18 Just imagine this combined with RPQs......an exponential explosion in the size of the output What are the use cases for these functionalities? Which paths users are looking for? Can they be represented compactly? Can they be enumerated with reasonable delay? From a theoretical point of view: Return a single shortest path (complexity in NLogspace) Return others, one by one, if needed

62 A representation problem How paths should be represented?

63 A representation problem How paths should be represented? As a tuple in a table?

64 A representation problem How paths should be represented? As a tuple in a table? As a graph?

65 A representation problem How paths should be represented? As a tuple in a table? As a graph? As a new path data type?

66 20 To complicate things more Existing QLs have an ungrouping operator: List set (bag) of nodes that belong to a path

67 20 To complicate things more Existing QLs have an ungrouping operator: List set (bag) of nodes that belong to a path Path Relational table

68 To complicate things more Existing QLs have an ungrouping operator: List set (bag) of nodes that belong to a path Path Relational table The power of ungrouping: 1. Find a path that visits each node exactly once 2. Find a path that does not repeat values for a given property 3. Find paths that intersect in a node

69 To complicate things more Existing QLs have an ungrouping operator: List set (bag) of nodes that belong to a path Path Relational table The power of ungrouping: 1. Find a path that visits each node exactly once 2. Find a path that does not repeat values for a given property 3. Find paths that intersect in a node Theory: Graph QLs for data Complexity is astronomical [Barceló, Fontaine & Lin, 2013] Tractability conditions are delicate [Libkin & Vrgoč, 2012]

70 21 Type transformations TABLES Patterns GRAPHS GRAPHS Path queries PATHS

71 21 Type transformations TABLES Patterns GRAPHS Ungrouping GRAPHS Path queries PATHS

72 21 Type transformations TABLES Patterns Construct GRAPHS Ungrouping GRAPHS Path queries PATHS

73 What have we learned? Real need from graph DBs to have a standard declarative QL

74 What have we learned? Real need from graph DBs to have a standard declarative QL Design is non-trivial due to efficiency/expressiveness trade-off

75 What have we learned? Real need from graph DBs to have a standard declarative QL Design is non-trivial due to efficiency/expressiveness trade-off Theory is not so much far away from practice

76 What have we learned? Real need from graph DBs to have a standard declarative QL Design is non-trivial due to efficiency/expressiveness trade-off Theory is not so much far away from practice It requires a political effort to close the gap

77 Semantic Web 23

78 24 Semantic Web The Semantic Web is an extension of the current web in which information is given well-defined meaning, better enabling computers and people to work in cooperation. [Tim Berners-Lee et al ]

79 24 Semantic Web The Semantic Web is an extension of the current web in which information is given well-defined meaning, better enabling computers and people to work in cooperation. [Tim Berners-Lee et al ] Specific goals:

80 24 Semantic Web The Semantic Web is an extension of the current web in which information is given well-defined meaning, better enabling computers and people to work in cooperation. [Tim Berners-Lee et al ] Specific goals: Build a description language with standard semantics

81 24 Semantic Web The Semantic Web is an extension of the current web in which information is given well-defined meaning, better enabling computers and people to work in cooperation. [Tim Berners-Lee et al ] Specific goals: Build a description language with standard semantics Make semantics machine-processable and understandable

82 24 Semantic Web The Semantic Web is an extension of the current web in which information is given well-defined meaning, better enabling computers and people to work in cooperation. [Tim Berners-Lee et al ] Specific goals: Build a description language with standard semantics Make semantics machine-processable and understandable Incorporate logical infrastructure to reason about resources

83 24 Semantic Web The Semantic Web is an extension of the current web in which information is given well-defined meaning, better enabling computers and people to work in cooperation. [Tim Berners-Lee et al ] Specific goals: Build a description language with standard semantics Make semantics machine-processable and understandable Incorporate logical infrastructure to reason about resources W3C proposals: Resource Description Framework (RDF) and SPARQL

84 25 RDF in a nutshell RDF is the W3C proposal framework for representing information in the Web Abstract syntax based on directed labeled graph Extensible URI-based vocabulary

85 26 An RDF graph starring The Bridges of Madison County Clint Eastwood starring directs Meryl Streep

86 An RDF graph in real life: DBpedia

87 Prefixes simplify the notation Prefixes can be defined in an RDF graph to simplify notation They are defined also using triples

88 Prefixes simplify the notation Prefixes can be defined in an RDF graph to simplify notation They are defined also using triples We can include in an RDF graph the following owl: rdf: <

89 Prefixes simplify the notation Prefixes can be defined in an RDF graph to simplify notation They are defined also using triples We can include in an RDF graph the following owl: rdf: < Then < can be replaced by rdf:type

90 How are URIs created and assigned? 29

91 29 How are URIs created and assigned? There is no centralized mechanism

92 29 How are URIs created and assigned? There is no centralized mechanism A key component to deal with this issue: owl:sameas Streep owl:sameas Streepová. Streep owl:sameas Streep. Streep owl:sameas

93 29 How are URIs created and assigned? There is no centralized mechanism A key component to deal with this issue: owl:sameas Streep owl:sameas Streepová. Streep owl:sameas Streep. Streep owl:sameas Reasoning capabilities are needed to deal with owl:sameas

94 Querying RDF: SPARQL SPARQL is the W3C recommendation query language for RDF (January 2008) Originally it was a graph-matching query language SPARQL 1.1 is the new version of this language, its was released in March 2013

95 An example of a SPARQL query Retrieve all the movies in DBpedia

96 An example of a SPARQL query Retrieve all the movies in DBpedia?movie rdf:type <

97 An example of a SPARQL query Retrieve all the movies in DBpedia WHERE {?movie rdf:type < }

98 An example of a SPARQL query Retrieve all the movies in DBpedia SELECT?movie WHERE {?movie rdf:type < }

99 Important problems when querying RDF

100 Important problems when querying RDF Returning as much information as possible

101 Important problems when querying RDF Returning as much information as possible Reasoning with ontologies

102 Important problems when querying RDF Returning as much information as possible Reasoning with ontologies Dealing with incomplete information

103 Important problems when querying RDF Returning as much information as possible Reasoning with ontologies Dealing with incomplete information Exploiting the graph structure of RDF

104 Important problems when querying RDF Returning as much information as possible Reasoning with ontologies Dealing with incomplete information Exploiting the graph structure of RDF Working with highly distributed data

105 Returning as much information as possible RDF follows an open world assumption Users may be unaware of the structure of the data

106 Returning as much information as possible RDF follows an open world assumption Users may be unaware of the structure of the data Thus, the possibility of obtaining additional information if possible is important in this scenario In fact, this feature was present from the very beginning in SPARQL

107 34 An optional operator Retrieve each movie in DBpedia and its gross if this information is available SELECT?movie?gross WHERE {?movie rdf:type < OPTIONAL {?movie < } }

108 35 Part of the answer to the query?movie (2013 film) Souls?gross "1.274E9"

109 What is new? The OPTIONAL operator essentially corresponds to a left-outer join in relational algebra But...

110 What is new? The OPTIONAL operator essentially corresponds to a left-outer join in relational algebra But... The fragments of SPARQL that are natural to study are different than for the case of relational algebra The complexity of evaluating these fragments was not known [Pérez, A. & Gutierrez 2009; Schmidt, Meier & Lausen 2010] New notions of safeness are needed to avoid a counterintuitive behavior [Pérez, A. & Gutierrez 2009] New optimization techniques are needed [Pérez, A. & Gutierrez 2009; Letelier, Pérez, Pichler & Skritek 2013; Pichler & Skritek 2014]

111 37 Openness influences other operators SELECT?voice_actor?film_actor WHERE { {?voice_actor rdf:type } < } UNION {?film_actor rdf:type < }

112 38 The answer to the query?voice actor resource/alec Baldwin?film actor resource/meryl Streep

113 39 Reasoning with ontologies Reasoning capabilities are needed

114 39 Reasoning with ontologies Reasoning capabilities are needed We already mentioned owl:sameas

115 39 Reasoning with ontologies Reasoning capabilities are needed We already mentioned owl:sameas An RDF graph can use RDF Schema (RDFS) to establish hierarchies of classes and properties

116 39 Reasoning with ontologies Reasoning capabilities are needed We already mentioned owl:sameas An RDF graph can use RDF Schema (RDFS) to establish hierarchies of classes and properties The Web Ontology Language (OWL) can be used to define more complex relations between classes and properties

117 The use of RDFS vocabulary The following triples are included in DBpedia:

118 40 The use of RDFS vocabulary The following triples are included in DBpedia: rdfs:subclassof rdfs:subclassof rdfs:subpropertyof rdfs:subpropertyof

119 41 The use of RDFS vocabulary Some numbers in DBpedia: triples with rdfs:subclassof as predicate are at least 450K triples with rdfs:subpropertyof as predicate are at least 1K

120 41 The use of RDFS vocabulary Some numbers in DBpedia: triples with rdfs:subclassof as predicate are at least 450K triples with rdfs:subpropertyof as predicate are at least 1K We need reasoning capabilities to deal with: rdfs:subclassof, rdfs:subpropertyof

121 41 The use of RDFS vocabulary Some numbers in DBpedia: triples with rdfs:subclassof as predicate are at least 450K triples with rdfs:subpropertyof as predicate are at least 1K We need reasoning capabilities to deal with: rdfs:subclassof, rdfs:subpropertyof and other elements of RDFS such as rdfs:domain and rdfs:range

122 42 Answering a query with RDFS vocabulary Retrieve all the educators in DBpedia SELECT?educator WHERE {?educator rdf:type < }

123 43 Answering a query with RDFS vocabulary The answer to the previous query should be the same as for the following query: SELECT?educator WHERE { {?educator rdf:type. } UNION {?educator rdf:type < } }

124 Open issues about reasoning with ontologies Two important problems: Development of efficient query answering algorithms over large RDF graphs with RDFS vocabulary Identification of fragments of OWL that have good expressive power and can be efficiently evaluated

125 Exploiting the graph structure of RDF The structure of an RDF graph stores information It is important to have operators that can deal with this structure In particular, navigating an RDF graph is an important functionality

126 Exploiting the graph structure of RDF The structure of an RDF graph stores information It is important to have operators that can deal with this structure In particular, navigating an RDF graph is an important functionality Properties paths in SPARQL allow to express reachability queries

127 46 Navigating RDF graphs Get starring actors in the same movie: SELECT?actor1?actor2 WHERE {?movie < < }

128 47 Navigating RDF graphs Previous query can be rewritten by using navigation patterns:

129 47 Navigating RDF graphs Previous query can be rewritten by using navigation patterns: SELECT?actor1?actor2 WHERE {?actor1 ^< < }

130 47 Navigating RDF graphs Previous query can be rewritten by using navigation patterns: SELECT?actor1?actor2 WHERE {?actor1 ^< }?actor2.

131 47 Navigating RDF graphs Previous query can be rewritten by using navigation patterns: SELECT?actor1?actor2 WHERE {?actor1 ^< }?actor2. The expression in red is called a property path

132 Navigating RDF graphs Get starring actors that are connected: SELECT?actor1?actor2 WHERE {?actor1 (^< < }

133 Navigating RDF graphs Get starring actors that are connected: SELECT?actor1?actor2 WHERE {?actor1 (^< < } Can this query be answered? Can it be answered starting from a specific node?

134 Open issues in exploiting the graph structure of RDF Some important problems: Development of efficient evaluation algorithms for reachability queries over large RDF graphs Standardization of a query language where paths are first-class citizens

135 50 Web data is highly distributed Data can be stored in different repositories Different pieces of data have to be collected to answer a query

136 50 Web data is highly distributed Data can be stored in different repositories Different pieces of data have to be collected to answer a query An important notion to deal with this issue: SPARQL endpoint A Web service that accepts a SPARQL query as input, and returns (part of) the result to the query

137 Web data is highly distributed Data can be stored in different repositories Different pieces of data have to be collected to answer a query An important notion to deal with this issue: SPARQL endpoint A Web service that accepts a SPARQL query as input, and returns (part of) the result to the query SPARQL has an operator SERVICE to query an endpoint

138 The SPARQL endpoint of DBpedia 51

139 Querying DBpedia We want to retrieve the list of American actors in DBpedia

140 Querying DBpedia We want to retrieve the list of American actors in DBpedia

141 53 The answer to the query...

142 The SPARQL endpoint of DBLP We want to retrieve the list of authors in DBLP

143 The SPARQL endpoint of DBLP We want to retrieve the list of authors in DBLP

144 55 The answer to the query...

145 We would like to combine the previous results... SELECT?name WHERE {?actor rdf:type < foaf:name?name. SERVICE < { SELECT?name WHERE {?paper dc:creator?author.?author foaf:name?name. } } }

146 57 Open issues when dealing with distribution Some important problems: The notion of SPARQL endpoint needs to be formalized What queries are accepted? How is the time distributed between them? Should a pricing model be used? What is the protocol to return the answer to a query? A more general notion of endpoint should be formalized and studied

147 Open issues when dealing with distribution (cont d) Usability needs to be hugely improved schema/structure extraction and visualization play a fundamental role here Approaches for discovering relevant data should be studied Operators to distribute the execution of queries should be studied in more depth

148 A first issue about the SERVICE operator SERVICE operators can be nested SELECT... WHERE {... SERVICE <uri1> {... SERVICE <uri2> } }

149 A second issue about the SERVICE operator SERVICE can be used not only with a URI but also with a variable

150 60 A second issue about the SERVICE operator SERVICE can be used not only with a URI but also with a variable SELECT... WHERE {...?address rdf:type < SERVICE?address {... } }

151 A second issue about the SERVICE operator SERVICE can be used not only with a URI but also with a variable SELECT... WHERE {... SERVICE?address {...?uri rdf:type < example.org/sparqlendpoint>. } SERVICE?uri {... } }

152 A second issue about the SERVICE operator SERVICE can be used not only with a URI but also with a variable Several specific issues have to be addressed [Buil-Aranda, A. & Corcho 2011]: If?var does not have a value, how SERVICE?var should be evaluated? A notion of safeness is needed The situation gets more involved if we also have nested SERVICE operators The syntax of SPARQL 1.1 allows SERVICE?var, but its semantics is not defined

153 What have we learned? There are many interesting and challenging questions to be answered

154 What have we learned? There are many interesting and challenging questions to be answered Most of these questions can be considered as classical DB questions

155 What have we learned? There are many interesting and challenging questions to be answered Most of these questions can be considered as classical DB questions But answering them require of a combination of classical and new techniques

156 What have we learned? There are many interesting and challenging questions to be answered Most of these questions can be considered as classical DB questions But answering them require of a combination of classical and new techniques The Semantic Web community has been receptive to our ideas

157 Tabular Data (CSV) 62

158 Comma-separated values (CSV) documents 63

159 63 Comma-separated values (CSV) documents The IBM Fortran compiler supported CSV documents in 1972

160 63 Comma-separated values (CSV) documents The IBM Fortran compiler supported CSV documents in 1972 The W3C recommendation for CSV documents (tabular data) was released on December 2015

161 63 Comma-separated values (CSV) documents The IBM Fortran compiler supported CSV documents in 1972 The W3C recommendation for CSV documents (tabular data) was released on December 2015 Why such a recommendation is needed? This is just tabular data, what is new?

162 The main goals of the W3C recommendation Define a schema language for CSV For example, it can be used to specify the name and the type of the elements of each column in a CSV document Develop a metadata vocabulary for CSV It describes how the data should be interpreted Define mechanisms for transforming CSV into RDF, JSON and XML

163 65 CSV documents can be messy A use case from The US National Institute of Standards and Technology has run various conferences on extracting information from text. The extracted information is submitted in a tab-separated format.

164 65 CSV documents can be messy A use case from The US National Institute of Standards and Technology has run various conferences on extracting information from text. The extracted information is submitted in a tab-separated format. An example document:... :e4 type PER :e4 mention "Bart" D :e4 mention "JoJo" D :e4 per:siblings :e7 D :e4 per:age "10" D

165 66 A schema language for CSV We describe an approach proposed by [Martens, Neven & Vansummeren 2015]

166 66 A schema language for CSV We describe an approach proposed by [Martens, Neven & Vansummeren 2015] The model for a CSV document: :e4 type PER 2 :e4 mention "Bart" D :e4 mention "JoJo" D :e4 per:siblings :e7 D

167 66 A schema language for CSV We describe an approach proposed by [Martens, Neven & Vansummeren 2015] The model for a CSV document: :e4 type PER 2 :e4 mention "Bart" D :e4 mention "JoJo" D :e4 per:siblings :e7 D Every cell has a coordinate (i, j) The cell with content "Bart" has coordinate (2, 3)

168 67 Checking conditions on regions :e4 type PER 2 :e4 mention "Bart" D :e4 mention "JoJo" D :e4 per:siblings :e7 D

169 67 Checking conditions on regions :e4 type PER 2 :e4 mention "Bart" D :e4 mention "JoJo" D :e4 per:siblings :e7 D [root]

170 67 Checking conditions on regions :e4 type PER 2 :e4 mention "Bart" D :e4 mention "JoJo" D :e4 per:siblings :e7 D [root] right

171 67 Checking conditions on regions :e4 type PER 2 :e4 mention "Bart" D :e4 mention "JoJo" D :e4 per:siblings :e7 D [root] right right

172 67 Checking conditions on regions :e4 type PER 2 :e4 mention "Bart" D :e4 mention "JoJo" D :e4 per:siblings :e7 D [root] right right right

173 67 Checking conditions on regions :e4 type PER 2 :e4 mention "Bart" D :e4 mention "JoJo" D :e4 per:siblings :e7 D [root] right right right right

174 67 Checking conditions on regions :e4 type PER 2 :e4 mention "Bart" D :e4 mention "JoJo" D :e4 per:siblings :e7 D [root] right right right right down

175 67 Checking conditions on regions :e4 type PER 2 :e4 mention "Bart" D :e4 mention "JoJo" D :e4 per:siblings :e7 D [root] right right right right down R

176 68 An annotation language We describe our proposal in [A., Maturana, Riveros & Vrgoč 2016]

177 68 An annotation language We describe our proposal in [A., Maturana, Riveros & Vrgoč 2016] A CSV document is represented as a string

178 An annotation language We describe our proposal in [A., Maturana, Riveros & Vrgoč 2016] A CSV document is represented as a string :e4 type PER :e4 mention "Bart" D :e4 mention "JoJo" D :e4 per:siblings :e7 D :e4 per:age "10" D is represented as: : e 4 \t t y p e \t P E R \n : e 4 \t m e n t i o n \t...

179 69 Annotation of regions Let w be the string representing a CSV document The main ingredients of our approach:

180 69 Annotation of regions Let w be the string representing a CSV document The main ingredients of our approach: A region of w is a span (i, j) [Fagin, Kimelfeld, Reiss & Vansummeren 2013]

181 69 Annotation of regions Let w be the string representing a CSV document The main ingredients of our approach: A region of w is a span (i, j) [Fagin, Kimelfeld, Reiss & Vansummeren 2013] Regular expression with variables are used to extract spans from w Fragments that can be evaluated very efficiently have been identified

182 69 Annotation of regions Let w be the string representing a CSV document The main ingredients of our approach: A region of w is a span (i, j) [Fagin, Kimelfeld, Reiss & Vansummeren 2013] Regular expression with variables are used to extract spans from w Fragments that can be evaluated very efficiently have been identified Datalog programs are used to combine the results of the extraction process and annotate spans [Shen, Doan, Naughton & Ramakrishnan 2007]

183 Annotation of regions Recall that w is the string representation of: :e4 type PER :e4 mention "Bart" D :e4 mention "JoJo" D :e4 per:siblings :e7 D :e4 per:age "10" D Assume that the alphabet of w is Σ and = (Σ {\t, \n})

184 Annotation of regions Recall that w is the string representation of: :e4 type PER :e4 mention "Bart" D :e4 mention "JoJo" D :e4 per:siblings :e7 D :e4 per:age "10" D Assume that the alphabet of w is Σ and = (Σ {\t, \n}) The following Datalog program extracts and annotates all spans in the first column of w:

185 Annotation of regions Recall that w is the string representation of: :e4 type PER :e4 mention "Bart" D :e4 mention "JoJo" D :e4 per:siblings :e7 D :e4 per:age "10" D Assume that the alphabet of w is Σ and = (Σ {\t, \n}) The following Datalog program extracts and annotates all spans in the first column of w: w.x\tσ x. FC(x)

186 Annotation of regions Recall that w is the string representation of: :e4 type PER :e4 mention "Bart" D :e4 mention "JoJo" D :e4 per:siblings :e7 D :e4 per:age "10" D Assume that the alphabet of w is Σ and = (Σ {\t, \n}) The following Datalog program extracts and annotates all spans in the first column of w: w.x\tσ x. FC(x) w.σ \nx\tσ x. FC(x)

187 Open issues in the area Many questions need to be answered: What is a good schema language for CSV? What is a good annotation language for CSV? How metadata should be specified for a CSV document? What is a good mapping language to specify the transformation of a CSV document to an RDF graph? The same question needs to be answered for JSON and XML

188 72 Final remarks Do not underestimate the theoretical interest behind new apps: Classical DB questions gain new flavors over new data models It might involve developing techniques of independent interest Some areas are in need of theoretical help It requires an effort to understand problems and transfer tools

189 72 Final remarks Do not underestimate the theoretical interest behind new apps: Classical DB questions gain new flavors over new data models It might involve developing techniques of independent interest Some areas are in need of theoretical help It requires an effort to understand problems and transfer tools Many problems are still in need of formalization structure extraction, access, trust,...

190 Thank you! 73