CIP Multiple Graphs. Presentation at ocig 8 on April 11, Stefan Plantikow, Andrés Taylor, Petra Selmer (Neo4j)

Transcription

1 CIP Multiple Graphs Presentation at ocig 8 on April 11, 2018 Stefan Plantikow, Andrés Taylor, Petra Selmer (Neo4j)

2 History Started working on multiple graphs early 2017 Parallel work in LDBC query language task force Built first non-prototype proposal for ocim 3 in Nancy (Nov 2017) Lots of great feedback but outcome incomplete Inspired by Nancy we started to reconstruct the MG proposal Starting from first principles (data model, execution model, language semantics) Incrementally add orthogonal concern Aim for simple, minimal design Reduce scope of work Today: Primer for core of new proposal draft

3 Outline Multiple property graphs model Query processing model (Catalog and working graph) Working with multiple graphs (Matching, updating, returning graphs) Graph operations (Graph projection and graph union) Updating the catalog Examples Discussion (Status, timeline, next steps)

4 Why multiple graphs? - Combining and transforming graphs from multiple sources Graph Management - Versioning, snapshotting, computing difference graphs - Graph views for access control - Provisioning applications with tailored graph views - Shaping and integrating heterogeneous graph data Graph Modeling - Roll-up and drill-down at different levels of detail 4

5 Multiple Property Graph Model

6 Cypher today: (Single) property graph model Graph Database System (e.g. a cluster) Application Server Client 1 Client 2 Client 3 The (single) Graph 6

7 Cypher 10: Multiple property graphs model Graph Database System (e.g. a cluster) Application Server Client 1 Client 2 Client 3 Multiple Graphs 7

8 Classic Property Graph Model A single graph Labeled nodes with properties Typed, directed relationships with properties Multiple Property Graphs Model Multiple graphs Labeled nodes with properties and typed, directed relationships with properties existing in a single graph The nodes of a relationship R must be part of the same graph as R; i.e. each graph is independent 8

9 Graph and entity ids The graph(entity) function returns the graph id of the entity An entity (a node or a relationship) in Cypher 10 needs to track its graph (to differentiate whether or not two entities are from the same graph) graph(entity) is a value that uniquely identifies a graph within the implementation This is a two-way process: a graph will also be aware of the entities it contains The id(entity) function returns the (entity) id of entity in graph(entity) id(entity) is a value that uniquely identifies an entity within its graph and types (i.e. nodes and relationships do not share the id space) The format of ids is implementation specific This does not change between Cypher 9 and Cypher 10 9

10 Id validity and comparability Validity of ids describes the duration for which an id is not assigned to another object. Validity of graph and entity ids are only guaranteed for the duration of a query! Validity of entity ids is only guaranteed from the matching of an entity to the consumption of a query result. In other words: for the duration of the query. Validity, entity ids are guaranteed to not be re-used per graph for the duration of the query. These are the minimal lifetime guarantees mandated by Cypher 10. Implementations may provide extended validity guarantees as they see fit. Comparability of ids Graph and entity ids are suggested to be comparable values, i.e. can be compared with <, <=, =, >=, > 10

11 Query Processing Model

12 Query processor Query Processor = Query Processing Service (e.g. a single server or a cluster) G1 G2 G3 Catalog 12

13 Graph catalog A graph catalog maps graph names to graphs A qualified graph name is an (optionally dotted) identifier A graph name may be contextual; i.e. only work inside a specific server database... There are no further restrictions placed on graphs; e.g. graphs may be mutable or immutable exist only for a limited time (session, transaction,...) visible to all or only some of the users of the system... 13

14 Syntax for qualified graph names Qualified graph name syntax follows function name syntax Parts graph namespace (optional) graph name Syntax foo.bar.baz `foo`.`bar` `foo.bar` `foo.bar`.`baz.grok` (namespace foo.bar, graph baz.grok) Number of periods can be 0 to * Namespace and name separation is implementation-dependent Last part is always a suffix of the graph name 14

15 Working graph 1. A clause is always operating within the context of a working graph a. by reading from it b. by updating it 2. During the course of executing a query a. the working graph may be assigned by name (i.e. assigned to a graph from the catalog) b. the working graph may be assigned by graph projection (i.e. assigned by creating a new working graph) c. the working graph may be returned as a query result d. the graph id of the current working graph may be obtained using the graph() function 15

16 Initial working graph Determined by Query Processor This allows for implementation freedom Example This includes having no initial working graph Queries that need an initial working graph fail if no working graph has been set In Neo4j: Upon establishing a session via Bolt, an initial working graph to be used by subsequent queries is established (perhaps per user etc.) In CAPS: Cypher queries are always executed explicitly against a graph which implicitly becomes the initial working graph 16

17 Working graph operations Assign working graph by name (from catalog lookup) Access multiple graphs within one query Assign working graph by graph projection (next section) Consolidate data from one or more data sets into one graph Return (working) graph from query Construct temporary graph via multiple update queries (e.g. during interactive workflow) Return (export) graphs (e.g. system graph) 17

18 Working with multiple graphs

19 Select working graph // Set the working graph to the graph foo in the catalog for reading FROM foo // Set the working graph to the graph foo in the catalog for updating UPDATE foo // Change between reading and updating FROM GRAPH UPDATE GRAPH In any case, Does not consume the current cardinality Retains all variable bindings 19

20 Example: Reading from multiple graphs Which friends to invite to my next dinner party? [1] FROM social_graph [2] MATCH (a:person {ssn: $ssn})-[:friend]-(friend) [3] FROM salary_graph [4] MATCH (f:entity {ssn: friend.ssn})<-[:income]-(event) [5] WHERE $startdate < event.date < $enddate [6] RETURN friend.name, sum(event.amount) as incomes 20

21 Matching with multiple graphs FROM persons MATCH (a:person) FROM cars MATCH (b:car)-[:owned_by]->(a) // Does NOT match Updating multiple graphs FROM persons MATCH (a:person) UPDATE cars CREATE (b:car)-[:owned_by]->(a) // Error

22 Copy patterns How to copy data between graphs? FROM persons MATCH (a:person) UPDATE cars CREATE (b:car)-[:owned_by]->(x COPY OF a) // This works COPY OF node Copies labels, properties COPY OF rel Copies relationship type, properties (ignores start, end!) Also usable in matching: MATCH (COPY OF a)... 22

23 Returning a graph Cypher 10 queries may return a graph Three variations of consuming a graph returned by a query a. Deliver graph to client b. Pass graph to another query; e.g. within a subquery (not covered in these slides) c. Store graph in the catalog; i.e. registering a returned graph under a new name Delivering a result graph from the query to the client is always by value a. All nodes with their labels and properties b. All relationships with their relationship type, start node, end node, and their properties c. Result graph is completely independent from graphs managed by the query processor 23

24 Syntax for returning a graph // Returns the working graph... RETURN GRAPH // Removing the current cardinality while keeping the working graph... WITH GRAPH... // Return a graph from the catalog FROM graphname RETURN GRAPH 24

25 Graph operations

26 Graph operations Graph construction Common graph union Disjoint graph union

27 Graph construction Graph construction dynamically constructs a new working graph for querying, storing in the catalog, later updating using entities from other graphs (this is called cloning) Simple example MATCH (a)-[:knows {from: "Berlin"}]->(b) CONSTRUCT CLONE a AS x, b AS y CREATE (x)-[:met_in_berlin]->(y) RETURN GRAPH

28 Cloning nodes Take an original entity a and create a representative clone c in the constructed graph MATCH (a) CONSTRUCT CLONE a AS c // properties(a) = properties(c) AND labels(a) = labels(c) // but: a <> c because graph(a) <> graph(c)... Cloning the same original multiple times still only creates a single clone MATCH (root)-[:parent_of*]->(child) CONSTRUCT CLONE root, child // CLONE can shadow variables like WITH MERGE (root)-[:ancestor_of]->(child)...

29 Cloning relationships Cloning a relationship r implicitly clones startnode(r) and endnode(r) Cloning is powerful enough to reconstruct whole subgraphs! FROM car_owners MATCH (a:person {city: "Berlin"})-[r:OWNERS]->(c:Car) CONSTRUCT CLONE r // cars and their owners from berlin RETURN GRAPH

30 Clone paths and complex values Cloning a path p clones all nodes(p) and rels(p) More generally, cloning a value clones all contained entities! MATCH (a:person), (b:person) MATCH p = (a)-[:knows*]-(b) CONSTRUCT CLONE a, b, p CREATE (a)<-[:start]-(x:path {steps: size(p)})-[:end]->(b) RETURN GRAPH

31 Cloning graphs Clone all entities from given graphs into a new graph CONSTRUCT ON social_graph Clone all entities from working graph into new graph CONSTRUCT ON GRAPH Cloning a graph and a contained entity still only creates a single clone for the entity MATCH (a) WITH * LIMIT 1 CONSTRUCT ON GRAPH CLONE a AS x // creates only one x for the given a in the whole graph // (re-using clones created by CONSTRUCT ON GRAPH)

32 Finding clones Sometimes it may be necessary to find clones instead of creating them MATCH (x)-[:owns]->(y) CONSTRUCT CLONE x AS c CREATE (x)-[:owns]->(copy OF y)... WITH x, count(*) AS num_clones // lost x here! MATCH (CLONE OF x) // found the clone of x again...

33 General form of CONSTRUCT CONSTRUCT [ON GRAPH] [ON graph1, graph2,...] [CLONE expr [AS alias], variable,... * ] <updating clause 1> <updating clause 2>... WITH... WITH GRAPH RETURN... RETURN GRAPH

34 Provenance tracking So far, we've only looked at one step of graph construction. What about? [1] MATCH (a) // single node a [2] CONSTRUCT [3] CLONE a AS c // c <> a, it is a clone of a [4] WITH a, c [5] CONSTRUCT [6] CLONE a AS x, c AS y // x <> a is a clone of a, y <> c is a clone of c [7]... But x = y (i.e.they are the same clone). This is called provenance tracking. It's useful for updatable views and graph union.

35 Updatable views CONSTRUCT... // build the view... WITH GRAPH UPDATE GRAPH... // update the view...

36 Updates during graph construction General rule: Maintain logical consistency regarding provenance tracking CREATE Create in the constructed graph MERGE Merge in the constructed graph SET REMOVE Not allowed on clones (for now) [DETACH] DELETE Delete from the constructed graph

37 Updates after graph construction General rule: Only allow updates that can be propagated safely CREATE Not allowed MERGE SET REMOVE Not allowed Update base data [DETACH] DELETE Not allowed

38 Common graph union Clones of same original node are cloned, other entities are copied CONSTRUCT... RETURN GRAPH UNION CONSTRUCT... RETURN GRAPH

39 Disjoint graph union All entities (including clones) are copied (i.e. this breaks provenance tracking) CONSTRUCT... RETURN GRAPH UNION ALL CONSTRUCT... RETURN GRAPH

40 Updating the catalog

41 Catalog side-effects CREATE GRAPH foo // Create new graph 'foo' DELETE GRAPH foo // Delete graph 'foo' COPY foo TO bar // Copy graph 'foo' with schema RENAME foo TO bar // Rename graph 'foo' to 'bar' TRUNCATE foo // Remove data but keep schema in 'foo' 41

42 Create new base graph (snapshot) in catalog // create a new base graph in the catalog CREATE GRAPH mygraph // copy content of graph as new graph into the catalog CREATE GRAPH bar { <some query that returns a graph> } These are always top-level clauses; i.e. they may not be nested 42

43 Copy data from one base graph to another We can accomplish this by using DML and hand crafting the queries MATCH (a)-[r]->(b) UPDATE foo CREATE... MERGE... SET... 43

44 Schema updates Need to select the graph UPDATE foo CREATE CONSTRAINT... CREATE INDEX // Different keyword? 44

45 Catalog function catalog() catalog(graph(e)) Current working graph name in catalog or NULL Name of graph of e in catalog or NULL

46 Summary

47 Status Working on two CIPs Multiple graphs CIP (This talk, also: larger examples) Query composition CIP Features: views, nested subqueries, equivalence, more graph operations Future work: More elaborate updatable views, graph aggregation Goal is to finalize draft of multiple graphs CIP for next ocim Design heavily influenced by feedback from CAPS project and other implementation concerns

48 Feedback Please feedback by commenting on slides or CIP or opencypher implementers slack Thanks!