Academic research on graph processing: connecting recent findings to industrial technologies. Gábor Szárnyas opencypher NYC

Transcription

1 Academic research on graph processing: connecting recent findings to industrial technologies Gábor Szárnyas opencypher NYC

2 LINKED DATA BENCHMARK COUNCIL LDBC is a non-profit organization dedicated to establishing benchmarks, benchmark practices and benchmark results for graph data management software. LDBC s Social Network Benchmark is an industrial and academic initiative, formed by principal actors in the field of graph-like data management.

3 OVERVIEW OF GRAPH PROCESSING OLTP OLAP analytics local queries global queries global computations

4 OVERVIEW OF GRAPH PROCESSING OLTP local queries Example: Friends recent likes MATCH (u:user {id: $userid})-[:friend]- (f:user)-[l:likes]->(p:post) RETURN f, p ORDER BY l.timestamp DESC LIMIT 10 OLAP analytics global queries global computations

5 OVERVIEW OF GRAPH PROCESSING OLTP local queries limited data frequent up. Orri Erling et al., The LDBC Social Network Benchmark: Interactive Workload, SIGMOD queries and 8 updates OLAP analytics global queries global computations

6 OVERVIEW OF GRAPH PROCESSING OLTP local queries limited data frequent up. OLAP global queries Example: One-sided friendships MATCH (u1:user)-[:friend]-(u2:user)-[l:likes]->(p:post), (u1)-[:author_of]->(p) WITH u1, u2, count(l) AS likes WHERE likes > 10 AND NOT (u1)-[:likes]->(:post)<-[:author_of]-(u2) RETURN u1, u2 analytics global computations

7 OVERVIEW OF GRAPH PROCESSING OLTP local queries limited data frequent up. OLAP global queries lots of data infrequent up. Arnau Prat, Gábor Szárnyas, Alex Averbuch et al., The LDBC Social Network Benchmark: BI Workload, Technical report available, peer-reviewed paper in queries with infrequent executions analytics global computations

8 OVERVIEW OF GRAPH PROCESSING OLTP local queries limited data frequent up. OLAP global queries lots of data infrequent up. analytics global computations PageRank Shortest paths Clustering coefficient Example: Find the most central individuals. Neo4j: Graph Algorithms library

9 OVERVIEW OF GRAPH PROCESSING OLTP local queries limited data frequent up. OLAP global queries lots of data infrequent up. analytics global computations all data no updates Alexandru Iosup et al., LDBC Graphalytics: A Benchmark for Large-Scale Graph Analysis on Parallel and Distributed Platforms, VLDB 2016 One-time execution

10 OVERVIEW OF GRAPH PROCESSING OLTP OLAP analytics local queries limited data frequent up. global queries lots of data infrequent up. global computations all data no updates

11 OVERVIEW OF GRAPH PROCESSING OLTP local queries OLAP global queries analytics global computations validation global queries limited data lots of data all data frequent up. infrequent up. no updates Example: Emergency contact for juvenile users MATCH (u1:user) WHERE u1.age < 18 AND NOT (u1)-[:emergency_contact]->(:user) RETURN u1

12 OVERVIEW OF GRAPH PROCESSING OLTP OLAP analytics local queries global queries global computations limited data lots of data all data frequent up. infrequent up. no updates validation global queries lots of data frequent up. Gábor Szárnyas et al. The Train Benchmark: cross-technology performance evaluation of continuous model queries, Software and Systems Modeling, 2017

13 FAULT-TOLERANT SYSTEMS RESEARCH GROUP Critical systems Avionics Railway Automotive

14 MODEL-DRIVEN ENGINEERING Models are first class citizens during development o SysML / requirements, statecharts, etc. Validation and code generation techniques for correctness Technology: Eclipse Modeling Framework (EMF) Originally started at IBM as an implementation of the Object Management Group s (OMG) Meta Object Facility (MOF). i.e., an object-oriented model i.e., a property graph-like structure with a metamodel

15 MODEL VALIDATION Implemented with model queries Models are typed, attributed graphs Complex graph queries Typical queries o Get two components connected by a particular edge MATCH (r:r) (s:s) WHERE NOT (r)-[:e]->(s) o Check if two objects are reachable MATCH (r:r) (s:s) WHERE NOT (r)-[:e1 E2*]->(s) o Property checks MATCH (r:r)-->(s:s) WHERE r.a = 'x' OR (s:y)

16 route 1 2

17 1 2

18

19

20

21

22

23 route: Route swp: SwitchPosition NEG sensor: Sensor sw: Switch

24 route: Route swp: SwitchPosition NEG sensor: Sensor sw: Switch

25 route: Route swp: SwitchPosition NEG sensor: Sensor sw: Switch MATCH (route:route)

26 route: Route swp: SwitchPosition NEG sensor: Sensor sw: Switch MATCH (route:route) -->(swp:switchposition)

27 route: Route swp: SwitchPosition NEG sensor: Sensor sw: Switch MATCH (route:route) -->(swp:switchposition) -->(sw:switch)

28 route: Route swp: SwitchPosition NEG sensor: Sensor sw: Switch MATCH (route:route) -->(swp:switchposition) -->(sw:switch) <--(sensor:sensor)

29 route: Route swp: SwitchPosition NEG sensor: Sensor sw: Switch MATCH (route:route) -->(swp:switchposition) -->(sw:switch) <--(sensor:sensor) WHERE NOT (route)-->(sensor)

30 route: Route swp: SwitchPosition NEG sensor: Sensor sw: Switch MATCH (route:route) -->(swp:switchposition) -->(sw:switch) <--(sensor:sensor) WHERE NOT (route)-->(sensor) RETURN route, sensor, swp, sw

31

32

33

34

35

36

37

38

39 LOCAL SEARCH-BASED QUERY EVALUATION Matching: P G (graph morphism) Constraints satisfaction on a finite domain (CSP/FD): o Variables: vertices of P o Constraints: edges of P o Domain values: G Complexity: G P G. Varró, F. Deckwerth, M. Wieber, A. Schürr, An algorithm for generating model-sensitive search plans for pattern matching on EMF models, Software and Systems Modeling, 2013

40

41

42 00

43 00 01

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75 00

76 00 01

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102 SEARCH SPACE OF THE GRAPH SEARCH PROBLEM Information on cardinalities o Metamodel-level o Model-level Index structures Homogeneity Frequency of changes and queries K. Zeng et al. (Microsoft Research), A Distributed Graph Engine for Web Scale RDF Data, VLDB 2013

103 TOP VENUES ON MODEL-DRIVEN ENGINEERING Journal on Software and Systems Modeling (Springer) Journal of Systems and Software (Elsevier) ACM/IEEE MODELS JOURNALS CONFERENCES odd years even years FASE: Fundamental Approaches to Software Engineering STAF: Software Technology Applications and Foundations o ICMT/ICGT: International Conference on Model/Graph Transformation o TTC: Transformation Tool Contest

104 MATCH (a1:actor)-[:plays_in]->(m:movie) <-[:PLAYS_IN]-(a2:Actor) WITH a1, a2, count(m) AS moviecount WHERE moviecount >= 3 RETURN a1, a2, moviecount

105 TRAIN BENCHMARK FRAMEWORK Scalable graph generator EMF Property graph RDF SQL Validation queries and model transformations Implemented for 12+ tools G. Szárnyas, B. Izsó, I. Ráth, D. Varró, The Train Benchmark: cross-technology performance evaluation of continuous model queries, Software and Systems Modeling, 2017 ftsrg/trainbenchmark

106 MODEL-DRIVEN ENGINEERING TOOLS VIATRA framework: reactive model transformations

107 OTHER COMPUTER SCIENCE FIELDS Semantic web o Semantic graphs built from triples o Ontologies for metamodeling o SPARQL graph queries Object-oriented databases o Big hype in the 90s o Lots of similarity to EMF and potentially others.

108 SUMMARY MDE has a lot of graph query problems Lots of research has been conducted Chance to avoid reinventing the wheel o Pattern matching algorithms o Transformation semantics o Performance benchmarks

109 RELATED RESOURCES Train Benchmark Incremental Graph Engine LDBC Benchmarks List of papers: github.com/ftsrg/trainbenchmark github.com/ftsrg/ingraph github.com/ldbc github.com/szarnyasg/mde-graph-processing Siddhartha Sahu et al. (University of Waterloo), The Ubiquity of Large Graphs and Surprising Challenges of Graph Processing: A User Survey, arxiv preprint, 2017