Batch & Stream Graph Processing with Apache Flink. Vasia

Transcription

1 Batch & Stream Graph Processing with Apache Flink Vasia

2

3 Outline Distributed Graph Processing Gelly: Batch Graph Processing with Flink Gelly-Stream: Continuous Graph Processing with Flink

4 WHEN DO YOU NEED DISTRIBUTED GRAPH PROCESSING?

5 MISCONCEPTION # MY GRAPH IS SO BIG, IT DOESN T FIT IN A SINGLE MACHINE Big Data Ninja

6 A SOCIAL NETWORK

7 YOUR INPUT DATASET SIZE IS _OFTEN_ IRRELEVANT

8 INTERMEDIATE DATA: THE OFTEN Naive Who(m) to Follow: compute a friends-of-friends list per user exclude existing friends rank by common connections

9 MISCONCEPTION # DISTRIBUTED PROCESSING IS ALWAYS FASTER THAN SINGLE-NODE Data Science Rockstar

10

11 GRAPHS DON T APPEAR OUT OF THIN AIR Expectation

12 GRAPHS DON T APPEAR OUT OF THIN AIR Reality!

13 HOW DO WE EXPRESS A DISTRIBUTED GRAPH ANALYSIS TASK?

14 GRAPH APPLICATIONS ARE DIVERSE Iterative value propagation PageRank, Connected Components, Label Propagation Traversals and path exploration Shortest paths, centrality measures Ego-network analysis Personalized recommendations Pattern mining Finding frequent subgraphs

15 LINEAR ALGEBRA 3 5 Adjacency Matrix Partition by rows, columns, blocks - Efficient representation of non-zero elements - Algorithms expressed as vector-matrix multiplications

16 BREADTH-FIRST SEARCH

17 BREADTH-FIRST SEARCH

18 BREADTH-FIRST SEARCH X =

19 BREADTH-FIRST SEARCH X =

20 BREADTH-FIRST SEARCH X =

21 RECENT DISTRIBUTED GRAPH PROCESSING HISTORY Graph Traversals Pattern Matching MapReduce Pregel Giraph++ Arabesque Pegasus Signal-Collect PowerGraph NScale Tinkerpop Iterative value propagation Ego-network analysis

22 PREGEL: THINK LIKE A VERTEX 3,,

23 PREGEL: SUPERSTEPS Superstep i Superstep i+ 3, 3,,., (V i+, outbox) < compute(v i, inbox)

24 PAGERANK: THE WORD COUNT OF GRAPH PROCESSING VertexID Out-degree Transition Probability / / /3 5

25 PAGERANK: THE WORD COUNT OF GRAPH PROCESSING VertexID Out-degree Transition Probability / / /3 5 PR(3) = 0.5*PR() *PR() + PR(5)

26 PREGEL EXAMPLE: PAGERANK void compute(messages): sum = 0.0 sum up received messages for (m <- messages) do sum = sum + m end for update vertex rank setvalue(0.5/numvertices() *sum) for (edge <- getoutedges()) do sendmessageto( edge.target(), getvalue()/numedges) end for distribute rank to neighbors

27 SIGNAL-COLLECT Signal Superstep i Collect Superstep i+ 3, 3, 3,,,, outbox < signal(v i ) V i+ < collect(inbox)

28 SIGNAL-COLLECT EXAMPLE: PAGERANK void signal(): for (edge <- getoutedges()) do sendmessageto( edge.target(), getvalue()/numedges) end for distribute rank to neighbors void collect(messages): sum = 0.0 for (m <- messages) do sum = sum + m end for sum up received messages update vertex rank setvalue(0.5/numvertices() *sum)

29 GATHER-SUM-APPLY (POWERGRAPH) Superstep i Superstep i+ Gather Sum Apply Gather

30 GSA EXAMPLE: PAGERANK double gather(source, edge, target): return target.value() / target.numedges() double sum(rank, rank): return rank + rank combine partial ranks compute partial rank double apply(sum, currentrank): return *sum update rank

31 PROBLEMS WITH VERTEX-CENTRIC MODELS Excessive communication Worker load imbalance Global Synchronization High memory requirements inbox /outbox can grow too large overhead for low-degree vertices in GSA

32 Vertex-Centric Connected Components Propagate the minimum value through the graph In each superstep, the value propagates one hop Requires diameter + supersets to converge

33 THINK LIKE A (SUB)GRAPH compute() on the entire partition - Information flows freely inside each partition - Network communication between partitions, not vertices 3 5

34 Subgraph-Centric Connected Components In each superstep, the value propagates throughout each subgraph Communication between partitions only Requires less (possibly) supersteps to converge

35 RECENT DISTRIBUTED GRAPH PROCESSING HISTORY Graph Traversals Pattern Matching MapReduce Pregel Giraph++ Arabesque Pegasus Signal-Collect PowerGraph NScale Tinkerpop Iterative value propagation Ego-network analysis

36 CAN WE HAVE IT ALL? Data pipeline integration: built on top of an efficient distributed processing engine Graph ETL: high-level API with abstractions and methods to transform graphs Familiar programming model: support popular programming abstractions

37 Gelly the Apache Flink Graph API

38 Flink Stack Hadoop M/R Table Gelly ML Dataflow Cascading Table CEP SAMOA Dataflow (WiP) DataSet (Java/Scala) DataStream (Java/Scala) Streaming dataflow runtime Local Remote Yarn Embedded

39 Why Graph Processing with Apache Flink? Native Iteration Operators DataSet Optimizations Ecosystem Integration

40 Flink Iteration Operators Result Result Iterative Update Function Replace Iterative Update Function State Input Workset Solution Set

41 Optimization the runtime is aware of the iterative execution no scheduling overhead between iterations caching and state maintenance are handled automatically Push work out of the loop Cache Loop-invariant Data Maintain state as index

42 Beyond Iterations Performance & Scalability Memory management Efficient serialization framework Operations on binary data Automatic Optimizations Choose best execution strategy Cache invariant data

43 Meet Gelly Java & Scala Graph APIs on top of Flink graph transformations and utilities iterative graph processing library of graph algorithms Can be seamlessly mixed with the DataSet Flink API to easily implement applications that use both record-based and graph-based analysis

44 Hello, Gelly! Java ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); DataSet<Edge<Long, NullValue>> edges = getedgesdataset(env); Graph<Long, Long, NullValue> graph = Graph.fromDataSet(edges, env); DataSet<Vertex<Long, Long>> verticeswithminids = graph.run( new ConnectedComponents(maxIterations)); Scala val env = ExecutionEnvironment.getExecutionEnvironment val edges: DataSet[Edge[Long, NullValue]] = getedgesdataset(env) val graph = Graph.fromDataSet(edges, env) val components = graph.run(new ConnectedComponents(maxIterations))

45 Graph Methods Graph Properties getvertexids getedgeids numberofvertices numberofedges getdegrees... Transformations map, filter, join subgraph, union, difference reverse, undirected gettriplets Mutations add vertex/edge remove vertex/edge

46 Example: mapvertices // increment each vertex value by one val graph = Graph.fromDataSet(...) // increment each vertex value by one val updatedgraph = graph.mapvertices(v => v.getvalue + )

47 Example: subgraph val graph: Graph[Long, Long, Long] =... // keep only vertices with positive values // and only edges with negative values val subgraph = graph.subgraph( ) vertex => vertex.getvalue > 0, edge => edge.getvalue < 0

48 Neighborhood Methods Apply a reduce function to the st-hop neighborhood of each vertex in parallel graph.reduceonneighbors( new MinValue, EdgeDirection.OUT)

49 Iterative Graph Processing Gelly offers iterative graph processing abstractions on top of Flink s Delta iterations vertex-centric scatter-gather gather-sum-apply partition-centric*

50 Vertex-Centric SSSP final class SSSPComputeFunction extends ComputeFunction { override def compute(vertex: Vertex, messages: MessageIterator) = { var mindistance = if (vertex.getid == srcid) 0 else Double.MaxValue while (messages.hasnext) { val msg = messages.next if (msg < mindistance) mindistance = msg } } if (vertex.getvalue > mindistance) { setnewvertexvalue(mindistance) for (edge: Edge <- getedges) sendmessageto(edge.gettarget, vertex.getvalue + edge.getvalue)

51 Library of Algorithms PageRank* Single Source Shortest Paths* Label Propagation Weakly Connected Components* Community Detection Triangle Count & Enumeration Clustering Coefficient Jaccard & Adamic-Adar Similarity Graph Summarization val ranks = inputgraph.run(new PageRank(0.85, 0)) *: both scatter-gather and GSA implementations

52 Gelly-Stream single-pass stream graph processing with Flink

53 Real Graphs are dynamic Graphs are created from events happening in real-time

54

55 How we ve done graph processing so far. Load: read the graph from disk and partition it in memory

56 How we ve done graph processing so far. Load: read the graph from disk and partition it in memory. Compute: read and mutate the graph state

57 How we ve done graph processing so far. Load: read the graph from disk and partition it in memory. Compute: read and mutate the graph state 3. Store: write the final graph state back to disk

58 What s wrong with this model? It is slow wait until the computation is over before you see any result pre-processing and partitioning It is expensive lots of memory and CPU required in order to scale It requires re-computation for graph changes no efficient way to deal with updates

59 Graph Streaming Challenges Maintain the dynamic graph structure Provide up-to-date results with low latency Compute on fresh state only

60 Single-Pass Graph Streaming Each event is an edge addition Maintain only a graph summary Recent events are grouped in graph windows

61

62 Graph Summaries spanners for distance estimation sparsifiers for cut estimation sketches for homomorphic properties graph summary algorithm R ~ R algorithm

63 Batch Connected Components i=

64 Batch Connected Components 3 i=

65 Batch Connected Components i= 6 6 6

66 Batch Connected Components i=

67 Batch Connected Components i= 6 6 6

68 6 8 Stream Connected Components 5 3 Graph Summary: Disjoint Set (Union-Find) Only store component IDs and vertex IDs

69 3 6 8 Cid =

70 7 8 3 Cid = Cid =

71 7 8 3 Cid = Cid =

72 7 8 Cid = Cid = Cid = 6 7 3

73 Cid = Cid = Cid =

74 Cid = Cid = Cid =

75 Cid = Cid = Cid =

76 Cid = Cid =

77 Distributed Stream Connected Components

78 Stream Connected Components with Flink DataStream<DisjointSet> cc = edgestream.keyby(0).timewindow(time.of(00, TimeUnit.MILLISECONDS)).fold(new DisjointSet(), new UpdateCC()).flatMap(new Merger()).setParallelism();

79 Stream Connected Components with Flink DataStream<DisjointSet> cc = edgestream.keyby(0).timewindow(time.of(00, TimeUnit.MILLISECONDS)).fold(new DisjointSet(), new UpdateCC()).flatMap(new Merger()).setParallelism(); Partition the edge stream

80 Stream Connected Components with Flink DataStream<DisjointSet> cc = edgestream.keyby(0).timewindow(time.of(00, TimeUnit.MILLISECONDS)).fold(new DisjointSet(), new UpdateCC()).flatMap(new Merger()).setParallelism(); Define the merging frequency

81 Stream Connected Components with Flink DataStream<DisjointSet> cc = edgestream.keyby(0).timewindow(time.of(00, TimeUnit.MILLISECONDS)).fold(new DisjointSet(), new UpdateCC()).flatMap(new Merger()).setParallelism(); merge locally

82 Stream Connected Components with Flink DataStream<DisjointSet> cc = edgestream.keyby(0).timewindow(time.of(00, TimeUnit.MILLISECONDS)).fold(new DisjointSet(), new UpdateCC()).flatMap(new Merger()).setParallelism(); merge globally

83 Gelly on Streams Static Graphs Multi-Pass Algorithms Full Computations Dynamic Graphs Single-Pass Algorithms Approximate Computations Gelly Gelly-Stream DataSet DataStream Distributed Dataflow Deployment

84 Introducing Gelly-Stream Gelly-Stream enriches the DataStream API with two new additional ADTs: GraphStream: A representation of a data stream of edges. Edges can have state (e.g. weights). Supports property streams, transformations and aggregations. GraphWindow: A time-slice of a graph stream. It enables neighborhood aggregations

85 GraphStream Operations Property Streams Transformations GraphStream -> DataStream.getEdges().getVertices().numberOfVertices().numberOfEdges().getDegrees().inDegrees().outDegrees() GraphStream -> GraphStream.mapEdges();.distinct();.filterVertices();.filterEdges();.reverse();.undirected();.union();

86 Graph Stream Aggregations graphstream.aggregate( new MyGraphAggregation(window, fold, combine, transform)) graph stream edges fold (window) fold agg global aggregates can be persistent or transient reduce combine result aggregate property stream local summaries global summary

87 Slicing Graph Streams graphstream.slice(time.of(, MINUTE)); :0 : : :3

88 Aggregating Slices source graphstream.slice(time.of(, MINUTE), direction) target Aggregations.reduceOnEdges();.foldNeighbors();.applyOnNeighbors(); Slicing collocates edges by vertex information Neighborhood aggregations on sliced graphs

89 Finding Matches Nearby graphstream.filtervertices(graphgeeks()).slice(time.of(5, MINUTE), EdgeDirection.IN).applyOnNeighbors(FindPairs()) GraphWindow :: user-place GraphStream :: graph geek check-ins wendy wendy checked_in soap_bar steve checked_in soap_bar tom checked_in joe s_grill sandra checked_in soap_bar rafa checked_in joe s_grill slice steve soap bar tom rafa joe s grill sandra FindPairs {wendy, steve} {steve, sandra} {wendy, sandra} {tom, rafa}

90 Feeling Gelly? Gelly Guide gelly_guide.html Gelly-Stream Repository Gelly-Stream Related Papers