Approximate Integration of Streaming data

Transcription

1 Approximate Integration of Streaming data Michel de Rougemont, Guillaume Vimont University Paris II & Irif

2 Plan 1. Approximation for Datawarehouses: Boolean queries Analytic queries 2. Streaming Datawarehouses Reservoir sampling Community in Graphs via Uniform sampling A good approximation for some random graphs 3. Data Integration for streams Compress streams with «good representations» with h.p. Define the Integration from this compressed forms The «value» of the data increases with the integration

3 1. OLAP Queries for a Datawarehouse OLAP queries (Analytic queries): filter, dimensions, measure, aggregation Dimension: Channel Measure: Sentiment Analysis Aggregation: Sum Datawarehouse Tweets with Sentiment Analysis (measure in [0 9])

4 Approximation of OLAP queries PBS 200 2/5 CNN 300 3/5 PBS 225 CNN 275 Result is a distribution μ Distance between two distributions: L 1 [μ-μ ]=0.1 i.e. 10%

5 Uniform samples, Weighted samples Take N uniform samples: Prob Ω [[μ-μ ]<ε ] > 1-δ N=O(log(1/ δ ).c/ε 2 ) Prob Ω [[μ-μ ]<0.1 ] > 0.9

6 Approximation of Queries Assume a large G and a small random subgraph G Approximate Q on G by Q on G : 1. Approximate an OLAP query Q (distribution): Prob Ω [ [Q G - Q G ] < ε ] > 1-δ 1. Approximate a graph property Q(): Prob Ω [ Q() G = Q () G ] > 1-δ 1. Approximate a graph search property Q(x). Prob Ω [ Q(x) G Q (x) G Φ ] > 1-δ

7 Approximate Q on G by Q on G : Refinement of Approximation Strict Approximation a graph property Q(): Prob Ω [ Q() G = Q () G ] > 1-δ Property Tester (ε,δ)-approximation: If Q is true on G then Q is true on G If Q is ε-far from G then : Prob Ω [ Q () G false ] > 1-δ Size of G only depends on (ε,δ) Note: Q is ε-far from G if dist(q,g)>ε

8 2. Streaming data Assume a stream of tuples of G: t 1, t 2, t n. Can t store all the tuples: keep a subset G of size N Tool: Reservoir sampling to approximate an OLAP query Q. Reservoir Sampling: keeps k edges with a uniform distribution with a weighted distribution Theorem 1: If N>O(log(1/ δ ).c/ε 2 ), then Prob Ω [ [Q G - Q G ] < ε ] > 1-δ G G

9 Streaming Graph edges Assume a stream of edges of G: e 1, e 2, e n. Can t store all the edges: keep a subgraph G and the «most recent» G t G t G G t Tools: Reservoir for G and Window Reservoir G t Store: all the nodes (Mysql) and only edges in the Reservoirs Complexity : Avoid storing O(n 2 ) edges

10 Examples of Queries Assume a stream of Twitter edges of G: e 1, e 2, e n. Can t store all the edges: keep a subgraph G Boolean queries: Q():- is there one community (dense subgraph)? Q():- are there k disjoint communities? Search queries: Q(x):- x is in a community (largest dense subgraph (V,E) s.t [E]>α. [V] 2 ) Analytic queries: Q :- the distribution of the sizes of the communities

11 Streaming Graph edges Assume a stream of edges of G: e 1, e 2, e n. Can t store all the edges: keep a subgraph G and the «most recent» G t G t G G t Search for a community Q(x) Algorithm: keep large connected components in G : Q (x) G Goal: Prob Ω [ Q(x) G Q (x) G Φ ] > 1-δ

12 Nodes Uniform Sampling: nodes vs. Edges Edges Edges witness the concentration. Equivalently, Nodes could be chosen with their degree distribution.

13 Random graphs Erdos-Renyi: G(n,p) Preferential Attachment: PA(n,m) Degree distribution: Power law: Prob Ω [ degree(x)=i ] =c/i 2 Example: [15,6,4,3,3,2] as histogram Concentration property: O( m/2) nodes of high degrees S concentrated if v in S then Majority (edges(v)) in S S is a community. We can build a graph with several communities of different sizes.

14 Graph with 2 communities of the same size

15 Uniform Sampling on the edges G λ G t G G t Algorithm: Keep large connected components of G t at times λ, 2λ,.. Theorem 2: If a graph G follows a power law and has «a concentration property», then Prob Ω [ Q(x) G Q (x) G Φ ] > 1-δ C 2,i C 1,i

16 Several streams: S 1, S 2, 3. Integration of Streaming data S j : we compress the stream to V, C j the union of the large connected components C j =Union C i,j All the nodes are stored in a Mysql database

17 2 streams

18 Integration of Streaming data How can we correlate two streams of edges? Store large connected components of the reservoir, at discrete times, for each stream. Let V C1, V C2 the nodes of the communities. Nodes correlation (t): (V 1 V 2 )/Max(V 1,V 2 ) Communities correlation (t): (V C1 V C2 )/Max(V C1,V C2 ) t Série 1 Série 2

19 Conclusion 1. Approximation of queries Boolean, search, analytic queries 2. Streaming Data Streaming tuples Streaming edges: dynamic community detection 3. Data Integration for streams Compress streams with «good representations» with h.p. Define the Integration from this compressed forms The «value» of the data increases with the integration