Sampling Large Graphs for Anticipatory Analysis

Size: px

Start display at page:

Download "Sampling Large Graphs for Anticipatory Analysis"

Norah Mosley
6 years ago
Views:

1 Sampling Large Graphs for Anticipatory Analysis Lauren Edwards*, Luke Johnson, Maja Milosavljevic, Vijay Gadepally, Benjamin A. Miller IEEE High Performance Extreme Computing Conference September 16, 2015 This work is sponsored by the Intelligence Advanced Research Projects Activity (IARPA) under Air Force Contract #FA C Opinions, interpretations, conclusions and recommendations are those of the author and are not necessarily endorsed by the United States Government.

2 Outline Introduction Sampling Techniques Link Prediction Experimental Setup Results Summary

3 Common Big Data Challenge Operators Analysts Commanders Users Rapidly increasing - Data volume - Data velocity - Data variety - Data veracity (security) Data Gap Users & Beyond Data <html> OSINT Weather HUMINT C2 Ground Maritime Air Space Cyber

4 What is a Graph? Graphs are mathematical representations of physical or logical relationships between sets of objects A graph is a pair of sets: A set of vertices/ nodes (entities) A set of edges/links (connections, transactions, and relationships)

Applications of Graph Analytics NSA-RD-2013-05 Brain scale.

08 mna bytes 2 (molar bytes) adjacency Biomatrix 2.84 PB adjacency list 2.

, Frontiers in Neuroinformatics 5(3), 2011 Graphs represent entities and

between individuals or documents Graphs represent Graphs represent 2 N = 6.

An NSA Big Graph experiment patterns of computers brain regions on a network

interactions ~1M 1B network events GOAL: Identify anomalous patterns GOAL:

5 Applications of Graph Analytics NSA-RD Brain scale... ISR Social 100 billion vertices, 100 trillion edges Cyber2.08 mna bytes 2 (molar bytes) adjacency Biomatrix 2.84 PB adjacency list 2.84 PB edge list Human connectome. Gerhard et al., Frontiers in Neuroinformatics 5(3), 2011 Graphs represent entities and relationships detected through multiple data Graphs represent relationships between individuals or documents Graphs represent Graphs represent 2 N = mol communication connectivity between Paul Burkhardt, Chris Waring An NSA Big Graph experiment patterns of computers brain regions on a network ~1B 1T regions and ~1K 1M entities and connections ~10K 10M individuals and interactions ~1M 1B network events GOAL: Identify anomalous patterns GOAL: Identify hidden social networks GOAL: Detect attack or malicious software A 23 1 connections GOAL: Detect regions affected by a neurological condition

Big Data Challenge NSA-RD-2013-056001v1 Brain scale... 100 billion vertices, 100 trillion edges Memory required for storing Sparse Matrix (GB) 2,097,152 2.

.. 262,144 50 billion vertices, 1 trillion edges 271 EB adjacency matrix 29.5 TB adjacency list 29.1 TB edge list 32,768 NSA-RD-2013-056001v1 Social scale.

022 1023 mol 1 Paul Burkhardt, Chris Waring An NSA Big Graph experiment 1 billion vertices, 100 billion edges 111 PB adjacency matrix 4,096 2.

opte.org/maps) Paul Burkhardt, Chris Waring 64 An NSA Big Graph experiment Web Social Twitter graph from Gephi dataset (http://www.gephi.

6 Big Data Challenge NSA-RD v1 Brain scale billion vertices, 100 trillion edges Memory required for storing Sparse Matrix (GB) 2,097, mna bytes 2 (molar bytes) adjacency matrix 2.84 PB adjacency list Petabyte 2.84 PB edge list NSA-RD v1 Web scale , billion vertices, 1 trillion edges 271 EB adjacency matrix 29.5 TB adjacency list 29.1 TB edge list 32,768 NSA-RD v1 Social scale... Brain Human connectome. Gerhard et al., Frontiers in Neuroinformatics 5(3), NA = mol 1 Paul Burkhardt, Chris Waring An NSA Big Graph experiment 1 billion vertices, 100 billion edges 111 PB adjacency matrix 4, TB adjacency list Terabyte 512 Web graph from the SNAP database ( TB edge list Internet graph from the Opte Project ( Paul Burkhardt, Chris Waring 64 An NSA Big Graph experiment Web Social Twitter graph from Gephi dataset ( Cyber 8 Paul Burkhardt, Chris Waring An NSA Big Graph experiment P. Burkhardt and C. Waring, An NSA Big Graph experiment, National Security Agency, Tech. Rep. NSA-RD v1, ISR ,096 65,536 1,048,576 NSA Big Graph Experiment Graph Vertices (millions) How can these massive graphs be leveraged to provide accurate anticipatory intelligence?

7 Outline Introduction Sampling Techniques Link Prediction Experimental Setup Results Summary

8 Methods for Sampling Graphs Edge sampling methods Consider each edges individually, keep it in the sample based on some criterion Vertex sampling methods Sample individual vertices or local substructures Random walk methods Walk from vertex to vertex by some criterion, keep edges along the path Snowball sampling methods Start with a set of seed vertices, grow the sample from those

9 Random Edge Sampling Each edge in the graph has equal probability of being included in the sample Methods Used Popularity-Based Sampling Probability of sampling a given edge is proportional to a decreasing function of the vertex degrees Wedge Sampling Choose a vertex at random, and randomly select two of its adjacent edges to be included in the sample

Random Edge Sampling Each edge in the graph has equal probability of being included in the sample Methods Used Popularity-Based Sampling Probability of sampling a given edge is

10 Random Edge Sampling Each edge in the graph has equal probability of being included in the sample Methods Used Popularity-Based Sampling Probability of sampling a given edge is proportional to a decreasing function of the vertex degrees Wedge Sampling Choose a vertex at random, and randomly select two of its adjacent edges to be included in the sample

11 Random Edge Sampling Each edge in the graph has equal probability of being included in the sample Methods Used Popularity-Based Sampling Probability of sampling a given edge is proportional to a decreasing function of the vertex degrees Wedge Sampling Choose a vertex at random, and randomly select two of its adjacent edges to be included in the sample

12 Random Edge Sampling Each edge in the graph has equal probability of being included in the sample Methods Used Popularity-Based Sampling Probability of sampling a given edge is proportional to a decreasing function of the vertex degrees Wedge Sampling Choose a vertex at random, and randomly select two of its adjacent edges to be included in the sample

13 Random Edge Sampling Each edge in the graph has equal probability of being included in the sample Methods Used Popularity-Based Sampling Probability of sampling a given edge is proportional to a decreasing function of the vertex degrees Wedge Sampling Choose a vertex at random, and randomly select two of its adjacent edges to be included in the sample

14 Random Edge Sampling Each edge in the graph has equal probability of being included in the sample Methods Used Popularity-Based Sampling Probability of sampling a given edge is proportional to a decreasing function of the vertex degrees Wedge Sampling Choose a vertex at random, and randomly select two of its adjacent edges to be included in the sample

15 Random Edge Sampling Each edge in the graph has equal probability of being included in the sample Methods Used Popularity-Based Sampling Probability of sampling a given edge is proportional to a decreasing function of the vertex degrees Wedge Sampling Choose a vertex at random, and randomly select two of its adjacent edges to be included in the sample

16 Random Edge Sampling Each edge in the graph has equal probability of being included in the sample Methods Used Popularity-Based Sampling Probability of sampling a given edge is proportional to a decreasing function of the vertex degrees Wedge Sampling Choose a vertex at random, and randomly select two of its adjacent edges to be included in the sample

17 Random Walk Sampling Move along edges from vertex to vertex, adding edges to the sample along the way At each step, with probability 0.15, jump to any vertex in the graph Random Area Sampling Choose a set of seed vertices, and add all edges adjacent to those vertices to the sample Methods Used, Cont d

18 Random Walk Sampling Move along edges from vertex to vertex, adding edges to the sample along the way At each step, with probability 0.15, jump to any vertex in the graph Random Area Sampling Choose a set of seed vertices, and add all edges adjacent to those vertices to the sample Methods Used, Cont d

19 Random Walk Sampling Move along edges from vertex to vertex, adding edges to the sample along the way At each step, with probability 0.15, jump to any vertex in the graph Random Area Sampling Choose a set of seed vertices, and add all edges adjacent to those vertices to the sample Methods Used, Cont d

20 Random Walk Sampling Move along edges from vertex to vertex, adding edges to the sample along the way At each step, with probability 0.15, jump to any vertex in the graph Random Area Sampling Choose a set of seed vertices, and add all edges adjacent to those vertices to the sample Methods Used, Cont d

21 Random Walk Sampling Move along edges from vertex to vertex, adding edges to the sample along the way At each step, with probability 0.15, jump to any vertex in the graph Random Area Sampling Choose a set of seed vertices, and add all edges adjacent to those vertices to the sample Methods Used, Cont d

22 Random Walk Sampling Move along edges from vertex to vertex, adding edges to the sample along the way At each step, with probability 0.15, jump to any vertex in the graph Random Area Sampling Choose a set of seed vertices, and add all edges adjacent to those vertices to the sample Methods Used, Cont d

23 Random Walk Sampling Move along edges from vertex to vertex, adding edges to the sample along the way At each step, with probability 0.15, jump to any vertex in the graph Random Area Sampling Choose a set of seed vertices, and add all edges adjacent to those vertices to the sample Methods Used, Cont d

24 Random Walk Sampling Move along edges from vertex to vertex, adding edges to the sample along the way At each step, with probability 0.15, jump to any vertex in the graph Random Area Sampling Choose a set of seed vertices, and add all edges adjacent to those vertices to the sample Methods Used, Cont d

25 Random Walk Sampling Move along edges from vertex to vertex, adding edges to the sample along the way At each step, with probability 0.15, jump to any vertex in the graph Random Area Sampling Choose a set of seed vertices, and add all edges adjacent to those vertices to the sample Methods Used, Cont d

26 Random Walk Sampling Move along edges from vertex to vertex, adding edges to the sample along the way At each step, with probability 0.15, jump to any vertex in the graph Random Area Sampling Choose a set of seed vertices, and add all edges adjacent to those vertices to the sample Methods Used, Cont d

27 Random Walk Sampling Move along edges from vertex to vertex, adding edges to the sample along the way At each step, with probability 0.15, jump to any vertex in the graph Random Area Sampling Choose a set of seed vertices, and add all edges adjacent to those vertices to the sample Methods Used, Cont d

28 Random Walk Sampling Move along edges from vertex to vertex, adding edges to the sample along the way At each step, with probability 0.15, jump to any vertex in the graph Random Area Sampling Choose a set of seed vertices, and add all edges adjacent to those vertices to the sample Methods Used, Cont d

29 Methods Used, Cont d Random Walk Sampling Move along edges from vertex to vertex, adding edges to the sample along the way At each step, with probability 0.15, jump to any vertex in the graph Random Area Sampling Choose a set of seed vertices, and add all edges adjacent to those vertices to the sample Selected methods comprise a broad array of network sampling techniques

30 Outline Introduction Sampling Techniques Link Prediction Experimental Setup Results Summary

31 Link Prediction Scenario: Two vertices currently do not share an edge Question: How likely are these vertices to form a connection in the near future?? Examples: How likely is a connection to be formed between two people in a social network? How likely are two political entities to engage in diplomatic or military conflict?? Link prediction: Anticipation of new connections that currently do not exist

approximation Use a low-rank approximation for the square of the adjacency matrix 2 U Σ 2 U T Tensor

32 Link Prediction Statistics Common neighbors Vertices with more neighbors in common are more likely to connect Jaccard s coefficient Normalize common neighbors by total number of neighbors Low-rank approximation Use a low-rank approximation for the square of the adjacency matrix 2 U Σ 2 U T Tensor decomposition Break data into vertex-vertextime factors, extrapolate along temporal axis vertex vertex time time

33 How Does Sampling Hurt? If edge sampling probability is p, probability of maintaining a common neighbor is p 2 This smears the distribution of common neighbors, inhibiting detection Sample 1/8 of Edges For eigenvector-based techniques, removing edges causes the distribution of eigenvalues to contract The outlier eigenvalues, which provide the most information, contract more quickly eigenvalue

34 Outline Introduction Sampling Techniques Link Prediction Experimental Setup Results Summary

Experiments aggregate input graph sampling sampling sampling sampling

prediction statistics prediction statistics evaluation Simulated graph Method

which data size should be reduced Link Prediction Common neighbors Jaccard s

35 Experiments aggregate input graph sampling sampling sampling sampling sampling prediction statistics prediction statistics prediction statistics prediction statistics prediction statistics evaluation Simulated graph Method Random edge Popularity-based Random walk Random area Wedge Amount Factor by which data size should be reduced Link Prediction Common neighbors Jaccard s coefficient Low-rank approximation Tensor decomposition Prediction performance Running time

36 Simulation Setup Graph at time t Normalized common neighbors Random changes Graph at time t+1 BTER Model: Simulate graphs evolving over several time steps First time step: Create a graph from a generative model Future time steps: Compute common neighbors Normalize using cosine similarity Randomly select based on scores Create new random edges from generative model Simulates chance encounters Randomly remove edges

37 Simulated Graphs Approximately 10,000 vertices, average degree of 17, 5% increase in edge count per time step Two degree distributions: one lighter tailed and one heavier tailed Variations designed to determine most robust sampling methods

Probability of Detection ROC shaded area = 0.144 AUC = 0.

38 Performance Metrics Area under the ROC curve (AUC) Measure area under the receiver operating characteristic (ROC) curve, which maps false alarm rate to detection rate Running time Time required (in Matlab implementation) to compute prediction statistics Probability of Detection ROC shaded area = AUC = Probability of False Alarm Memory Footprint Storage required for prediction statistics Fundamental metrics for anticipatory network analytics: Predictive performance Latency Computing resource use

39 Outline Introduction Sampling Techniques Link Prediction Experimental Setup Results Summary

Link Prediction in Baseline Simulation Common Neighbors Low-Rank Approximation Jaccard s Coefficient Tensor Decomposition Common neighbors and Jaccard s coefficient are virtually

40 Link Prediction in Baseline Simulation Common Neighbors Low-Rank Approximation Jaccard s Coefficient Tensor Decomposition Common neighbors and Jaccard s coefficient are virtually identical Random area sampling has the most significant initial performance drop in all cases Random walk sampling improves AUC at a higher sampling factor with tensor factorization

41 Time for Link Prediction Common Neighbors Jaccard s Coefficient Common neighbors and Jaccard: time is substantially reduced as sampling becomes more aggressive Best initial decrease is with random area sampling However, this comes with the biggest decrease in predictive performance

42 Time for Link Prediction Low-Rank Common Approximation Neighbors Tensor Jaccard s Decomposition Coefficient Low Rank: time increases with higher sampling rates Changes in the distribution of eigenvalues alters convergence rate Tensor: High sampling rates significantly reduces runtime Random area sampling produces high number of isolated vertices, reduces the dimensionality of the adjacency matrix

43 Outline Introduction Sampling Techniques Link Prediction Experimental Setup Results Summary

44 Summary Network analytics are of tremendous importance for anticipatory analytics Forecast future connections Networks of interest are often extremely large, and data sampling enables computation on a dataset of a reasonable size Study considered the effect of sampling on anticipatory network analytics Random edge sampling is typically the most stable Random area sampling loses performance the fastest Promising areas for future work: test on larger simulated and real graph data, investigate emergent community detection, aggregate samples for higher predictive capabilities, integration with high-performance graph analytics hardware

45 Backup

Sampling Operations on Big Data

Sampling Operations on Big Data Vijay Gadepally, Taylor Herr, Luke Johnson, Lauren Milechin, Maja Milosavljevic, Benjamin A. Miller Lincoln Laboratory Massachusetts Institute of Technology Lexington, MA