Sub-Graph Detection Theory

Size: px

Start display at page:

Download "Sub-Graph Detection Theory"

Pierce Fox
5 years ago
Views:

1 Sub-Graph Detection Theory Jeremy Kepner, Nadya Bliss, and Eric Robinson This work is sponsored by the Department of Defense under Air Force Contract FA C Opinions, interpretations, conclusions, and recommendations are those of the author and are not necessarily endorsed by the United States Government. Slide-1

2 Outline Introduction Backgrounds and foregrounds Goals Detection Theory Sparse Matrix Duality Tree Finding Summary Slide-2

3 Goals Detection Theory Apply basic postulates of detection theory (signal, background, ) Quantitatively estimate difficulty of problem (SNR) Develop better detection algorithms Linear Algebraic Graph algorithms Additional tools for algorithm development Compact representation Parallel implementation well understood Slide-3

Detection Theory DETECTION OF SIGNAL IN NOISE THRESHOLD DETECTION OF SUBGRAPHS IN GRAPHS Example subgraph of interest: Fully connected (complete) SIGNAL NOISE SIGNAL H 0 H 1 NOISE N-D SPACE

4 Detection Theory DETECTION OF SIGNAL IN NOISE THRESHOLD DETECTION OF SUBGRAPHS IN GRAPHS Example subgraph of interest: Fully connected (complete) SIGNAL NOISE SIGNAL H 0 H 1 NOISE N-D SPACE ASSUMPTIONS Background (noise) statistics Foreground (signal) statistics Foreground/background separation Model reality Example background model: Powerlaw graph Goal: Develop basic detection theory for finding subgraphs of interest in large background graphs Slide-4

5 Graphs as Matrices A T x A T x Graphs can be represented as a sparse matrices Multiply by adjacency matrix step to neighbor vertices Work-efficient implementation from sparse data structures Most algorithms reduce to products on semi-rings: C = A +. x B x : associative, distributes over + Slide-5 + : associative, commutative Examples: +.* min.+ or.and

6 Algorithm Comparison Algorithm (Problem) Canonical Complexity Array-Based Complexity Critical Path (for array) Bellman-Ford (SSSP) Θ(mn) Θ(mn) Θ(n) Generalized B-F (APSP) NA Θ(n 3 log n) Θ(log n) Floyd-Warshall (APSP) Θ(n 3 ) Θ(n 3 ) Θ(n) Prim (MST) Θ(m+n log n) Θ(n 2 ) Θ(n) Borůvka (MST) Θ(m log n) Θ(m log n) Θ(log 2 n) Edmonds-Karp (Max Flow) Θ(m 2 n) Θ(m 2 n) Θ(mn) Push-Relabel (Max Flow) Θ(mn 2 ) O(mn 2 )? (or Θ(n 3 )) Greedy MIS (MIS) Θ(m+n log n) Θ(mn+n 2 ) Θ(n) Luby (MIS) Θ(m+n log n) Θ(m log n) Θ(log n) Majority of selected algorithms can be represented with array-based constructs with equivalent complexity. Slide-6 (n = V and m = E.)

7 Adjacency Matrix Types: Distributed Array Mapping RANDOM TOROIDAL POWER LAW (PL) PL SCRAMBLED Distributions: 1D BLOCK 2D BLOCK 2D CYCLIC ANTI-DIAGONAL EVOLVED Slide-7 Sparse Matrix duality provides a natural way of exploiting distributed data distributions

(MIT-LL & Brown) Kegelmeyer (Sandia) Kepner (MIT-LL) Kleinberg (Cornell) Kolda (Sandia) Leskovec (CMU) Madduri (Ga Tech)

8 Reference Book: Graph Algorithms in the Language of Linear Algebra Editors: Kepner (MIT-LL) and Gilbert (UCSB) Contributors Bader (Ga Tech) Chakrabart (CMU) Dunlavy (Sandia) Faloutsos (CMU) Fineman (MIT-LL & MIT) Gilbert (UCSB) Kahn (MIT-LL & Brown) Kegelmeyer (Sandia) Kepner (MIT-LL) Kleinberg (Cornell) Kolda (Sandia) Leskovec (CMU) Madduri (Ga Tech) Robinson (MIT-LL & NEU), Shah (UCSB) Graph Algorithms in the Language of Linear Algebra Jeremy Kepner and John Gilbert (editors) Slide-8

9 Outline Introduction Background and foregrounds Tree Finding Random Power Law Clique Source/Sink Tree Summary Slide-9

10 Background: Random (Erdos-Renyi) Graph N Adjacency Matrix N A M = s N 2 Slide-10 A : B N N A(i,j) = (r < s) r [0,1] Algebraic Form number of nodes s N node degree Degree Distribution

11 Background: Power Law (Kronecker) Graph N Adjacency Matrix = N A M ~ 8 N G : R n n A G k = G k-1 G Slide-11 M Algebraic Form number of nodes node degree Degree Distribution

12 Graph Foreground: Dense Subgraph n Adjacency Matrix n A m = s n 2 Slide-12 A : B n n A(i,j) = (r < s) r [0,1] Algebraic Form number of nodes s n node degree Degree Distribution

13 Foreground: Source Sink Graph k kn+2 Adjacency Matrix n kn+2 A m = (k+1)n A= = 1 Algebraic Form Slide-13 k k k k n n nn 1 k k n 11 n number of nodes kn 2 2 n node degree Degree Distribution

14 Foreground: Trees Graph k n k Adjacency Matrix n n k A m = n k A= n 1 n n k -1 1 ( 1 1) 1 n number of nodes (n-1)n k n n k-1-1 n+1 node degree Algebraic Form Slide-14 Degree Distribution

15 Background/Foreground Combinations Dense Subgraph Foregrounds Source/Sink Tree Backgrounds Random Power Law X Many interesting background/foreground combinations Rest of talk will focus on power law/tree Slide-15

16 Outline Introduction Background and foregrounds Tree Finding Summary Embedding Cued vs Uncued Set-Vector Representation Algorithm Results Slide-16

17 Tree Embedding A(T,T) = A T T A T T A Tree Foreground N T vertices, M T edges Power Law Background N vertices, M edges Assignment of A T T to a random set of vertices T in A embeds Tree in background Detection problem: find T given A Assume N >> N T and M >> M T Slide-17

18 Cued vs. Uncued Detection Uncued detection No information about T is provided Signal-to-noise ratio ~ N T /N Extremely difficult Cued detection T is divided into two sets V (given) and U* (unknown) More tractable V U* A VxV A U*xV A VxU* A U*xU* Slide-18

Set-Vector Dual Representation V U* Set Representation 1 3 6 8 9 11 13 14 2 4 5 7 10 12 15 16 1 3 6 8 9 11 13 14 2 4 5 7 10 12 15 16 A VxV A U*xV A VxU* A U*xU* Vector

N element vector where v(v) = 1, allows multiple adjacency matrix representations A V V = A(V,V) or A v v = I v A I v Set representation better for visualization V

19 Set-Vector Dual Representation V U* Set Representation A VxV A U*xV A VxU* A U*xU* Vector Representation A vxv A vxu* A u*xv A u*xu* v = ( ) u*= ( ) Set of vertices V can also be represented as an N element vector where v(v) = 1, allows multiple adjacency matrix representations A V V = A(V,V) or A v v = I v A I v Set representation better for visualization V contains only elements of interest Vector better for algorithm development and implementation v allows linear algebraic transformations and preserves graph context Slide-19

20 Tree Finding Algorithm Summary Step 0: Find all vertices that are 1st neighbors of v A u0 v = A I v - A v v d u0 v = A u 0 v + A v u 0 u 0 = d u0 v > 0 Step 1a: Eliminate vertices that create too many connections to v Step 1b: Eliminate vertices that connect to v that are filled Step 2: Find all vertices that are 1st neighbors of v that satisfy 1a & 1b Step 3: Select highest probability vertices based on (edges available) / (number candidates) Step 4: Select vertices with multiple connections into v Slide-20

21 Signal-to-Noise Estimate Background power law: N = 2 20 Foreground binary tree N T = 2 7, f = 0.5 (fraction known) Baseline SNR ~ 2-14 ~ st and 2nd neighbors SNR ~ 5/2 12 ~ st neighbors SNR ~ 7/2 8 ~ 0.03 Step 0 Multiply attached neighbors SNR ~ 2 4 ~ 16 Step 4 Slide-21

estimates Can find most of tree with high PFA Can find subset of tree with very

22 Probability of Detection (PD) vs Probability of False Alarm (PFA) 1st & 2nd neighbors probability of detection 1st neighbors PD vs PFA consistent with SNR estimates Can find most of tree with high PFA Can find subset of tree with very low PFA multiply attached high probability 1st neighbors Slide-22 probability of false alarm

23 Summary Detection Theory Apply basic postulates of detection theory (signal, background, ) Quantitatively estimate difficulty of problem (SNR) Develop better detection algorithms Linear Algebraic Graph algorithms Additional tools for algorithm development Compact representation Parallel implementation well understood Slide-23

global_index agg SendMsg/RecvMsg Jeremy Kepner Parallel MATLAB for Multicore and Multinode Computers 1 2 3 4 Distributed arrays have been recognized as

24 Parallel Matlab Book Universal Parallel Matlab programming Amap = map([np 1],{},0:Np-1); Bmap = map([1 Np],{},0:Np-1); A = rand(m,n,amap); B = zeros(m,n,bmap); B(:,:) = fft(a); pmatlab runs in all parallel Matlab environments Only a few functions are needed Np Pid map local put_local global_index agg SendMsg/RecvMsg Jeremy Kepner Parallel MATLAB for Multicore and Multinode Computers Distributed arrays have been recognized as the easiest way to program a parallel computers since the 1970s Only a small number of distributed array functions are necessary to write nearly all parallel programs Slide-24

Analysis and Mapping of Sparse Matrix Computations

Analysis and Mapping of Sparse Matrix Computations Nadya Bliss & Sanjeev Mohindra Varun Aggarwal & Una-May O Reilly MIT Computer Science and AI Laboratory September 19th, 2007 HPEC2007-1 This work is sponsored