2 E6885 Network Science Lecture 4: Network Visualization

Transcription

1 E 6885 Topics in Signal Processing -- Network Science E6885 Network Science Lecture 4: Network Visualization Ching-Yung Lin, Dept. of Electrical Engineering, Columbia University October 3rd, 2011 Course Structure Class Date 09/12/11 09/19/11 09/26/11 10/03/11 10/10/11 10/17/11 10/24/11 10/31/11 11/14/11 11/21/11 11/28/11 12/05/11 12/12/11 12/19/11 Class Number Topics Covered Overview Social, Information, and Cognitive Network Analysis Network Representations and Characteristics Network Partitioning, Clustering, and Use Case Network Visualization, Sampling and Estimation Network Modeling Network Topology Inference Dynamic Networks Info Diffusion in Networks Final Project Proposal Presentation Dynamic Probabilistic Networks Privacy, Security, and Economy Issues in Networks Behavior Understanding and Cognitive Networks Large-Scale Network Processing System Final Project Presentation 2 E6885 Network Science Lecture 4: Network Visualization 1

2 Methods of Graph Visualization Drawing Graphs with Special Structure: Planar Graphs Orthogonal paths for edges K-sided convex polygons for each cycle of length k Trees Layering of Edges Horizontal and vertical edges (hv-layout) Radiate outward on concentric circles (radial layout) Drawing Graphs Using Analogies to Physical Systems 3 E6885 Network Science Lecture 4: Network Visualization Tips for Graph Drawing Minimize Edge Crossing Planar Graph and Planar Drawing Graph Planarity Test O(n) algorithm at best Intuitive method: test for subdivision of K5 or K3,3 Minimize and Uniform Edge Length Minimize edge length Minimize the total edge length Minimize the maximum of edge length Uniform edge length (for unweighted graph) Minimize the variance of edge length For weighted graph: edge length proportional to the weight Planar Drawings Non-Planar Drawings Uniform Edge Length Drawing 4 E6885 Network Science Lecture 4: Network Visualization 2

3 Tips for Graph Drawing Minimize and Uniform Edge Bends Minimize total edge bends Minimize the variance of edge bends Maximize Angular Resolution Maximize the smallest angle between two edges incident on the same node Graph with and without edge bend Graph with different angles (45 and 30 ) Minimize Aspect Ratio Adapt to the real-world screens Graph Symmetry Symmetrical and non-symmetrical graph 5 E6885 Network Science Lecture 4: Network Visualization Graph Drawings Graph drawing without modern techniques Hand-made images Drawing with cluster/factor analysis, multidimensional scaling Various modern drawing approaches Graph drawing approaches according to elaborately-defined aesthetics Each approach follows a list of aesthetics with precedence since: Aesthetics conflict with each other Hard to design algorithm to fit all the aesthetics Hand-made drawing of friendships 6 E6885 Network Science Lecture 4: Network Visualization 3

4 Examples of Tree Drawing Layering of Edges Horizontal and vertical edges (hvlayout) Radiate outward on concentric circles (radial layout) 7 E6885 Network Science Lecture 4: Network Visualization Hierarchical Graph Drawing Hierarchical Graph Drawing For dependency relationships abstracted as acyclic digraph Sugiyama-style 3-step layout approach Hierarchical drawing of a citation network 8 E6885 Network Science Lecture 4: Network Visualization 4

5 Tree Drawing Orthogonal Graph Drawing For circuit placement in VLSI and PCB board layout Topology-Shape-Metrics approach Orthogonal Drawing Radial Graph Drawing For rooted tree drawing Foldable for hierarchical graphs Radial Drawing 9 E6885 Network Science Lecture 4: Network Visualization Force-Directed Graph Drawing Force-directed Graph Drawing For undirected straight-line drawing Simulate physical systems with forces, the final graph layout reaches the local system equilibrium (i.e. a local minimization of energy) Generate graphs with good node uniformity and symmetric Two types of force-directed algorithms Spring Embedder by Eades and the seminar work by Fructerman and Reingold Kamada and Kawai algorithm, trying to maintain graph theoretic distances between nodes, the recent solution by stress majorization is popular Multi-Scale force-directed algorithms for large graphs Force-directed layout with spring embedder 10 E6885 Network Science Lecture 4: Network Visualization 5

6 Algorithm Assign a measure of force to each vertex. Iteratively adjust the position p (, ) T v = xv yv until the forces on each have converged. A standard setup defines the force F = ( F, ) T x Fy on v in the x direction to be: x x x x F v k p p len k (1) v u (2) v u x ( ) = uv ( u v uv ) + uv { u, v} E p 2 u pv ( u, v) V pu pv 3 where k (1) uv is a spring stiffness (2) k uv is s strength of electric repulsion stiffness is Euclidean distance is a parameter for desired dist. len uv 11 E6885 Network Science Lecture 4: Network Visualization Social Network Visualization -- Metaphors Typical Social Network Visualization with Graph Basic Metaphors with Graph Nodes ~ People Visual attribute mapping: hue, transparency, icon types, labels, etc. Edges ~ Relationships Visual attribute mapping: length, thickness, color, direction, labels, etc. Community Advanced Metaphors with Graph Shading ~ Community (cluster) Other Non-Graph Metaphors Adjacency Matrix 12 E6885 Network Science Lecture 4: Network Visualization 6

7 Social Network Visualization -- Matrix Adjacency Matrix as an Alternative of Node-Link Graph Good for huge, scale-free graph Hybrid visualization MatrixExplorer: synchronized view of both adjacency matrix and node-link graph NodeTrix: combined visualization for social network with communities User Studies on Performance Matrix is better than node-link graph in several major tasks when node number is over 20 Negatives: the mass people (information consumer) can only perceive simple visualization such as maps, line-charts and graphs 13 E6885 Network Science Lecture 4: Network Visualization Social Network Visualization Huge Structural Network Social Networks could be quite large, posing challenge to visualization Traditional layout algorithms do not scale, with O(n 2 )~O(n 3 ) computation complexity With multi-scale fast layout algorithm, huge graph can be drawn with degraded quality, but raises severe visual clutter Social Networks are intrinsically clustered, with well-known feature of scale-free, small-world Visualization should reveal this structural information A Huge Graph Visualization with Multi-Scale Layout Algorithm (by AT&T) 14 E6885 Network Science Lecture 4: Network Visualization 7

8 Social Network Visualization Huge Structural Network Graph Skeleton Visualization by Edge Pruning Filter edges by edge betweenness centrality from low to high Edge pruning with betweenness centrality 15 E6885 Network Science Lecture 4: Network Visualization Social Network Visualization Huge Structural Network Graph Skeleton Visualization by Edge Pruning Construct a minimal spanning tree (MST) or path-finder network from the original huge graph Graph skeleton visualization highlights the topology and intrinsic structure of huge graph by reducing visual clutter Skeleton extraction by constructing MST 16 E6885 Network Science Lecture 4: Network Visualization 8

9 Case study Mapping Science Fig 3.5 in Statistical Analysis of Network Data. Original source: Kevin Boyack. 17 E6885 Network Science Lecture 4: Network Visualization Case study Mapping Science Fig 3.6 in Statistical Analysis of Network Data. Original source: Kevin Boyack. 18 E6885 Network Science Lecture 4: Network Visualization 9

10 Social Network Visualization Huge Structural Network Graph Visual Clutter Reduction through Edge Bundling Edge bundling steps Generate control mesh Cluster the edges Let the clustered edge traverse through the same set of control points The effectiveness of edge bundling 19 E6885 Network Science Lecture 4: Network Visualization Social Network Visualization Huge Structural Network Graph Visual Clutter Reduction through Edge Bundling Hierarchical edge bundling For hierarchically clustered graph Maintain high-level topology information of edges Hierarchical Edge Bundling 20 E6885 Network Science Lecture 4: Network Visualization 10

11 Social Network Visualization Huge Structural Network Huge Graph Visualization through Advanced Interactions Focus+Context: to highlight part of the network while leaving context Topology Fisheye Hyperbolic Tree Focus + Context Topology Fisheye Hyperbolic Tree 21 E6885 Network Science Lecture 4: Network Visualization Social Network Visualization Huge Structural Network Huge Graph Visualization through Advanced Interactions Overview + Detail Overview graph with lowered quality Overview graph show hierarchy information Traditional Zoom & Pan Overview + Detail 22 E6885 Network Science Lecture 4: Network Visualization 11

12 Social Network Visualization Huge Structural Network Avoid huge graph with user-interest centric visualization Ego-centric network visualization Filter the huge graph by hop from the central node Allow expansion with interactions Radial graph layout of ego-centric network 23 E6885 Network Science Lecture 4: Network Visualization Social Network Visualization Huge Structural Network Avoid huge graph with user-interest centric visualization Spring-board by Frank Van Ham The search, show context, expand on Demand paradigm Degree of Interest (DOI) function combining graph topology (distance to the central node) and user interest (search context) Allow further navigation by expanding a subgraph Springboard: visualize huge graph by user-interest 24 E6885 Network Science Lecture 4: Network Visualization 12

13 Social Network Visualization Temporal Network Dynamic Network Visualization with Network Movie A Key Guideline: preserving user s mental map during the visualization Balance the static graph layout aesthetic and the graph stability between time frame Use Staged Animation to Help User Perceive Complex Changes Animation planning according to cognitive studies Intermediate layout result had better satisfy graph aesthetics rather than interpolations Some Negative Comments from User Studies Animation leads to many participant errors, though participants find it enjoyable Animation is the least effective form for analysis, even worse than static depictions Dynamic Network Visualization 25 E6885 Network Science Lecture 4: Network Visualization Social Network Visualization Temporal Network Static Depiction of Network Evolutions (Still with Stable Layout) 26 E6885 Network Science Lecture 4: Network Visualization 13

14 Example Wiki Revert Graph Understanding Social Dynamics in Wikipedia with Revert Graph Visualization Revert Graph: user as the node and revert relationship as the link Revert Graph Layout Opposite to the traditional spring embedder Links impose pushing forces and node attracts each other Revert Graph for a Wiki Page 27 E6885 Network Science Lecture 4: Network Visualization Example Wiki Revert Graph Understanding Social Dynamics in Wikipedia with Revert Graph Visualization This tool helps in discover: Formation of opinion group Pattern of mediation Fighting of vandalism Major controversial user and topic Administrators (in the bottom group) are fighting vandalisms (in the upper group) 28 E6885 Network Science Lecture 4: Network Visualization 14

15 CAIDA router-level Internet data K-core decomposition Fig 3.8 in Statistical Analysis of Network Data. Original source: Ignacio Alvarez-Hamelin 29 E6885 Network Science Lecture 4: Network Visualization Case Study SmallBlue Overview SmallBlue is an IBM social search engine that enables IBMers to locate knowledgeable colleagues, communities, and knowledge networks within IBM. IBMers use their social networks everyday to help them in their daily work, SmallBlue automates such practice. SmallBlue is made up of: A Client installed by users who want to contribute their social network data to SmallBlue A web-based tool suite - containing 4 tools: Ego, Find, Net, Reach Back-end processing / search engine Smallblue Infrastructure Smallblue Visualization 30 E6885 Network Science Lecture 4: Network Visualization 15

16 Case Study SmallBlue Net Hubs SmallBlue gauges the efficiency / health of enterprise network Network by business unit / geography Bridges people who span / bridge clusters Hubs most connected people Ability to identify (and correct) issues in collaboration, connectedness and knowledge flow across specific parts of the organisation Usage Scenario Includes Visualize community membership Succession planning implication of loosing people View by business unit Bridges 31 E6885 Network Science Lecture 4: Network Visualization Case Study SmallBlue Ego SmallBlue Ego helps IBMer understand and manage personal social network Distance from central = closeness Diversity of network = color coded for IBM business units Location of network = pictures overlay on map Shows relative social value of a person (how many people they could introduce you to) Shows changes and trends SmallBlue Ego Visualization 32 E6885 Network Science Lecture 4: Network Visualization 16

17 Acknowledgement Thanks to Dr. Lei Shi of IBM China Research Labs for providing several slides in this lecture. 33 E6885 Network Science Lecture 4: Network Visualization Network Sampling and Estimation 34 E6885 Network Science Lecture 4: Network Visualization 17

18 Why Network Sampling and Estimation is important? Frequently, only a portion of the nodes and edges is observed in a complex system. What is the outcome? Are the network characteristics measurements from a subgraph representing the whole network? What s the difference between principles of statistical sampling theory and sampling of graphs? 35 E6885 Network Science Lecture 4: Network Visualization Definitions Population graph: G = (V,E) Sampled graph: G* = (V*,E*) In principle, G* is a subgraph of G. Error may exist in assessing the existence of vertices or edges, through observations. Assume we are interested in a particular characteristic of G: η(g) For instance, η(g) is: Number of edges of G Average degree Distribution of vertex betweenness centrality scores Attributes of vertices such as the proportion of men with more female than male friends in a social network. Are we able to get a good estimate of η(g), say, ηˆ from G*? 36 E6885 Network Science Lecture 4: Network Visualization 18

19 Estimation based on sampled graph? How accurate if we directly use: * ˆ = ( G ) η η This implicity is used in many network study that asserts the properties of an observed network graph are indicative of those same properties for the graph of the network from which the data were sampled. Statistic sampling theories: means, standard deviations, quantiles, etc., are measurement of individual s properties. 37 E6885 Network Science Lecture 4: Network Visualization Examples Supposed that the characteristic of interest is the average degree of a graph G, η( G) = (1/ Nv ) d i V * Let the sample graph G* be based on the n vertices, V = i, 1, i * n observed degree sequence by d { i} * i V ˆ η= η( G ) = (1/ n) di * * i i V * { }, and denote its Begin with a simple random sample without replacement. Scenario 1: for each vertex * i V, we observe all edges { i, j} E * Scenario 2: for each vertex i V, we only observe all edges { i, j} E when i, j V * 38 E6885 Network Science Lecture 4: Network Visualization 19

20 Estimated average degree of the previous example 5,151 vertices and 31,201 edges. Original average degree Sample n =1,500. Scenario 1: (mean, std) = (12.117, ). Scenario 2: (mean, std) = (3.528, ). A typical adjustment of Scenario 2 is: d n d / N mean* = * i i v 39 E6885 Network Science Lecture 4: Network Visualization Choices of Sampling Designs Statistical Properties of Sampled Networks : Physical Review, Lee, Kim and Jeong, 2006 Effect of sampling on topology predictions of protein-protein interaction networks, Han et. al., Nature Biotechnology, In principle, this shall depend on: The topology of the graph G. The characteristics of η(g) The nature of the sampling design 40 E6885 Network Science Lecture 4: Network Visualization 20

21 Estimation for Totals Suppose we have: Ω= {1,, N u } Population :,and y i is the attribute value of interest. τ = µ = τ / Nu Let y i i and be the total and average values of the y s in the population. S = { i,, i n } Let 1 be a sample of n units from Ω In the canonical case in which S is chosen by drawing n units uniformly from Ω, with replacement, a nature estimate of µ is: y = (1/ n) yi i S ˆ τ = N y ( ) and u. These estimates are unbiased (i.e., E y = µ and E( τ ) = τ ). 41 E6885 Network Science Lecture 4: Network Visualization Estimation for Totals (cont d) The variances of these estimators take the forms: σ V y =σ 2 ( ) / 2 where is the variance of the values y in the full population Ω. n V N n 2 2 ( τ ) = uσ / In practice: Seldom simple random sample with replacement. Some units are more likely than others to be included. E.g.: Marketing Census Unequal Probability Sampling 42 E6885 Network Science Lecture 4: Network Visualization 21

22 Horvitz-Thompson Estimation for Totals The Horvitz-Thompson estimator through the use of weighted-averaging. i Ω Suppose that, under a given sampling design, each unit has probability i of being included in a sample of size n. π Let S be the set of distinct units in the sample. Then the Horvitz-Thompson estimate of the total τ takes the form: ˆ τ π = i S yi π i Let Z be a set of binary random variables, which is 1 if unit i is in S, and zero otherwise. Since, yi yi yi E( ˆ τ π ) = E( ) = E( Zi ) = E( Zi ) i S π i i Ωπ i i Ωπ i E( Z ) = P( Z = 1) = π ˆ µ = (1/ ) ˆ τ and i i i ˆπ Is an unbiased estimate of and π N u µ µ π 43 E6885 Network Science Lecture 4: Network Visualization Horvitz-Thompson Estimation for Totals (cont d) Variance of the estimator can be expressed as: π ij V ( ˆ τπ ) = yi y j ( 1) ππ i Ω j Ω i j Its estimation is an unbiased fashion by the quantity ˆ 1 1 V ( ˆ τ π ) = yi y j ( ) ππ π i Ω j Ω i j ij assuming π > 0 ij for all pairs i,j. 44 E6885 Network Science Lecture 4: Network Visualization 22

23 Simple Random Sampling Without Replacement Consider the case of sampling without replacement. Nu 1 n 1 n π i = = Nu N n u and π ij = n( n 1) N ( N 1) u u Then the Horvitz-Thompson estimates of the total and mean have the form: The variance may be shown to be ˆ τ = N y and ˆ µ = y π u N N nσ n π u ( u ) / Compare to τ = uσ while with replacement V ( ) N / n 45 E6885 Network Science Lecture 4: Network Visualization Probability Proportional to Size Sampling For instance, sampling household based on people. Sampling is done with replacement. If the probability is directly proportional to the value c i of some characteristics. π 1 (1 ) n i = p where i pi = ci / c i i The Horvitz-Thompson estimators are more appropriate than the sample mean. 46 E6885 Network Science Lecture 4: Network Visualization 23

24 Estimation of Group Size Many real cases, the size of the population is unknown. The capture-recapture estimators. The simplest version of capture-recapture involves two stages of simple random sampling without replacement, yielding two samples, say S 1 and S 2. Stage 1: the sample S 1 of size n 1 is taken. Mark all the units in S 1. All units are returned. Stage 2: Take another sample of size n 2. Then the estimation: n m ˆ ( c / r ) 2 where m is the number of intersection u = n1 N 47 E6885 Network Science Lecture 4: Network Visualization Common Network Graph Sampling Designs Procedures: Selection Stage: two inter-related sets of united being sampled. Observation Stage Induced and Incident Subgraph Sampling Star and Snowball Sampling Link Tracing 48 E6885 Network Science Lecture 4: Network Visualization 24

25 Induced Subgraph Sampling Random sample of vertices in a graph and observing their induced subgraph π = i n N V π i, j = n( n 1) N ( N 1) V V 49 E6885 Network Science Lecture 4: Network Visualization Incident Subgraph Sampling Uniform sampling based on edges Ne di 1 n, n Ne di π N i = e n 1, n> Ne di 50 E6885 Network Science Lecture 4: Network Visualization 25

26 Star and Snowball Sampling Nv L L + 1 n L Nv + L N i n π i, j = ( 1) π i, j Nv 2 n = 1 Nv n 51 E6885 Network Science Lecture 4: Network Visualization Link Tracing After selection of an initial sample, some subset of the edges from vertices in this sample are traced to additional vertices 52 E6885 Network Science Lecture 4: Network Visualization 26

27 Estimation of the number of edges. Example with different p by induced subgraph sampling 53 E6885 Network Science Lecture 4: Network Visualization Estimation of the number of edges. Example with different p by induced subgraph sampling (cont d) 54 E6885 Network Science Lecture 4: Network Visualization 27

28 Histograms of estimates of the number of triangles, connected triples and clustering coefficients. 55 E6885 Network Science Lecture 4: Network Visualization Questions? 56 E6885 Network Science Lecture 4: Network Visualization 28