Case Study: Social Network Analysis. Part II

Transcription

1 Case Study: Social Network Analysis Part II

2 Outline IoT Fundamentals and IoT Stream Mining Algorithms Predictive Learning Descriptive Learning Frequent Pattern mining Evolving Analytics including Novel Patterns Case Study: Social Network Analysis Challenges in mining networked data, Online sampling Evolving centralities and communities Tracking the dynamics of evolving communities Case Study: Predictive Maintenance Problem Definition Change, Anomaly and Novelty Detection Failure Prediction and Detection Case Study: Secure IoT Stream Mining Types of attacks (e.g., Controlled Channel and Timing) Securing data and system logs Defense against side-channel attacks Data-Obliviousness Randomization 79

3 Challenges in Mining Networked Data Unbounded, evolving, high speed, massive Continuous interactions between social entities Represent real world social structures Mining social structures to make powerful decisions Unique challenges in mining streams

4 Complex Evolutionary Network : Calls' Network Topology Continuous interactions between users ( Edges and Nodes) Multiple interactions between two users ( Multi-Graph) Long/short interactions between two users ( Weighted Graph) Who initiates the Call ( Bi-directional graph) More users making less calls ( Obeys Power Law Distribution ) Number of connected components > 1 ( Disconnected Graph ) Vertices's with different degrees ( Irregular Graph ) Density of Graph 0 (Sparse Graph) Interactions based on place and time ( Spatio-Temporal ) Nodes and edges get added and deleted ( Evolutionary )

5 Practical Applications of Call Networks Analysis Behavioral prediction Churn prediction Load prediction Word of mouth viral marketing Influence analysis Customer profiling Event detection etc.

6 Problems Unable to hold entire data on disk High cost in batch processing, out-dated results Difficult to compute continuous queries over large data streams Difficult to calculate interesting measures like centrality, path length, eccentricity etc Difficult to gain useful insights in real time

7 Online Sampling 84

8 Sampling Massive Streaming Call Graphs Sampling: Selecting a subset of individuals from with in a stream of data to represent the characteristics of the whole stream at a given point of time.

9 Applications Approximate answers to real time queries Real time computation of measures like centrality, clustering coefficient, path length etc for identifying graph properties Finding frequent items in real time Real time detection of communities, events, ego networks and key players etc

10 Scenario Anonymous CDR's from Telecommunication networks Data Stream over 31 days Approx. 8 millions to 16 millions calls per day Spread across 24 hrs per day Call-Graph Semantics Nodes as callers and callees Edges as calls Multi-graph with repetitive edges (calls between same nodes) Multi-graph mapped to weighted network as frequency of edges Incoming and outgoing calls as bi-directional edges

11 Evaluating Sampling methods and Algorithms Which structures are well preserved by the samples over the evolution of a weighted directed graph stream? Which samples maintain the properties of dynamic stream? Which methods and techniques have least time complexity? Which samples are biased towards some of the metrics? Which samples exhibit similar degree distribution as snapshot of stream?

12 Methods Node based methods Sample a set of nodes from the original graph. The samples posses only nodes and no structure. Edge based methods Samples are generated by selecting a subset of edges from the original graph. The resultant graph is a subgraph of original graph with nodes and edges.

13 Algorithms Reservoir Sampling Space Saving Biased Random Sampling

14 Reservoir Sampling Randomly choosing a sample of k items from a list S containing n items. Replaces elements with gradually decreasing probability All the elements are chosen with same probability Fill the reservoir of size k with first n elements For each element i after n Generate a random number r between 0 and i 1 If r<k; Let j be the element at position r in k Replace element j with i Else Skip I (Vitter 1985)

15 Space Saving Algorithm Approximate approach for finding most frequent items Maintain partial information of interest; monitor only a subset m of elements For each element e in stream If e is monitored: Increment Count Else Let m be the element with least hits min Replace m with e with count = min+1 (Metwally et al. 2005)

16 Biased Random Sampling Generates a random sample by inserting every element from the stream with equal probability Replacing elements from sample randomly Biased towards recent elements in stream Fill the list of size k with first n elements For each element i after n Generate a random number r between 0 and k Let j be the element at position r in k Replace j with i (Aggarwal 2006)

17 SSE Sample of size 10⁴ edges over 31 days using Fruchterman Reingold Layout

18 RS RS displays Least community structure

19 BRS Sample of nodes with different modularities

20 Evolving centralities and communities 97

21 A Temporal Networks C B E D F t1 Person-to-person communication Disease spreading Social networks Temporal Networks Editors: Holme, Petter, Saramäki, Jari (Eds.)

22 Temporal Networks C A B E F D t2

23 Temporal Networks C A B E F D t3

24 Temporal Networks C A B D E F t1 A B C D E F t2 A B C E F D t3

25 Temporal Networks C C C A B E F A B E F A B E F D t1 D t2 D t3 C A is the source of F contamination? A B E F D aggregate

26 Temporal Networks C A B E F D A B B C D E B C D E E F

27 Temporal Networks C A B E F D Observation window W =

28 Temporal Networks [1,2] C [1,2] [17,19] A B E [7,9] D [8,9] F [2,4], [11,20]

29 Temporal Networks A [1,2] C [1,2] [17,19] B E [7,9] D [8,9] F [2,4], [11,20] Person-to-person communication Disease spreading Social networks

30 Temporal Networks A [1,2] C [1,2] [17,19] B E [7,9] D [8,9] F [2,4], [11,20] Person-to-person communication Disease spreading Social networks

31 Temporal Metrics The concept of geodesic distance cannot be limited to the number of hops separating two nodes (the topology of the network) but should also take into account the temporal ordering of links.

32 Temporal Path A [1,2] C [1,2] [17,19] B E [7,9] D [8,9] F [2,4], [11,20] W = [1,20] R = 1 day T = 0 Temporal Path Duration Is fastest path? P(B,E) = <(B,C,1), (C,E,2)> P(B,E) = <(B,D,7), (D,E,9)> P(B,E) = <(B,D,7), (D,E,8)> P(B,E) = <(B,C,1), (C,E,1)> 1 yes 2 no 1 Yes 0

33 Temporal Metrics Revisiting centrality metrics. Fastest path duration How close a node is from the others nodes in the graph High closeness = best visibility into what is happening

34 Temporal Metrics Revisiting centrality metrics. Number of fastest paths between j and k that pass through v Number of fastest paths between j and k High betweenness = great influence over what flows High betweenness = control the flow of information (gatekeeper)

35 Betweenness centrality A [1,2] C [1,2] [17,19] B E [7,9] D [8,9] F [2,4], [11,20] W = [1,20] R = 1 day T = 0 Node Fastest Path (temporal) Shortest Path (static) C D E 3 4 B 0 4

36 Betweenness centrality [1,2] C [1,2] R = 1 day A B [7,7] D E [2,4] F T = 0 Ranking Node 1 E, C 3 A, B, D, F t W = [1,7] W = [1,14] W = [1,21]

37 Betweenness centrality [1,2] C [1,2] R = 1 day A B E [7,9] D [8,9] F [2,4], [11,14] T = 0 Ranking Node Ranking Node 1 E, C 1 E 2 C 3 A, B, D, F 3 D A, B, F t W = [1,7] W = [1,14] W = [1,21]

38 Betweenness centrality A [1,2] C [1,2] [17,19] B E [7,9] D [8,9] F [2,4], [11,20] R = 1 day T = 0 Ranking Node Ranking Node Ranking Node 1 E, C 1 E 1 E 3 A, B, D, F 2 C 2 C 3 D 3 D A, B, F 4 A, B, F t W = [1,7] W = [1,14] W = [1,21]

39 Centrality Change C i (t): Centrality of node i at time t C i (t) = pos i (t-1) pos i (t) / max(pos i (t-1), pos i (t)) Ranking Node 1 E, C 3 A, B, D, F ranking position of node i in time t Ranking Node 1 E 2 C 3 D 4 A, B, F Ranking Node 1 E 2 C 3 D 4 A, B, F t

40 Tracking the dynamics of evolving communities 117

41 Social Networks as Dynamic Structures Social networks are a hot topic and a focus of considerable a2enjon in recent research Growing availability of large volumes of relajonal data, boosted by the proliferation of social media web sites Number of social entities and the interactions among social entities change over time social networks have a dynamic nature One way to uncover the evolution patterns of social networks is by monitoring the evolujon of their communijes

42 Event-based Dynamic Community Mining Dynamic communities are unstable pakerns that can evolve in both membership and content Dynamic communities undergo a succession of events during their life-cycle in the network Community Evolution Events according to Palla et al. (2007)

43 Event-based Dynamic Community Mining How can we perform the mapping of communities between different snapshots of the dynamic network? Proposed solujon: compute condijonal probabilijes for each pair of communijes found at consecujve jme points? MAESTRA - Learning from Massive, Incompletely annotated, and Structured Data

44 Tracking Clusters Survival threshold

45 Event-based Dynamic Community Mining MAESTRA - Learning from Massive, Incompletely annotated, and Structured Data

46 Communities Life-cycle and Temporal Trajectory How can we represent the evolution of the dynamic communities? Community Life-cycle: temporal sequence of events (birth, split, merge, death) undergone by a given dynamic community, from the moment the community first appeared unjl the moment it fade away. Figure from Greene et al. (2010)

47 Telco Data Community Life-cycle July August September October November December Using Louvain Algorithm for Community Detection

48 Interpretation of Communities Dynamics What is happening in the structure of the underlying network that explains these community dynamics? Tucker3 model

49 Node-level Measures Eigenvector centrality: measures how well a given node is connected to other wellconnected nodes in the network Closeness centrality: measure of reachability that quanjfies how fast can a given node can reach everyone in the network Betweenness centrality: measures the extent to which a node lies between other nodes in the network

50 Assigning a meaning to the axes of the 2D space Social Activity Sociability VS Accessibility

51 Interpretation of a Community s Temporal Trajectory