PROJECT PROPOSALS: COMMUNITY DETECTION AND ENTITY RESOLUTION. Donatella Firmani

Transcription

1 PROJECT PROPOSALS: COMMUNITY DETECTION AND ENTITY RESOLUTION Donatella Firmani

2 PROJECT 1: COMMUNITY DETECTION

3 What is Community Detection? What Social Network Analysis is? Network Analysis is the keyword For the 21 st Century Researchers, Politicians, People talk about Social Networks. Community detection: discovering groups in a network where individuals group memberships are not explicitly given

4 Subjectivity of Community Definition A denselyconnected community Each connected component is a community Definition of a community can be subjective.

5 Node-Centric Community Detection Node-Centric Community: Each node in a group satisfies certain properties Sample properties: Complete Mutuality cliques Reachability of members k-clique, k-clan, k-club Nodal degrees k-plex, k-core Relative frequency of Within-Outside Ties 5

6 Complete Mutuality: Cliques Clique: a maximum complete subgraph in which all nodes are adjacent to each other Nodes 5, 6, 7 and 8 form a clique NP-hard to find the maximum clique in a network Straightforward implementation to find cliques is very expensive in time complexity 6

7 Enumerating all Maximal Cliques [CDMPT16] W U A F S G J H D E Y A J H H F S D D E

8 Clique is Very Strict Clique is a very strict definition Vertices of a clique are at distance 1 each other Diameter of induced subgraph is 1 Min-degree of induced subgraph s-1 (clique size s) Normally use relaxations of cliques as definition for communities Clique relaxations include: k-clique: vertices with distance* no greater than k from each other k-club / k-clan: subgraphs of diameter no greater than k k-plex: subgraphs of min-degree no greater than s-k 1-clique=1-club=1-clan=1-plex=clique (*) distance is computed on the input graph and can contain external edges 8

9 Enumerating large k-plexes [CFMPT17]

10 Enumerating large k-plexes [CFMPT17]

11 Graph Databases Store data as nodes and relationships Database full of linked nodes

12 Sample Graph DB AllegroGraph Bitsy Cayley GraphBase Graphd HyperGraphDB IBM System G imgraph InfiniteGraph InfoGrid Neo4j Sparksee/DEX Trinity TurboGraph

13 Sample GraphDB queries Pattern matching query Nodes with first name James Adjacency query Nodes that James knows direcly I.e., are adjacent to James in the knows relationship Reachability query Nodes that James knows I.e., are reachable from James in the knows relationship Graph Analytical query

14 Single-sql-query for #connected components (for FUN)

15 Neo4j query for #connected components mponents/follows Via Mazerunner REST API Integrates Apache Spark, GraphX and Neo4j for big scale graph analysis GraphX: Apache Spark's API for graphs and graph-parallel computation

16 Performance

17 Summary and open problems Network Analysis is the keyword For the 21st Century Researchers, Politicians, People talk about Social Networks. Problems: Communities Analysis of Structure & Social Space Technologies: GraphDB Big Data technologies

18 PROJECT 2: ENTITY RESOLUTION

19 What is Entity Resolution (ER)? Input data: modeled as a graph. Graph node = data record. Graph edge label = probability that record pair represents the same entity. Output: a set of clusters, each of which corresponds to an entity. 2 nodes in a cluster iff records represent the same entity. Traditional problems [EIV07, GM12]. Pairwise match: what is the probability that two records match? Clustering: how to partition records into an unknown # of entities? Blocking: how to perform ER in sub-quadratic time? 19

20 What is ER Using an Oracle? Input data: modeled as a graph. Output: a set of clusters = entities. Formal problem [WL+13, VBD14, FSS16]: Given an oracle that can correctly answer if a record pair is a match, what is an optimal strategy to ask oracle queries so as to minimize the number of queries for resolving the entire graph? Motivation: reduce crowdsourcing ER cost for data set. 20

21 What is Online ER Using an Oracle? Formal problem [FSS16]: Given an oracle that can correctly answer if a record pair is a match, what is an optimal strategy to ask oracle queries so as to maximize progressive recall wrt the sequence of oracle queries? Progressive recall = area under recall vs query sequence curve. Motivation: limited resolution time, early user termination. 21

22 Example: DB of Handwritten Characters Data from the Vatican Secret Archives Registri Vaticani: Pope letters throughout the 13 th -century. Linkage problem: entities = characters. 22

23 Example: DB of Handwritten Characters?? 23

24 Strategy 1: Edge Ordering [WL+13] Optimal strategy needs to ask N K + (K choose 2) oracle queries. Takes advantage of (matching and non-matching) transitivity. EO: ask oracle queries in edge probability order. Can grow multiple clusters and sub-clusters in parallel. Worst-case approximation ratio of O(N) [VBD14]. 24

25 Strategy 2: Node Ordering [VBD14] Optimal strategy needs to ask N K + (K choose 2) oracle queries. Takes advantage of (matching and non-matching) transitivity. NO: process nodes in order of their expected cluster sizes. Ask oracle queries in edge probability order to processed nodes. Can grow similar-sized clusters (but not sub-clusters) in parallel. Worst-case approximation ratio of O(K) [VBD14]. 25

26 Oracle Strategy for Progressive Recall Edge ordering: use benefit metric instead of edge probability. Iteratively query oracle with (u, v) having highest value of b e (u, v). Initially, edge with highest value of p(u, v) is queried. Subsequently, can query lower probability, higher benefit edge. 26

27 Strategy 3: Hybrid Ordering [FSS16] Hybrid ordering: use node ordering, then edge ordering. Iteratively: select node u with highest value of b n (u), then query oracle with (u, v), v є C, in decreasing order of b n (u, C). Heuristic: use a threshold on benefit b n (u, C). Finally, process non-inferable edges (u, v) in order of b e (u, v). 27

28 Errors in Oracle Answers Input data: modeled as a graph. Output: a set of noisy clusters. Formal problem: Given an oracle that can answers if a record pair is a match with some error probability, what is an optimal strategy to ask oracle queries so as to minimize the number of queries for resolving the entire graph and maximizing precision? 28

29 Example: DB of Handwritten Characters?? 29

30 Errors and Graph Cuts Vertex cut: a partition of the nodes (vertices) of a graph into two disjoint subsets. Cut-set: the set of edges that have one endpoint in each subset of the partition. What would you trust more??? 30

31 Errors and Graph Cuts Vertex cut: a partition of the nodes (vertices) of a graph into two disjoint subsets. Cut-set: the set of edges that have one endpoint in each subset of the partition. Formal problem: Build graphs with large cuts with as less as edges as possible So-called expander graphs Technical contribution: Prove that the output graph consists of expanders 31

32 Strategy 4: Hybrid Ordering with Expanders Hybrid ordering with Expanders: use node ordering by assigning a node to a cluster only if more than K answers are positive, then edge ordering. 32

33 Summary and open problems Formal study of maximizing progressive recall in online ER. Problem is NP-complete. Formal study of maximizing progressive recall and precision in presence of errors in oracle answers. Open problems: Design robust, online strategies for errors in oracle answers. Design a more powerful interface for queries than pairwise. Scalability (e.g. blocking) 33