Kara Greenfield, William Campbell, Joel Acevedo-Aviles

Size: px

Start display at page:

Download "Kara Greenfield, William Campbell, Joel Acevedo-Aviles"

Claud Burns
5 years ago
Views:

Kara Greenfield, William Campbell, Joel Acevedo-Aviles GraphEx 2014 8/21/2014 This

1 Kara Greenfield, William Campbell, Joel Acevedo-Aviles GraphEx /21/2014 This work was sponsored by the Defense Advanced Research Projects Agency under Air Force Contract FA C Opinions, interpretations, conclusions, and recommendations are those of the authors and are not necessarily endorsed by the United States Government.

2 VizLinc Software suite that integrates automatic information extraction, search, graph analysis and geo-location for interactive visualization and exploration of a text collection Designed for unstructured text Composed of two desktop applications: Ingestion Tool : Java-based application to pre-process unstructured text documents Graphical User Interface: Gephi plugin that provides visualization, search, geo- Standalone Can operate completely offline No specialized hardware Open Source location, graph analytics User Interface: Ingestion tool VizLinc- 2

3 VizLinc: Goals Data Characterization Understanding the type of information the data set under study contains Making patterns and connections between entities evident Narrow down the corpus Ideally: to a small fraction that users can quickly read VizLinc- 3

4 System for Content + Context Analysis Text Entity Extraction Pedro worked near Rio Grande Social Networks Community Detection Documents Twitter Newswire Reddit Topics Love Religion Holidays Money Entity Coref Leadership Prediction U s e r Wikipedia Location Extraction/ Geocoding Entity Linking Efficient Search Database Single-Message Content Analytics Knowledge-Base Construction High-Level Analytics VizLinc- 4

5 System Overview Graph File (GraphML) Ingestion Tool GUI Unstructured Text (docx, pdf, ) Relational Database (H2) User Index (Lucene) VizLinc- 5

6 Ingestion Tool Information Extraction Text Documents.docx.pdf Text Extraction (Tika).txt Named Entity Extraction (Stanford NER).csv Graph DB Creation.neo4j Geocoding.neo4j Count Computation.neo4j Coreference Resolution.neo4j Conversion Metadata Generation GraphML File Relational Database (H2) Index (Lucene) VizLinc- 6

7 Database Implementation for VizLinc Tool Search Time Flexible Schema Storage Scalability Graph Ops SQL DB Y Y G Y Graph DB R G G Y In-Memory Data Structure G R R G Goals for Database storage and operations: Standalone operation no enterprise structure Responsive user queries Moderate size data set Wide variability in database performance on different tasks no silver bullet solution VizLinc- 7

8 Graph Database Schema type name path xml document doc123.xml /a/b/2009/doc123.xml <doc> </doc> type name path xml document doc234.xml /a/b/2009/doc234.xml <doc> </doc> DOCUMENT_TO_MENTION type mention etype PERSON text Alice position 345 type mention etype PERSON text Bob Brown position 552 type mention etype PERSON text Bob position 1002 type mention etype LOCATION text Lexington position 445 type mention etype PERSON text Bob position 678 VizLinc- 8

9 Coreference Resolution Doc 123 Doc 234 Alice Alice Bob Brown Bob Brown Lexington Bob Brown Bob Brown Lexington Bob Brown merged entity Bob 1. Find entities in a single document 2. Merge entities across documents VizLinc- 9

10 Social Network Construction Named entities (people) extracted from documents Co-occurrence within document Pooled across document filter by number of mentions, entity coref, etc. VizLinc- 10

11 Community Detection with Infomap Look at random walks on the graph label each node Key idea: Compress random walk with a twolevel code: VizLinc- 11

12 VizLinc- 12 Community Detection with Infomap

13 Leadership Prediction Various centrality metrics Page rank Betweenness Typically a negligible difference in results on operational data VizLinc- 13

14 VizLinc User Interface github.com/mitll/vizlinc Text Search Social Network (not displayed) Shows people mentioned and their links Document Content Highlights search terms/ entities extracted Entity Search by person, place and/or organization Social Network Analytics Map Shows all locations in working document set Document List VizLinc- 14

15 VizLinc on Your Data Both the VizLinc UI and the VizLinc Ingester are open source VizLinc- 15

16 VizLinc- 16 Questions

VizLinc: Integrating information extraction, search, graph analysis, and geo-location for the visual exploration of large data sets

VizLinc: Integrating information extraction, search, graph analysis, and geo-location for the visual exploration of large data sets Joel C. Acevedo-Aviles, William M. Campbell, Daniel C. Halbert, Kara