CIS 4930/6930 Spring 2014 Introduction to Data Science /Data Intensive Computing. University of Florida, CISE Department Prof.

Transcription

1 CIS 4930/6930 Spring 2014 Introduction to Data Science /Data Intensive Computing University of Florida, CISE Department Prof. Daisy Zhe Wang

2 Text To Knowledge IR and Boolean Search Text to Knowledge (IE) Bayesian and Markov Models HMMs and CRFs Adapted Slides from Joseph Hellerstein from UC Berkeley, Raymond J. Mooney University of Texas at Austin

3 Text contains Information and Knowledge 3

4 Outline Text Retrieval Systems Elementary IR Scalable Boolean Text Search Knowledge construction from Text Text to Knowledge Probabilistic Graphical Models for Knowledge Extraction 4

5 Information Retrieval: History A research field traditionally separate from Databases Hans P. Luhn, IBM, 1959: Keyword in Context (KWIC) G. Salton at Cornell in the 60 s/70 s: SMART Around the same time as relational DB revolution Tons of research since then Especially in the web era Products traditionally separate Originally, document management systems for libraries, government, law, etc. Gained prominence in recent years due to web search Also used for non-web document management. ( Enterprise search ).

6 Today: Simple (naïve!) IR Boolean Search on keywords Goal: Show that you already have the tools to build an IR system based on relational DBs We ll skip: Text-oriented storage formats Parallelism Critical for modern relational DBs too Various bells and whistles (lots of little ones!) Engineering the specifics of (written) human language E.g. dealing with tense and plurals (stemmer, lemmatizer) E.g. identifying synonyms and related words (wordnet) E.g. disambiguating multiple meanings of a word (dictionary) E.g. clustering output

7 IR vs. DBMS Seem like very different beasts IR DBMS Imprecise Semantics Keyword search Unstructured data format Read-Mostly. Add docs occasionally Page through top k results Precise Semantics SQL Structured data Expect reasonable number of updates Generate full answer Under the hood, not as different as they might seem But in practice, you have to choose between the 2 most of the time Systems that support both IR and SQL GPText (Greenplum + Solr)

8 IR s Bag of Words Model Typical IR data model: Each document is just a bag of words ( terms ) Unigram, bigram, concepts High dimensionality, large feature space Detail 1: Stop Words Certain words are not helpful and increase the dimensionality, so not placed in the bag e.g. real words like the a for e.g. HTML tags like <H1> Detail 2: Stemming (word) Lemmatizing (context) Using language-specific rules, convert words to basic form e.g. surfing, surfed surf, cats cat e.g., meeting (v.) meet, better good Unfortunately have to do this for each language

9 Boolean Text Search Find all documents that match a Boolean containment expression: Windows AND ( Glass OR Door ) AND NOT Microsoft Note: query terms are also filtered via stemming and stop words When web search engines say 10,000 documents found, that is based on the Boolean search result size + keyword-based ranking + page rank (life of a query)

10 A Simple Relational Text Index Given: a corpus of text files Files(docID string, content string) Create and populate a bag of words table InvertedFile(term string, docid string) Build a B+-tree or Hash index on InvertedFile.term This is often called an inverted file or inverted index (e.g., lucene, solr) Maps from words -> docs, rather than docs -> words Given this, you can now do single-word text search queries!

11 An Inverted File Term docid Snippets from: Old class web page Old microsoft.com home page Search for databases microsoft do document document microsoft microsoft midnight midterm minibase million Monday more most ms msn must necessary need network term data database date day dbms decision demonstrate description design desire developer differ disability discussion division do document docurl

12 Handling Boolean Logic How to do term1 OR term2? Union of two docid sets! How to do term1 AND term2? Intersection of two docid sets! If docidset per key sorted (e.g., via B+tree index) Can be done via merge of docids How to do term1 AND NOT term2? Set subtraction Also easy because sorted (basically merge logic again) How to do term1 OR NOT term2 Union of term1 with NOT term2. Not term2 = all docs not containing term2. Usually not allowed!

13 Boolean Search in SQL Windows AND ( Glass OR Door ) AND NOT Microsoft (SELECT docid FROM InvertedFile WHERE word = window INTERSECT SELECT docid FROM InvertedFile WHERE word = glass OR word = door ) EXCEPT SELECT docid FROM InvertedFile WHERE word= Microsoft ORDER BY magic_rank() There s only one SQL query template in Boolean Search Single-table selects, UNION, INTERSECT, EXCEPT magic_rank() is the secret sauce in the search engines Combos of statistics, linguistics, and graph theory tricks!

14 Updates and Text Search Text search engines are designed to be query-mostly Deletes and modifications are rare Can postpone updates (nobody notices, no transactions!) So no concurrency control problems For these reasons, text search engines and DBMSs are usually separate products Also, text-search engines tune that one SQL query to death! The benefits of a special-case workload.

15 DBMS vs. IR Systems { Query Optimization and Execution Relational Operators Files and Access Methods Buffer Management Search String Modifier Ranking Algorithm The Query The Access Method Buffer Management OS Simple } DBMS Disk Space Management Disk Space Management Concurrency and Recovery Needed DB DBMS DB Search Engine

16 Document Stores (Review) Like Key-Value Stores except value is document Data model: (key, document) pairs Document: JSON, XML, other semistructured formats Basic operations: Insert(key,document), Fetch(key), Update(key), Delete(key) Also Fetch based on document contents Example systems CouchDB, MongoDB, SimpleDB,

17 You Know The Basics of an IR System Inverted files are the workhorses of all text search engines Just B+-tree or Hash indexes on bag-of-words Intersect, Union and Set Difference (Except) Usually implemented via sorting Or can be done with hash or index joins Most of the other stuff is not systems work A lot of it is cleverness in dealing with language Both linguistics and statistics (more the latter!)

18 From Language to Knowledge Bases For humans, going from the largely unstructured language on the web to information is easy But for computers, it s rather difficult! This has suggested to many that if we re going to produce the next generation of intelligent agents, which can make decisions on our behalf Answering our routine Booking our next trip to Fiji we still first need to construct knowledge bases to go from languages to information 19

19 Knowledge/Information Extraction (IE) We are pleased that today's agreement guarantees our corporation will maintain a significant and long term presence in the Big Apple,'' McGraw-Hill president Harold McGraw III said in a statement. --- From New York Times April 24, 1997

20 Knowledge/Information Extraction (IE) We are pleased that today's agreement guarantees our corporation will maintain a significant and long term presence in the Big Apple,'' McGraw-Hill president Harold McGraw III said in a statement. --- From New York Times April 24, 1997 Labels: Person Company Location Other