Sequence Labeling: The Problem

Transcription

1 Sequence Labeling: The Problem Given a sequence (in NLP, words), assign appropriate labels to each word. For example, POS tagging: DT NN VBD IN DT NN. The cat sat on the mat. 36 part-of-speech tags used in the Penn Treebank Project: g001/penn_treebank_pos.html

2 Sequence Labeling: The Problem Given a sequence (in NLP, words), assign appropriate labels to each word. Another example, partial parsing (aka chunking): B-NP I-NP B-VP B-PP B-NP I-NP The cat sat on the mat

3 Sequence Labeling: The Problem Given a sequence (in NLP, words), assign appropriate labels to each word. Another example, relation extraction (e.g., arguments and relation types): B-Arg I-Arg B-Rel I-Rel B-Arg I-Arg The cat sat on the mat Madam Curie won Turing Award

4 Knowledge Bases A knowledge base is a collection of entity, facts, relationships that conforms with a certain data model. A knowledge base helps machine understand humans, languages, and the world. Examples 1: Google Knowledge Graph

5 Knowledge Bases From Big Data Example 2: TREC Knowledge Base Acceleration [News, Blog, Tweets] KB Applications: Improve Search Engine (e.g., Google, Bing) Automatically populating Wikipedia (e.g., references, info box) Domain-specific Knowledge Bases (e.g., UMLS) 26

6 TREC Knowledge Base Acceleration System (GatorDSR) 1. Streaming/Data Processing System 2. POS 3. Chunking 4. Named entity extraction 5. Co-reference 6. Relation extraction 7. Slot filling 27

7 Probabilistic graphical models Graphical models Probabilistic models Directed (Bayesian networks) Undirected (Markov Random fields - MRFs)

8 Bayesian Networks Directed Acyclic Graph (DAG) Nodes are random variables Edges indicate causal influences Conditional independence: each RV is conditionally independent of other nodes given its parent nodes Burglary Earthquake Alarm JohnCalls MaryCalls

9 Conditional Probability Tables Each node has a conditional probability table (CPT) that gives the probability of each of its values given every possible combination of values for its parents (conditioning case). Roots (sources) of the DAG that have no parents are given prior probabilities. P(B).001 Burglary Earthquake P(E).002 Alarm B E P(A) T T.95 T F.94 F T.29 F F.001 A P(J) T.90 F.05 JohnCalls MaryCalls A P(M) T.70 F.01

10 Joint Distributions for Bayes Nets A Bayesian Network implicitly defines a joint distribution. P( x, x,... x ) n P( x Parents( X 1 2 n i i i1 Example P( J M AB E) P( J A) P( M A) P( A B E) P( B) P( E) ))

11 Naïve Bayes as a Bayes Net Naïve Bayes is a simple Bayes Net Y X 1 X 2 Xn Priors P(Y) and conditionals P(X i Y) for Naïve Bayes provide CPTs for the network.

12 Markov Networks Undirected graph over a set of random variables, where an edge represents a dependency. The Markov blanket of a node, X, in a Markov Net is the set of its neighbors in the graph (nodes that have an edge connecting to X). Every node in a Markov Net is conditionally independent of every other node given its Markov blanket.

13 Distribution for a Markov Network The distribution of a Markov net is most compactly described in terms of a set of potential functions (a.k.a. factors, compatibility functions), φ k, for each clique, k, in the graph. For each joint assignment of values to the variables in clique k, φ k assigns a non-negative real value that represents the compatibility of these values. The joint distribution of a Markov network is then defined by: 1 P x, x,... x ) ( x ) ( 1 2 n k { k} Z k Where x {k} represents the joint assignment of the variables in clique k, and Z is a normalizing constant that makes a joint distribution that sums to 1. Z ) x k k ( x{ k}

14 Sample Markov Network B A 1 T T 100 T F 1 F T 1 F F 200 Burglary Earthquake E A 2 T T 50 T F 10 F T 1 F F 200 Alarm J A 3 T T 75 T F 10 F T 1 F F 200 JohnCalls MaryCalls M A 4 T T 50 T F 1 F T 10 F F 200

15 Logistic Regression as a Markov Net Logistic regression is a simple Markov Net potential functions φ k (x{k}) instead of conditional probability tables P(X i Y) Y X 1 X 2 Xn But only models the conditional distribution, P(Y X) and not the full joint P(X,Y) Same as a discriminatively trained naïve Bayes.

16 Generative vs. Discriminative Generative models and are not directly designed to maximize the performance of sequence labeling. They model the joint distribution P(O,Q). Generative NLP models are trained to have an accurate probabilistic model of the underlying language, and not all aspects of this model benefit the sequence labeling task. Discriminative models (CRFs) are specifically designed and trained to maximize performance of labeling, which leads to more accurate results. They model the conditional distribution P(Q O).

17 Classification Models Y X 1 X 2 Xn Generative Naïve Bayes Conditional Y Discriminative X 1 X 2 Xn Logistic Regression

18 Sequence Labeling Models.. Y 1 Y 2 Y T HMM X 1 X 2 XT Generative Conditional Y 1 Y 2 Discriminative.. Y T Linear-chain CRF X 1 X 2 XT

19 A Graphical Model Conditional Random Fields (CRF) Text (address string): E.g., 2181 Shattuck North Berkeley CA USA CRF Model: 2181 Shattuck North Berkeley CA USA X=tokens Y=labels x 0 y 0 x 1 y 1 x 2 y 2 Possible Extraction Worlds: x 2181 Shattuck North Berkeley CA USA y1 apt. num street name city city state country (0.6) y2 apt. num street name street name city state country (0.1) x 3 y 3 x 4 y 4 x 5 y 5 40

20 Viterbi Algorithm MAP labeling Viterbi Dynamic Programming Algorithm: V max 0, if ( V ( i 1, i 1. y') y' k ( i, y) k 1 K f k f ( y, y', xi)), if i Shattuck North Berkeley CA USA pos stree t num street name city stat e country

21 Viterbi Implemented in SQL [ICDE10] V Viterbi Dynamic Programming Algorithm: max 0, if ( V ( i 1, i 1. y') y' k ( i, y) k 1 K f V (i, y) V (i, y) k f ( y, y', xi)), if i 0 Aggr(top-k) Group By Aggr(summation) Recursive Join TokenTable FactorTable Viterbi Topk_Array Recursive Join TokenTable FactorTable ViterbiArray 42

22 Summary Text Retrieval Systems Elementary IR Scalable Boolean Text Search Knowledge construction from Text Text to Knowledge Probabilistic Graphical Models HMMs and CRFs for sequence labeling 43