Introduction Interactive Information Extraction

Transcription

1 Introduction Interactive Information Extraction Trausti Kristjansson, Aron Culotta, Paul Viola, Andrew McCallum IBM Research In USA, 7 millions worers complete forms on a regular basis. The goal of this wor is to reduce the burden on the user to the largest extent possible, while ensuring the integrity of the data. Main Points Synergy of User Interface and Information Extraction Algorithm CRFs for information extraction Correction Propagation in CRFs Confidence Estimation in CRFs Expected Number of User Actions Add Contacts to Address Boo Web Text document Word document Excel Demo: Contact Assistant Data Integrity Fast Verification Input is automatically parsed and assigned to fields Color coded correspondence, user can quicly spot errors 1

2 Correction Propagation Show live demo Interactive Information Extraction UI shows automatic field assignment results and allows for fast verification and fast correction IE algorithm taes corrections into account and propagates correction to other fields IE algorithm calculates confidence scores UI uses confidence scores to alert user to possible errors Classes Database Fields Constrained Conditional Random Fields and Confidence Estimation Classes Database Fields Classes Database Fields 2

3 Classes Database Fields Toen Features f ( y,t) Capitalized All Caps In In 1 st Word on line 2 nd Word on line 3 rd Word on line Previous Toen in Contains Digits Contains 5 Digits Contains Hyphen Enclosed in Bracets and 2 more Toen Features f ( y,t) Toen Features f ( y,t) Capitalized All Caps In In 1 st Word on line 2 nd Word on line 3 rd Word on line Previous Toen in Contains Digits Contains 5 Digits Contains Hyphen Enclosed in Bracets and 2 more Capitalized All Caps In In 1 st Word on line 2 nd Word on line 3 rd Word on line Previous Toen in Contains Digits Contains 5 Digits Contains Hyphen Enclosed in Bracets and 2 more Conditional Random Fields Conditional Random Fields are globally normalized probability models, where hidden variables are conditioned on observed variables. Conditional Random Fields Conditional Random Fields are globally normalized probability models, where hidden variables are conditioned on observed variables. Hidden States Observed Variables x t!2 y t!1 p( x y) Hidden States Observed Variables x t!2 y t!1 p( x y) Do not model the distribution over the observed variables, as generative models do. Advantage over generative models (e.g. HMMs) is that independence of observations not necessary Do not model the distribution over the observed variables, as generative models do (e.g. HMMs). Advantage over generative models is that independence of observations not necessary for tractability. 3

4 ! In! In! Previous Toen in! In! In! Previous Toen in! In! In! Previous Toen in! In! In! Previous Toen in! In! In! Previous Toen in Hidden States Observed Variables p Conditional Random Fields Finding the best state assignment g ( x,t) Relate adjacent hidden state variables g ( xt = first name, xt + 1 = postal code) " small! g ( xt = first name, xt + 1 = last name) " large! Capitalized All City Caps x t!2 Class Variables x!2 In State First t Name Postal Code f ( x, y,t) relate hidden state variables to In Country observed variables. address f ( ) y xt = postal code, yt contains 5digits " large! Feature Variables 1 st Webpage Word on URL line t!1 y t!1 OBSERVED 2 nd Word on line 3 rd Word on line Previous Toen in T T 1 & # Contains Digits ( x y) = exp$ '') g( x, t) + ''( f( x, y, t)! Contains 5 Digits Z % t= 1 t= 1 " Contains Hyphen Normalizing factor, i.e. sum over all state sequences for given observation Enclosed in Bracets Finding the best state assignment Viterbi used to find best sequence p( x y) =.93! 36 p( x y) =.59" 1! 5 p( x y) =.19" 1 Viterbi algorithm may return the sequence of states shown below State Variables xy t t!2! 2 Feature Variables OBSERVED (vectors of binary values) y t!1 xy t!2 yx t!1 t!1 yx tt Correction Propagation User Correction yx t +1 t +1 User Corrects a field, e.g. dragging Stanley to the field yx t+2 t xy t!2 yx t!1 t!1 yx tt yx t +1 t +1 4 yx t+2 t

5 Remove Paths User Corrects a field, e.g. dragging Stanley to the field Constrained Viterbi Viterbi algorithm is constrained to pass through the designated state. t!2 t!2 Adjacent field changed: Correction Propagation Indicate Low Confident Confidence Estimation Confidence in a classification Constrained Forward algorithm used to calculate sum of subset of paths that agree and disagree with a classification CE = P P( Classification) ( Any classification) Sum of all paths that agree with classification = Sum of all paths Sum of agreeing states sequences Paths that agree with classification All paths Sum of all state sequences t!2 xy t!2 yx t!1 t!1 yx tt yx t +1 t +1 5 yx t+2 t

6 Standard Metrics Standard information retrieval metrics: Evaluation CRF Toen Acc F Precision Recall MaxEnt These metrics don t relate well to the stated goals, e.g. how much does the system speed up data acquisition. Expected Number of User Actions Number of Incorrect Fields UI designers often use the Number of Clics as an objective metric. We would lie a similar metric for measuring the effectiveness of Correction Propagation We can calculate the Expected Number of User Actions (ENUA) from statistics of the number of erroneous fields in each record processed by the system. ENUA manual = Totalfields Total Records = Fields automatically assigned 2. User corrects remaining errors ENUA =.73 CRF Correct one field Correct one field CRF CRF

7 Correct one field Run correction propagation CRF + one correction CRF + one correction CRF + one correction + Correction Propogation Run correction propagation Expected Number of User Actions 2 CRF + one correction CRF + one correction + Correction Propogation 1. Fill in fields automatically 2. User corrects a field 3. Correction Propagation 4. User corrects remaining errors ENUA =.63 Model/UI-Model ENUA Change Manual UIMm 8.5x 6.31 Baseline CRF UIM % 1x -13.9% CCRF UIM % 1 Confidence Estimation Summary 276 records had one or more errors. If the least confident field highlighted in a record with one or more errors, an error will be identified 81.9% of the time. If field is chosen at random, an error will be identified 29.% of the time. This illustrates the potential for using confidence to direct the users attention to an incorrect field. Synergy of User Interface and Information Extraction Algorithm ensuring confidence integrity of data Over 88% reduction of User Actions by Information Extraction alone Additional 13% reduction in User Actions due to Correction Propagation Confidence Scores effective at identifying incorrect fields. IIE in Microsoft Office 27??? 7

8 End 8