Fusion of Content and Context in Human Language Technology

Transcription

1 Fusion of Content and Context in Human Language Technology Allen Gorin Human Language Technology Research Na4onal Security Agency Fort Meade, Maryland Graph Exploitation Workshop, August 2011

2 Report Documentation Page Form Approved OMB No Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington VA Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. 1. REPORT DATE AUG REPORT TYPE N/A 3. DATES COVERED - 4. TITLE AND SUBTITLE Fusion of Content and Context in Human Language Technology 5a. CONTRACT NUMBER 5b. GRANT NUMBER 5c. PROGRAM ELEMENT NUMBER 6. AUTHOR(S) 5d. PROJECT NUMBER 5e. TASK NUMBER 5f. WORK UNIT NUMBER 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) Human Language Technology Research Na4onal Security Agency Fort Meade, Maryland 8. PERFORMING ORGANIZATION REPORT NUMBER 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR S ACRONYM(S) 12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release, distribution unlimited 11. SPONSOR/MONITOR S REPORT NUMBER(S) 13. SUPPLEMENTARY NOTES See also ADA Graph Exploitation Symposium. Held in Lexington, Massachusetts on August 9-10, ESC-TR ABSTRACT 15. SUBJECT TERMS 16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF ABSTRACT SAR a. REPORT unclassified b. ABSTRACT unclassified c. THIS PAGE unclassified 18. NUMBER OF PAGES 52 19a. NAME OF RESPONSIBLE PERSON Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std Z39-18

3 Collaborators Carey Priebe (JHU) John Grothendieck (BBN) Glen Coppersmith (JHU HLT COE) Walt Andrews (BBN) John Conroy (IDA CCS) Richard Cox (COE) Mike Decerbo (BBN) Nam Lee (COE) Dave MarcheIe (NSWC) Alan McCree (MIT LL) Youngser Park (COE) 2

4 Outline Mo4va4on: Coping with Informa4on Overload Examples of Context and Content Random AIributed Graphs Three Tasks Stream Characteriza4on Vertex Nomina-on Dyadic Priors 3

5 Coping with InformaHon Overload Data Streams and substreams _...,._..., 4

6 Coping with InformaHon Overload Data Streams and substreams Bandwidth Reduction IJI 4

7 Coping with InformaHon Overload Data Streams and substreams Bandwidth Reduction IJI Pick out e good stuff Filter and Select 4

8 Coping with InformaHon Overload Data Streams and substreams Bandwidth Reduction IJI Pick out e good stuff Filter and Select Boil it down Stream Characterization 4

9 Coping with InformaHon Overload 4

10 Coping with InformaHon Overload 4

11 Coping with InformaHon Overload 4

12 Coping with InformaHon Overload 4

13 Content has associated meta- data challenge: how to exploit? 5

14 Content has associated meta- data challenge: how to exploit? Associated meta- data is interpreted by humans as context 5

15 Content has associated meta- data challenge: how to exploit? Associated meta- data is interpreted by humans as context Humans acquire and use language during interac4on with a complex environment, e.g. Roy s Speechome project at MIT 5

16 Content has associated meta- data challenge: how to exploit? Associated meta- data is interpreted by humans as context Humans acquire and use language during interac4on with a complex environment, e.g. Roy s Speechome project at MIT Call centers have customer- profiles 5

17 Content has associated meta- data challenge: how to exploit? Associated meta- data is interpreted by humans as context Humans acquire and use language during interac4on with a complex environment, e.g. Roy s Speechome project at MIT Call centers have customer- profiles Voice messages have to/from telephone numbers 5

18 Content has associated meta- data challenge: how to exploit? Associated meta- data is interpreted by humans as context Humans acquire and use language during interac4on with a complex environment, e.g. Roy s Speechome project at MIT Call centers have customer- profiles Voice messages have to/from telephone numbers Enron corpus has date, 4me, sender and recipients 5

19 Content has associated meta- data challenge: how to exploit? Associated meta- data is interpreted by humans as context Humans acquire and use language during interac4on with a complex environment, e.g. Roy s Speechome project at MIT Call centers have customer- profiles Voice messages have to/from telephone numbers Enron corpus has date, 4me, sender and recipients Switchboard dialog corpus has demographics: age, gender,. 5

20 Content has associated meta- data challenge: how to exploit? Associated meta- data is interpreted by humans as context Humans acquire and use language during interac4on with a complex environment, e.g. Roy s Speechome project at MIT Call centers have customer- profiles Voice messages have to/from telephone numbers Enron corpus has date, 4me, sender and recipients Switchboard dialog corpus has demographics: age, gender,. Citeseer scien4fic ar4cles have authors and cita4ons 5

21 CommunicaHon Events from the Enron Corpus Date Time Sender Receiver Sender's Rank Topic :15:00 steven.k jeff.d Vice President (1) California Analysis :49:09 louise.k andy.z President (9) Daily Business :06:00 drew.f jeff.d Vice President (5) California Enron :30:00 james.s john.i Vice President (14) Energy Newsfeed :54:00 d iana.s kate.s Trader (5) California Enron :15:00 m ike.g john.i Manager (7) Newsfeed California :12:00 richard.s steven.k Vice President (9) Daily Business :02:00 andy.z joh n.l Vice President (11) Enron Online :44:24 s.. s geoff.s Vice President (9) Daily Business :36:53 geoff.s louise.k Director (12) Enrononline Daily :51:20 m.. p louise.k Vice President (12) Enrononline Daily :19:16 john.l louise.k CEO (11) Enron Online :49:05 j.. k richard.s Vice President (9) Daily Business :50:19 shelley.c darrell.s Vice President (1) California Analysis 6

22 SwitchBoard CommunicaHons Graph Vertex ~ speaker Edge ~ dialog 7

23 Time Series of AMributed Graphs Externals 3 ~ Content.=.c From: k(lnl'l(l':l'l,l;)y@enron.wm To: l'lasal\.kect.va.@etwen.com Time 8

24 Time Series of AMributed Graphs Externals 3 ~ Content.=.c From: k(lnl'l(l':l'l,l;)y@enron.wm To: l'lasal\.kect.va.@etwen.com Time Generated by some random process G t? 8

25 Random AMributed Graphs (RAGs) 9

26 Random AMributed Graphs (RAGs) There is significant literature on random graphs, ignoring content. 9

27 Random AMributed Graphs (RAGs) There is significant literature on random graphs, ignoring content. There is significant literature on stochas4c models for language and documents streams, ignoring context. 9

28 Random AMributed Graphs (RAGs) There is significant literature on random graphs, ignoring content. There is significant literature on stochas4c models for language and documents streams, ignoring context. There is a computer science literature on aiributed graphs, e.g. as produced by en4ty and rela4ons, ignoring stochas4c modeling. 9

29 Random AMributed Graphs (RAGs) There is significant literature on random graphs, ignoring content. There is significant literature on stochas4c models for language and documents streams, ignoring context. There is a computer science literature on aiributed graphs, e.g. as produced by en4ty and rela4ons, ignoring stochas4c modeling. Before this research effort, no literature that we know of addressing 4me series of random aiributed graphs. 9

30 GeneraHve Models for RAGs Build RAG models by extending random graph models Erdos- Renyi (binomial) graphs, where a pair of ver4ces is connected with iid probability p. Kidney/Egg models, Block models Latent Posi4on and Random Dot Product Models where p ij = h(x i, x j ) Construct from 4me series of communica4on events M = { (t, u t, v t, s t ) } t 10

31 Vertex NominaHon Cf. fraud and social network analysis significant literature using graphs Intui4on for fusion is clear Experimental evalua4on on Enron corpus Summer workshop at JHU Human Language Technology COE par4cipants from all over the U.S.

32 Experimental Methodology Given a set of red ver4ces Occlude subset of red ver4ces Develop method for nomina4ng ver4ces as red Evaluate on how well it discovers those occluded red ver4ces versus false nomina4ons 12

33 Enron Example: Red Vertices -> Red Documents k.lay e.haedicke j.lovarato 13

34 Enron Example: Red Vertices -> Red Documents k.lay e.haedicke j.lovarato

35 Enron Example: Red Vertices -> Red Documents k.lay e.haedicke j.lovarato

36 Enron Example: Red Vertices -> Red Documents k.lay WBI Energy Services billions of cubic feet e.haedicke j.lovarato

37 Enron Example: Red Vertices -> Red Documents Portland....San Diego.... WBI.. k.lay.. Portland.. Portland..... bcf.. WBI Energy Services billions of cubic feet e.haedicke j.lovarato San Diego..... WBI...bcf

38 Edge AMributed Graph - > Latent Vertex AMributes 14

39 Edge AMributed Graph - > Latent Vertex AMributes From red ver4ces, now have induced red topic model 14

40 Edge AMributed Graph - > Latent Vertex AMributes From red ver4ces, now have induced red topic model Use red model which is used to airibute all edges in the graph 14

41 Edge AMributed Graph - > Latent Vertex AMributes From red ver4ces, now have induced red topic model Use red model which is used to airibute all edges in the graph Es4mate latent vertex aiributes for an aiributed rdp model that best fit this aiributed graph 14

42 Edge AMributed Graph - > Latent Vertex AMributes From red ver4ces, now have induced red topic model Use red model which is used to airibute all edges in the graph Es4mate latent vertex aiributes for an aiributed rdp model that best fit this aiributed graph The vertex aiributes are x 0, x 1, x 2, where 14

43 Edge AMributed Graph - > Latent Vertex AMributes From red ver4ces, now have induced red topic model Use red model which is used to airibute all edges in the graph Es4mate latent vertex aiributes for an aiributed rdp model that best fit this aiributed graph The vertex aiributes are x 0, x 1, x 2, where x 1 is the tendency of the vertex to engage in red communica4ons 14

44 Edge AMributed Graph - > Latent Vertex AMributes From red ver4ces, now have induced red topic model Use red model which is used to airibute all edges in the graph Es4mate latent vertex aiributes for an aiributed rdp model that best fit this aiributed graph The vertex aiributes are x 0, x 1, x 2, where x 1 is the tendency of the vertex to engage in red communica4ons abuse jargon and call this the redness of the vertex 14

45 Edge AMributed Graph - > Latent Vertex AMributes From red ver4ces, now have induced red topic model Use red model which is used to airibute all edges in the graph Es4mate latent vertex aiributes for an aiributed rdp model that best fit this aiributed graph The vertex aiributes are x 0, x 1, x 2, where x 1 is the tendency of the vertex to engage in red communica4ons abuse jargon and call this the redness of the vertex x 2 is the tendency to engage in non- red communica4on 14

46 Edge AMributed Graph - > Latent Vertex AMributes From red ver4ces, now have induced red topic model Use red model which is used to airibute all edges in the graph Es4mate latent vertex aiributes for an aiributed rdp model that best fit this aiributed graph The vertex aiributes are x 0, x 1, x 2, where x 1 is the tendency of the vertex to engage in red communica4ons abuse jargon and call this the redness of the vertex x 2 is the tendency to engage in non- red communica4on call this the greenness of the vertex 14

47 Edge AMributed Graph - > Latent Vertex AMributes From red ver4ces, now have induced red topic model Use red model which is used to airibute all edges in the graph Es4mate latent vertex aiributes for an aiributed rdp model that best fit this aiributed graph The vertex aiributes are x 0, x 1, x 2, where x 1 is the tendency of the vertex to engage in red communica4ons abuse jargon and call this the redness of the vertex x 2 is the tendency to engage in non- red communica4on call this the greenness of the vertex x 0 = 1 - x 1 - x 2 = non- edginess = tendency of the vertex to stay mum 14

48 Latent Vertex Attributes live in the 2D simplex x0 ~ mum-ness x1 ~ redness x2 ~ greenness 15

49 Distribution of 184 Latent Vertex Attributes denote initial red vertices denote other vertices X0 X2 mum-ness green-ness X1 redness 16

50 Distribution of 184 Latent Vertex Attributes Sparse Communication Graph most vertices are not very communicative, and their s are not dominated by red or green denote initial red vertices denote other vertices X0 X2 mum-ness green-ness X1 redness 16

51 Anomalous Chatter Group in Enron Time Series Induced Egg Egg? p>0.99 p~ 0.7 p < 0.01 Time Weeks Weeks Weeks

52 Conclusions New Methods for Fusion of Context and Content Pioneered at JHU Human Language Technology COE Theory, Algorithms and Experimental Evalua4on Tasks Stream Characteriza4on Vertex Nomina4on Dyadic Priors Experimentally evaluated on Enron corpus Switchboard speech corpus other data 18

53 Some References Sta5s5cal Inference on Random Graphs: Fusion of Graph Features and Content, Grothendieck, Priebe, and Gorin, Computa4onal Sta4s4cs and Data Analysis (2010) Sta5s5cal Inference on random a?ributed Graphs: Fusion of Graph Features and Content: An Experiment on Time- series of Enron Graphs, Priebe et al, Computa4onal Sta4s4cs and Data Analysis (2010). Towards Link Characteriza5on from Content: Recovering Distribu5ons from Classifier Output, Grothendieck and Gorin, IEEE Transac4ons on Speech and Audio, May 2008 Vertex Nomina5on via Content and Context, Coppersmith and Priebe submi5ed for publica8on Vertex Nomina5on via A?ributed Random Dot Product Graphs, Marche5e, Priebe, Coppersmith, Proc. Interna8onal Sta8s8cal Ins8tute, Latent Process Model for Time Series of A?ributed Random Graphs, Lee and Priebe, Sta8s8cal Inference for Stochas8c Processes,