Vertex Nomina-on. Glen A. Coppersmith. Human Language Technology Center of Excellence Johns Hopkins University. improved fusion of content and context

Size: px

Start display at page:

Download "Vertex Nomina-on. Glen A. Coppersmith. Human Language Technology Center of Excellence Johns Hopkins University. improved fusion of content and context"

Barry Williamson
6 years ago
Views:

1 Vertex Nomina-on improved fusion of content and context Glen A. Coppersmith Human Language Technology Center of Excellence Johns Hopkins University Presented at Interface Symposium: May 17, 2012

2 Vertex Nomina-on improved fusion of content and context Glen A. Coppersmith Human Language Technology Center of Excellence Johns Hopkins University Presented at Interface Symposium: May 17, 2012

3 Vertex Nomina-on improved fusion of content and context Human Language Content Communica/ons Graph Glen A. Coppersmith Human Language Technology Center of Excellence Johns Hopkins University Presented at Interface Symposium: May 17, 2012

4 Our data: Enron Corpus

5 Mo-va-on and Problem Statement We know the iden--es of a few fraudsters. We observe the content and the context of both fraudsters and non- fraudsters. We want to know the iden--es of more fraudsters. Inference Task Nominate persons likely to be fraudsters.

6 Outline Introduc-on Method Importance Sampling Evalua-on Analy-cs Content and Context Fusions Conclusions & Future Direc-ons

7 Outline Introduc-on Method Importance Sampling Evalua-on Analy-cs Content and Context Fusions Conclusions & Future Direc-ons

8 Mo-va-on and Problem Statement We know the iden--es of a few fraudsters. We observe the content and the context of both fraudsters and non- fraudsters. We want to know the iden--es of more fraudsters. Inference Task Nominate persons likely to be fraudsters.

9 Mo-va-on and Problem Statement We know the iden--es of a few fraudsters. [Graph a[ributed with human language]interes-ng and non- interes-ng. We want to know the iden--es of more fraudsters. Inference Task Nominate persons likely to be fraudsters.

10 With loss of generality Ne^lix [See Trevor s Keynote] Genomics [Half the talks I ve seen at IF12] Noted similari-es Recommender Systems We focus on those with a graph and those with human language

11 With loss of generality Ne^lix [See Trevor s Keynote] Genomics [Half the talks I ve seen at IF12] Noted similari-es Recommender Systems We focus on those with a graph and those with human language

12 What s already been done? Theory (Nam) Lee & Priebe 2011 (Dominic) Lee & Priebe (submi[ed 2012) Simula-ons and Experiments Marche[e, Priebe, and Coppersmith (2011) Coppersmith & Priebe (submi[ed 2011) (Content + Context) > (Content Context) Human Language Technology and Graph Theory Assump-ons are valid for the Enron data

13 What s already been done? (Content + Context) > (Content Context) Human Language Technology and Graph Theory Assump-ons are valid for the Enron data Importance Sampling provides sensible par--ons

14 Assump-ons The fraudsters talk to each other more than expected of a random pair of people. The fraudsters talk about different things than expected of a random pair of people.

15 Fusion of Disparate Informa-on Some signal from the communica-ons graph Some signal from the human language content How do you fuse them? A principled fusion would be nice A useful fusion is more important Robust to real world problems Scalable to real world applica-ons

16 Ques-ons for this talk How should we fuse? (both performance and scalability are important) What kind of HLTs should we use? (Do they do different things?)

17 Outline Introduc-on Method Importance Sampling Evalua-on Analy-cs Content and Context Fusions Conclusions & Future Direc-ons

18 Mathema-cal Model: κ graph V =n M =m M =m V\M =n- m p = [p 0,p 1 ] s = [s 0,s 1 ] p 0 = s 0 p 1 < s 1 p p V\M s M κ(n, p, m, m, s)

19 Mathema-cal Model: κ graph V =n M =m M =m V\M =n- m p = [p 0,p 1 ] s = [s 0,s 1 ] p 0 = s 0 p 1 < s 1 p κ(n, p, m, m, s) p s V\M M fraudsters innocents

20 Assump-ons The fraudsters talk to each other more than expected of a random pair of people. M is more dense than V\M The fraudsters talk about different things than expected of a random pair of people. p[p 0,p 1 ] and s[s 0,s 1 ] p 0 = s 0, p 1 < s 1 V\M M

21 Our data: Enron Corpus

22 Our data: Enron Corpus but where are the fraudsters, Glen!?

23 Importance Sampling Procedure Randomly par--on Enron into M and V\M Ques-on assump-ons Density M > Density V\M Topic Distribu-on M Topic Distribu-on V\M Discard par--ons that violate assump-ons. Collect 5000 par--ons. V\M M

24 Importance Sampling Hand- labels provided by Michael Berry, 2004

25 Importance Sampling Hand- labels provided by Michael Berry, 2004

26 Importance Sampling M j.lovarato@enron e.haedicke@enron k.lay@enron r.shapiro@enron a.zipper@enron b.rogers@enron V\M

27 Tes-ng Par--ons: Density ρ(m) = observed edges in M possible edges in M Δρ = ρ(m) ρ(v\m) The fraudsters talk to each other more than expected of a random pair of people. V\M M

28 Tes-ng Par--ons: Topic Distribu-on Topic(M) = Topic(V\M) = Δ Topic = The fraudsters talk about different things than expected of a random pair of people. V\M M

29 Importance Sampling M j.lovarato@enron e.haedicke@enron k.lay@enron r.shapiro@enron a.zipper@enron b.rogers@enron V\M

30 Method M =10 M =5 V\M =179 For each vertex (v) in V\M we calculate each analy-c. Rank ver-ces according to each analy-c or fusion. Evaluate quality of ranked lists. M M V\M

31 One experiment M j.lovarato@enron e.haedicke@enron k.lay@enron r.shapiro@enron a.zipper@enron b.rogers@enron V\M

32 One experiment M j.lovarato@enron e.haedicke@enron k.lay@enron r.shapiro@enron a.zipper@enron b.rogers@enron V\M

33 Evalua-on Ranked lists evaluated by standard Informa-on Retrieval measures What is our inference task? Need to find all of them Mean Average Precision (MAP) Need to find one more of them Mean Reciprocal Rank (MRR) Can only examine k ver-ces (p@k), k = {5,10}

34 Outline Introduc-on Method Importance Sampling Evalua-on Analy-cs Content and Context Fusions Conclusions & Future Direc-ons

35 Context Analy-c The fraudsters talk to each other more than expected of a random pair of people. Number of known fraudsters in 1- hop neighborhood of candidate vertex. M j.lovarato@enron e.haedicke@enron k.lay@enron

36 Content Analy-cs (HLTs) The fraudsters talk about different things than expected of a random pair of people. How similar is the content of each candidate vertex to the known fraudsters?

37 Training HLTs M j.lovarato@enron e.haedicke@enron k.lay@enron r.shapiro@enron a.zipper@enron b.rogers@enron V\M

38 Training HLTs M j.lovarato@enron e.haedicke@enron k.lay@enron r.shapiro@enron a.zipper@enron b.rogers@enron V\M

39 Training HLTs M e.haedicke@enron j.lovarato@enron Interes-ng k.lay@enron r.shapiro@enron a.zipper@enron b.rogers@enron V\M

40 Training HLTs M e.haedicke@enron j.lovarato@enron Interes-ng k.lay@enron r.shapiro@enron a.zipper@enron b.rogers@enron V\M

41 Training HLTs M e.haedicke@enron j.lovarato@enron Interes-ng k.lay@enron r.shapiro@enron Uninteres-ng a.zipper@enron b.rogers@enron V\M

42 HLT 1 : Average Word Count Histogram What propor-on of the document d i is made up of word w j? Each d i represented as probability vector x i. x = W, W word types in the corpus. Vector I is average of all interes-ng ( ) x i. Interes-ng

43 HLT 1 : Average Word Count Histogram Score each d i by 1- JS( x i, I ) 0.83 (High) Interes-ng 0.23 (Low)

44 HLT 1 : Average Word Count Histogram Score all d i for each vertex: (k.lay@enron) Average scores Interes-ng M j.lovarato@enron e.haedicke@enron k.lay@enron

45 HLT 1 : Average Word Count Histogram Score all d i for each vertex: (k.lay@enron) Average scores Interes-ng M j.lovarato@enron e.haedicke@enron k.lay@enron

46 HLT 1 : Average Word Count Histogram Score all d i for each vertex: (k.lay@enron) Average scores Interes-ng M j.lovarato@enron e.haedicke@enron k.lay@enron

47 HLT 1 : Average Word Count Histogram Score all d i for each vertex: (k.lay@enron) Average scores Interes-ng M j.lovarato@enron e.haedicke@enron k.lay@enron

48 HLT 1 : Average Word Count Histogram Score all d i for each vertex: (k.lay@enron) Average scores 0.72 Interes-ng M j.lovarato@enron e.haedicke@enron k.lay@enron

49 HLT 2 : Closest Word Count Histogram Score all d i for each vertex: (k.lay@enron) Score only by closest document 0.72 Interes-ng M j.lovarato@enron e.haedicke@enron k.lay@enron

50 HLT 3 : Compression Language Modeling How well does a given message compress? Repeated sequences compress well Novel sequences do not compress well

51 HLT 3 : Compression Language Modeling Discrimina-ve Model which model fits be[er? Interes-ng Red Uninteres-ng

52 HLT 3 : Compression Language Modeling M e.haedicke@enron j.lovarato@enron Interes-ng k.lay@enron r.shapiro@enron Uninteres-ng a.zipper@enron b.rogers@enron V\M

53 HLT 4 : Topic Modeling Latent Dirichlet Alloca-on (ala Blei, Wallach, McCallum, Mimno,...) [we use mallet] Each document is a mixture of topics Each topic is a probability distribu-on over W mallet is available from h[p://mallet.cs.umass.edu/

54 HLT 4 : Topic Modeling j.lovarato@enron e.haedicke@enron k.lay@enron r.shapiro@enron a.zipper@enron b.rogers@enron

55 HLT 4 : Topic Modeling j.lovarato@enron e.haedicke@enron k.lay@enron r.shapiro@enron a.zipper@enron b.rogers@enron

56 HLT 4 : Topic Modeling M j.lovarato@enron e.haedicke@enron k.lay@enron r.shapiro@enron a.zipper@enron b.rogers@enron V\M

57 HLT 4 : Topic Modeling M j.lovarato@enron e.haedicke@enron k.lay@enron r.shapiro@enron a.zipper@enron b.rogers@enron V\M

58 HLT 4 : Topic Modeling M j.lovarato@enron e.haedicke@enron k.lay@enron r.shapiro@enron a.zipper@enron b.rogers@enron V\M

59 HLT 4 : Topic Modeling Score all d i for each vertex: (k.lay@enron) Sum the weight given to topics of interest M j.lovarato@enron e.haedicke@enron k.lay@enron

60 HLT 4 : Topic Modeling Score all d i for each vertex: (k.lay@enron) Sum the weight given to topics of interest M j.lovarato@enron e.haedicke@enron k.lay@enron

61 HLT 4 : Topic Modeling Score all d i for each vertex: (k.lay@enron) Sum the weight given to topics of interest M j.lovarato@enron e.haedicke@enron k.lay@enron

62 HLT 4 : Topic Modeling Score all d i for each vertex: (k.lay@enron) Sum the weight given to topics of interest 0.12 M j.lovarato@enron e.haedicke@enron k.lay@enron

63 Individual Analy-c Performance MAP Context Average WCH Minimum WCH Compression Topic Modeling

64 Individual Analy-c Performance MAP * Context Average WCH Minimum WCH Compression Topic Modeling

65 Outline Introduc-on Method Importance Sampling Evalua-on Analy-cs Content and Context Fusions Conclusions & Future Direc-ons

66 Linear Fusion F = γ HLT 1 + (1- γ) Context 2 Analy-cs

67 Linear fusion 2 Analy-cs Performance MAP Context γ 1 Content

68 Linear fusion 2 Analy-cs Performance MAP Grid search over Content and Context Context γ 1 Content

69 Linear Fusion F = γ HLT 1 + (1- γ) Context 2 Analy-cs F = Σ i (γ i HLT i ) + (1- Σ i γ i ) Context Arbitrary number of Analy-cs NB: Scores must be calibrated.

70 MAP MRR Gridsearch Linear Combina-ons MAP

71 Rank Fusion Fuse ranks instead of scores. Rank ver-ces by each analy-c. Each vertex represented by vector of ranks Fusion score is a func-on of that vector Min() One measure can damn you Max() One measure can save you Median() Something in between

72 Rank Fusion MAP min median max min median max 3 Analy-cs 5 Analy-cs

73 Rank Fusion MAP min median max min median max 3 Analy-cs 5 Analy-cs

74 Rank Fusion MAP min median max min median max 3 Analy-cs 5 Analy-cs

75 Individual Rank Fusions Gridsearch Linear Combina-ons Overall Comparisons MAP

76 Individual Rank Fusions Gridsearch Linear Combina-ons Overall Comparisons MAP

77 Individual Rank Fusions Gridsearch Linear Combina-ons O(n) in number of Analy-cs MAP

78 Individual Rank Fusions Gridsearch Linear Combina-ons O(x n ) in number of Analy-cs!! MAP

79 Individual Rank Fusions Gridsearch Linear Combina-ons Nevermind O(.) for a moment MAP ~11 minutes ~13 days

80 Individual Rank Fusions Gridsearch Linear Combina-ons Nevermind O(.) for a moment MAP ~11 minutes ~13 days

81 Linear fusion 3 Analy-cs

82 Which HLTs should I use? Average WCH + Minimum WCH + Topic Modeling 0.1 Other HLTs Compression Context

83 Which HLTs should I use? Need to find them all Need to find only one more 0.1 Other HLTs Compression Can only look at 5 or 10 Context

84 Outline Introduc-on Method Importance Sampling Evalua-on Analy-cs Content and Context Fusions Conclusions & Future Direc-ons

85 Ques-ons How should we fuse? For small number of analy-cs grid search linear For large number of analy-cs rank fuse (or think harder) What kind of HLTs should we use? Depends on your inference task, of course. We can actually recommend what to use! Insight into rela-ve strengths of HLTs exposed.

86 Future and Related Work This is post- hoc fusion of scores or ranks, what about more na-ve fusion? Different data and inference tasks What movie should I watch? (Ne^lix) Who should I cite? (ACL) Who should I work with? (Github) Vote predic-on (Congressional Bills)

87 Collaborators Carey Priebe [JHU HLTCOE] Allen Gorin [JHU HLTCOE] Richard Cox [JHU HLTCOE] David Marche[e [Navy] Yongser Park [JHU CIS] Minh Tang [JHU AMS] Michael Decerbo [Raytheon BBN] Hanna Wallach [UMass Amherst] Jim Mayfield [JHU APL/HLTCOE] Paul McNamee [JHU APL/HLTCOE] William Szewczyk [DoD]

88 Collaborators Carey Priebe [JHU HLTCOE] Allen Gorin [JHU HLTCOE] Richard Cox [JHU HLTCOE] David Marche[e [Navy] Yongser Park [JHU CIS] Minh Tang [JHU AMS] Michael Decerbo [Raytheon BBN] Hanna Wallach [UMass Amherst] Jim Mayfield [JHU APL/HLTCOE] Paul McNamee [JHU APL/HLTCOE] William Szewczyk [DoD]

Thank you. Coppersmith & Priebe (submi[ed): [arxiv.org/1201.

89 Thank you. Coppersmith & Priebe (submi[ed): [arxiv.org/ ] Latest papers (oƒen) available from glencoppersmith.com

91 Content of Importance Sample California Power.. Portland...San Diego..... WBI California Power.. Portland.. Portland..... bcf.. WBI Energy Services billions of cubic feet California Power.. San Diego..... WBI...bcf....

Content of Importance Sample r.shapiro@enron Pro Football.. Raiders... touchdown.

92 Content of Importance Sample Pro Football.. Raiders... touchdown.....implode 9/11.. 9/11.. New York.... tragedy

93 Context of Importance Sample

94 Context of Importance Sample Observed s 1 = 0.45

95 Context of Importance Sample Observed p 1 = 0.13

96 Context of Importance Sample Observed p 1 = 0.12

97 Context of Importance Sample Observed p 1 = 0.13 Observed s 1 = 0.45 Observed p 1 = 0.12

98 Content of Importance Sample Portland...San Diego..... WBI k.lay@enron.. Portland.. Portland..... bcf.. WBI Energy Services billions of cubic feet e.haedicke@enron San Diego..... WBI...bcf.... j.lovarato@enron

99 Comparing HLT Methods MRR X X N X X X X X + + N X N N N N N X X N + + N N + X + X N N N + X N N N N N N N N N N + + N X + + N N+ N N X N N N N N N N N N N N N N N N+ N S 1

100 Importance Sampled Joint Δ Topic Δ ρ

101 Injec-on Importance P P 1 P S S 1 S 1

Vertex Nomina-on. Glen A. Coppersmith. Human Language Technology Center of Excellence Johns Hopkins University. fusion of content and context

Vertex Nomina-on. Glen A. Coppersmith. Human Language Technology Center of Excellence Johns Hopkins University. fusion of content and context Vertex Nomina-on fusion of content and context Glen A. Coppersmith Human Language Technology Center of Excellence Johns Hopkins University Presented at MIT GraphEx: April 16, 2013 Vertex Nomina-on fusion