KNOWLEDGE GRAPH CONSTRUCTION

Transcription

1 KNOWLEDGE GRAPH CONSTRUCTION Jay Pujara University of Maryland, College Park Max Planck Institute 7/9/2015

2 Can Computers Create Knowledge? Internet Massive source of publicly available information Knowledge

3 Computers + Knowledge =

4 What does it mean to create knowledge? What do we mean by knowledge?

5 Defining the Questions Extraction Representation Reasoning and Inference

6 Defining the Questions Extraction Representation Reasoning and Inference

7 A Revised Knowledge-Creation Diagram Internet Extraction Knowledge Graph (KG) Massive source of publicly available information Cutting-edge IE methods Structured representation of entities, their labels and the relationships between them

8 Knowledge Graphs in the wild

9 Motivating Problem: Real Challenges Internet Extraction Knowledge Graph Difficult! Noisy! Contains many errors and inconsistencies

10 NELL: The Never-Ending Language Learner Large-scale IE project (Carlson et al., AAAI10) Lifelong learning: aims to read the web Ontology of known labels and relations Knowledge base contains millions of facts

11 Examples of NELL errors

12 Entity co-reference errors Kyrgyzstan has many variants: Kyrgystan Kyrgistan Kyrghyzstan Kyrgzstan Kyrgyz Republic

13 Missing and spurious labels Kyrgyzstan is labeled a bird and a country

14 Missing and spurious relations Kyrgyzstan s location is ambiguous Kazakhstan, Russia and US are included in possible locations

15 Violations of ontological knowledge Equivalence of co-referent entities (sameas) SameEntity(Kyrgyzstan, Kyrgyz Republic) Mutual exclusion (disjointwith) of labels MUT(bird, country) Selectional preferences (domain/range) of relations RNG(countryLocation, continent) Enforcing these constraints requires jointly considering multiple extractions across documents

16 Examples where joint models have succeeded Information extraction ER+Segmentation: Poon & Domingos, AAAI07 SRL: Srikumar & Roth, EMNLP11 Within-doc extraction: Singh et al., AKBC13 Social and communication networks Fusion: Eldardiry & Neville, MLG10 Acts: Carvalho & Cohen, SIGIR05 GraphID: Namata et al., KDD11

17 GRAPH IDENTIFICATION

18 Slides courtesy Getoor, Namata, Kok Transformation Graph Identification Input Graph Available but inappropriate for analysis Output Graph Appropriate for further analysis

19 Slides courtesy Getoor, Namata, Kok Motivation: Different Networks Neil Smith Mary Taylor Robert Lee Anne Cole Mary Jones Label: CEO Manager Assistant Programmer Communication Network Nodes: Address Edges: Communication Node Attributes: Words Organizational Network Nodes: Person Edges: Manages Node Labels: Title

20 Slides courtesy Getoor, Namata, Kok Graph Identification Neil Smith Graph Iden+fica+on Mary Taylor Robert Lee Anne Cole Mary Jones Label: CEO Manager Assistant Programmer Input Graph: Communication Network Output Graph: Social Network

21 Slides courtesy Getoor, Namata, Kok Graph Identification Graph Iden+fica+on Input Graph: Communication Network Output Graph: Social Network What s involved?

22 Slides courtesy Getoor, Namata, Kok Graph Identification Neil Smith ER Mary Taylor Robert Lee Anne Cole Mary Jones Input Graph: Communication Network Output Graph: Social Network What s involved? Entity Resolution (ER): Map input graph nodes to output graph nodes

23 Slides courtesy Getoor, Namata, Kok Graph Identification Neil Smith ER+LP Mary Taylor Robert Lee Anne Cole Mary Jones Input Graph: Communication Network Output Graph: Social Network What s involved? Entity Resolution (ER): Map input graph nodes to output graph nodes Link Prediction (LP): Predict existence of edges in output graph

24 Slides courtesy Getoor, Namata, Kok Graph Identification Neil Smith ER+LP+NL Mary Taylor Robert Lee Anne Cole Mary Jones Label: CEO Manager Assistant Programmer Input Graph: Communication Network Output Graph: Social Network What s involved? Entity Resolution (ER): Map input graph nodes to output graph nodes Link Prediction (LP): Predict existence of edges in output graph Node Labeling (NL): Infer the labels of nodes in the output graph

25 Slides courtesy Getoor, Namata, Kok Problem Dependencies ER LP Input Graph NL Most work looks at these tasks in isolation In graph identification they are: Evidence-Dependent Inference depend on observed input graph e.g., ER depends on input graph Intra-Dependent Inference within tasks are dependent e.g., NL prediction depend on other NL predictions Inter-Dependent Inference across tasks are dependent e.g., LP depend on ER and NL predictions

26 KNOWLEDGE GRAPH IDENTIFICATION Pujara, Miao, Getoor, Cohen, ISWC 2013 (best student paper)

27 (Pujara et al., ISWC13) Motivating Problem (revised) (noisy) Extraction Graph Knowledge Graph Internet Large-scale IE Joint Reasoning =

28 (Pujara et al., ISWC13) Knowledge Graph Identification Problem: Knowledge Graph Extraction Graph Performs graph identification: entity resolution node labeling link prediction Knowledge Graph Identification Solution: Knowledge Graph Identification (KGI) Enforces ontological constraints Incorporates multiple uncertain sources =

29 Illustration of KGI: Extractions (Pujara et al., ISWC13) Uncertain Extractions:.5: Lbl(Kyrgyzstan, bird).7: Lbl(Kyrgyzstan, country).9: Lbl(Kyrgyz Republic, country).8: Rel(Kyrgyz Republic, Bishkek, hascapital)

30 (Pujara et al., ISWC13) Illustration of KGI: Ontology + ER Uncertain Extractions:.5: Lbl(Kyrgyzstan, bird).7: Lbl(Kyrgyzstan, country).9: Lbl(Kyrgyz Republic, country).8: Rel(Kyrgyz Republic, Bishkek, hascapital) Extraction Graph Kyrgyzstan Lbl bird country Kyrgyz Republic Rel(hasCapital) Bishkek

31 (Pujara et al., ISWC13) Illustration of KGI: Ontology + ER Uncertain Extractions:.5: Lbl(Kyrgyzstan, bird).7: Lbl(Kyrgyzstan, country).9: Lbl(Kyrgyz Republic, country).8: Rel(Kyrgyz Republic, Bishkek, hascapital) Ontology: Dom(hasCapital, country) Mut(country, bird) Entity Resolution: SameEnt(Kyrgyz Republic, Kyrgyzstan) (Annotated) Extraction Graph SameEnt Kyrgyzstan Lbl country bird Kyrgyz Republic Dom Rel(hasCapital) Bishkek

32 (Pujara et al., ISWC13) Illustration of KGI Uncertain Extractions:.5: Lbl(Kyrgyzstan, bird).7: Lbl(Kyrgyzstan, country).9: Lbl(Kyrgyz Republic, country).8: Rel(Kyrgyz Republic, Bishkek, hascapital) Ontology: Dom(hasCapital, country) Mut(country, bird) Entity Resolution: SameEnt(Kyrgyz Republic, Kyrgyzstan) (Annotated) Extraction Graph SameEnt Kyrgyzstan Lbl country bird Kyrgyz Republic Dom Rel(hasCapital) Bishkek After Knowledge Graph Identification Kyrgyzstan Lbl country Kyrgyz Republic Rel(hasCapital) Bishkek

33 (Pujara et al., ISWC13) Modeling Knowledge Graph Identification

34 (Pujara et al., ISWC13) Viewing KGI as a probabilistic graphical model Lbl(Kyrgyzstan, bird) Rel(hasCapital, Kyrgyzstan, Bishkek) Lbl(Kyrgyzstan, country) Lbl(Kyrgyz Republic, country) Lbl(Kyrgyz Republic, bird) Rel(hasCapital, Kyrgyz Republic, Bishkek)

35 (Pujara et al., ISWC13) Background: Probabilistic Soft Logic (PSL) (Broecheler et al., UAI10; Kimming et al., NIPS-ProbProg12) Templating language for hinge-loss MRFs, very scalable! Model specified as a collection of logical formulas SameEnt(E 1,E 2 ) ^ Lbl(E 1,L) ) Lbl(E 2,L) Uses soft-logic formulation Truth values of atoms relaxed to [0,1] interval Truth values of formulas derived from Lukasiewicz t-norm p ^q = max(0,p+ q 1) p _q =min(1,p+ q) p =1 p p )q =min(1,q p + 1)

36 Soft Logic Tutorial: Rules to Groundings Given a database of evidence, we can convert rule templates to instances (grounding) Rules are grounded by substituting literals into formulas SameEnt(E 1,E 2 ) ^ Lbl(E 1,L) ) Lbl(E 2,L) SameEnt(Kyrgyzstan, Kyrygyz Republic) ^ Lbl(Kyrgyzstan, country) ) Lbl(Kyrygyz Republic, country) The soft logic interpretation assigns a satisfaction value to each ground rule

37 Soft Logic Tutorial: Groundings to Satisfaction SameEnt(Kyrgyzstan, Kyrygyz Republic) : 0.9 ^ Lbl(Kyrgyzstan, country) : 0.8 p _q = max(0,p+ q 1) SameEnt(Kyrgyzstan, Kyrygyz Republic) ^ Lbl(Kyrgyzstan, country) = max(0, )

38 Soft Logic Tutorial: Groundings to Satisfaction (SameEnt(Kyrgyzstan, Kyrygyz Republic) ^ Lbl(Kyrgyzstan, country)) : 0.7 ) Lbl(Kyrygyz Republic, country) : 0.6 p )q =min(1,q p + 1) SameEnt(Kyrgyzstan, Kyrygyz Republic) ^ Lbl(Kyrgyzstan, country) ) Lbl(Kyrygyz Republic, country) =min(1, ) = 0.9

39 Soft Logic Tutorial: Inferring Satisfaction (SameEnt(Kyrgyzstan, Kyrygyz Republic) ^ Lbl(Kyrgyzstan, country)) : 0.7 ) Lbl(Kyrygyz Republic, country) :?

40 Soft Logic Tutorial: Distance to Satisfaction

41 (Pujara et al., ISWC13) Background: PSL Rules to Distributions Rules are grounded by substituting literals into formulas w EL : SameEnt(Kyrgyzstan, Kyrygyz Republic) ^ Lbl(Kyrgyzstan, country) ) Lbl(Kyrygyz Republic, country) Each ground rule has a weighted distance to satisfaction derived from the formula s truth value P(G E) = 1 Z exp $ % r R w r ϕ (G)& r ' The PSL program can be interpreted as a joint probability distribution over all variables in knowledge graph, conditioned on the extractions

42 (Pujara et al., ISWC13) Background: Finding the best knowledge graph MPE inference solves max G P(G) to find the best KG In PSL, inference solved by convex optimization Efficient: running time empirically scales with O( R ) (Bach et al., NIPS12)

43 (Pujara et al., ISWC13) PSL Rules for KGI Model

44 (Pujara et al., ISWC13) PSL Rules: Uncertain Extractions Weight for source T (relations) Predicate representing uncertain relation extraction from extractor T Relation in Knowledge Graph w CR-T : CandRel T (E 1,E 2,R) w CL-T : CandLbl T (E,L) ) Rel(E 1,E 2,R) ) Lbl(E,L) Weight for source T (labels) Predicate representing uncertain label extraction from extractor T Label in Knowledge Graph

45 (Pujara et al., ISWC13) PSL Rules: Entity Resolution SameEnt predicate captures confidence that entities are co-referent Rules require co-referent entities to have the same labels and relations Creates an equivalence class of co-referent entities

46 (Pujara et al., ISWC13) PSL Rules: Ontology Inverse: w O : Inv(R, S) ^ Rel(E 1,E 2,R) ) Rel(E 2,E 1,S) Selectional Preference: w O : Dom(R, L) ^ Rel(E 1,E 2,R) ) Lbl(E 1,L) w O : Rng(R, L) ^ Rel(E 1,E 2,R) ) Lbl(E 2,L) Subsumption: w O : Sub(L, P ) ^ Lbl(E,L) ) Lbl(E,P) w O : RSub(R, S) ^ Rel(E 1,E 2,R) ) Rel(E 1,E 2,S) Mutual Exclusion: w O : Mut(L 1,L 2 ) ^ Lbl(E,L 1 ) ) Lbl(E,L 2 ) w O : RMut(R, S) ^ Rel(E 1,E 2,R) ) Rel(E 1,E 2,S) Adapted from Jiang et al., ICDM 2012

47 [ 1] CandLbl struct (Kyrgyzstan, bird) ) Lbl(Kyrgyzstan, bird) φ 1 φ φ [ 2] CandRel pat (Kyrgyz Rep., Asia, locatedin) ) Rel(Kyrgyz Rep., Asia, locatedin) Lbl(Kyrgyzstan, bird) Lbl(Kyrgyz Rep., bird) [ 3] SameEnt(Kyrgyz Rep., Kyrgyzstan) ^ Lbl(Kyrgyz Rep., country) ) Lbl(Kyrgyzstan, country) φ 5 φ [ 4] Dom(locatedIn, country) ^ Rel(Kyrgyz Rep., Asia, locatedin) ) Lbl(Kyrgyz Rep., country) Lbl(Kyrgyzstan, country) Lbl(Kyrgyz Rep., country) [ 5] Mut(country, bird) ^ Lbl(Kyrgyzstan, country) ) Lbl(Kyrgyzstan, bird) φ φ 2 φ 3 φ 4 Rel(Kyrgyz Rep., Asia, locatedin) φ

48 Probability Distribution over KGs P(G E) = 1 Z exp $ % r R w r ϕ (G)& r ' CandLbl T (kyrgyzstan, bird) ) Lbl(kyrgyzstan, bird) Mut(bird, country) ^ Lbl(kyrgyzstan, bird) ) Lbl(kyrgyzstan, country) SameEnt(kyrgz republic, kyrgyzstan) ^ Lbl(kyrgz republic, country) ) Lbl(kyrgyzstan, country)

49 Evaluation (Pujara et al., ISWC13)

50 (Pujara et al., ISWC13) Two Evaluation Datasets Description LinkedBrainz Community-supplied data about musical artists, labels, and creative works NELL Real-world IE system extracting general facts from the WWW Noise Realistic synthetic noise Imperfect extractors and ambiguous web pages Candidate Facts 810K 1.3M Unique Labels and Relations Ontological Constraints K

51 (Pujara et al., ISWC13) LinkedBrainz Open source communitydriven structured database of music metadata Uses proprietary schema to represent data Built on popular ontologies such as FOAF and FRBR Widely used for music data (e.g. BBC Music Site) LinkedBrainz project provides an RDF mapping from MusicBrainz data to Music Ontology using the D2RQ tool

52 (Pujara et al., ISWC13) LinkedBrainz dataset for KGI mo:release mo:record mo:record mo:track mo:track mo:published_as mo:signal mo:label mo:label foaf:maker mo:musicalartist inverseof subclassof subclassof foaf:made mo:solomusicartist mo:musicgroup Mapping to FRBR/FOAF ontology DOM rdfs:domain RNG rdfs:range INV owl:inverseof SUB rdfs:subclassof RSUB rdfs:subpropertyof MUT owl:disjointwith

53 (Pujara et al., ISWC13) LinkedBrainz experiments Comparisons: Baseline PSL-EROnly PSL-OntOnly PSL-KGI Use noisy truth values as fact scores Only apply rules for Entity Resolution Only apply rules for Ontological reasoning Apply Knowledge Graph Identification model AUC Precision Recall F1 at.5 Max F1 Baseline PSL-EROnly PSL-OntOnly PSL-KGI

54 NELL Evaluation: two settings (Pujara et al., ISWC13) Target Set: restrict to a subset of KG (Jiang, ICDM12) Complete: Infer full knowledge graph?? Closed-world model Uses a target set: subset of KG Derived from 2-hop neighborhood Excludes trivially satisfied variables Open-world model All possible entities, relations, labels Inference assigns truth value to each variable

55 (Pujara et al., ISWC13) NELL experiments: Target Set Task: Compute truth values of a target set derived from the evaluation data Comparisons: Baseline Average confidences of extractors for each fact in the NELL candidates NELL Evaluate NELL s promotions (on the full knowledge graph) MLN Method of (Jiang, ICDM12) estimates marginal probabilities with MC-SAT PSL-KGI Apply full Knowledge Graph Identification model Running Time: Inference completes in 10 seconds, values for 25K facts AUC F1 Baseline NELL MLN (Jiang, 12) PSL-KGI

56 (Pujara et al., ISWC13) NELL experiments: Complete knowledge graph Task: Compute a full knowledge graph from uncertain extractions Comparisons: NELL NELL s strategy: ensure ontological consistency with existing KB PSL-KGI Apply full Knowledge Graph Identification model Running Time: Inference completes in 130 minutes, producing 4.3M facts AUC Precision Recall F1 NELL PSL-KGI

57 KNOWLEDGE GRAPH ENTITY RESOLUTION

58 [Pujara, BayLearn14] Problem: Merge domain KG to global KG

59 [Pujara, BayLearn14] Approach: Factored Entity Resolution model Goal: Build a generic entity resolution model for KGs Build on vast amount of work on Entity Resolution PSL provides an easy, flexible, sophisticated models Local Collective General String similarity Sparsity; Transitivity New Entity New Entity prior New Entity penalty Knowledge Graph Type compatibility Relation compatibility Domain-Specific (Album length) (Artist s country)

60 [Pujara, BayLearn14] Preliminary Results Task: ER from MusicBrainz to Google KG Data: 11K MusicBrainz entities (5/5-6/29/14) 330K Freebase entities 15.7M relations 11K human labels Methods F1 AUPRC General Collective NewEntity

61 FASTER KNOWLEDGE GRAPH CONSTRUCTION

62 Partitioning

63 (Pujara et al., AKBC13) Problem: Knowledge Graphs are HUGE

64 (Pujara et al., AKBC13) Solution: Partition the Knowledge Graph

65 (Pujara et al., AKBC13) Partitioning: advantages and drawbacks Advantages Smaller problems Parallel Inference Speed / Quality Tradeoff Drawbacks Partitioning large graph time-consuming Key dependencies may be lost New facts require re-partitioning

66 (Pujara et al., AKBC13) Key idea: Ontology-aware partitioning Partition the ontology graph, not the knowledge graph Location City Mut Dom citysportsteam Inv teamplaysincity locatedin Rng Dom teamplayssport SportsTeam State Rng Rng Sport Induce a partitioning of the knowledge graph based on the ontology partition

67 (Pujara et al., AKBC13) Considerations: Ontology-aware Partitions Advantages: Ontology is a smaller graph Ontology coupled with dependencies New facts can reuse partitions Disadvantages: Insensitive to data distribution All dependencies treated equally

68 (Pujara et al., AKBC13) Refinement: include data frequency Annotate each ontological element with its frequency Location 10 citysportsteam City 1706 Inv teamplaysincity 822 Mut locatedin teamplayssport State 1177 Dom Rng Rng Dom SportsTeam Rng Sport 2568 Partition ontology with constraint of equal vertex weights

69 (Pujara et al., AKBC13) Refinement: weight edges by type Weight edges by their ontological importance Location City Mut 3 Dom citysportsteam Inv 116 teamplaysincity locatedin Rng 116 Dom teamplayssport SportsTeam State Rng Rng Sport

70 (Pujara et al., AKBC13) Experiments: Partitioning Approaches Comparisons (6 partitions): NELL Default promotion strategy, no KGI KGI No partitioning, full knowledge graph model baseline KGI, Randomly assign extractions to partition Ontology KGI, Edge min-cut of ontology graph O+Vertex KGI, Weight ontology vertices by frequency O+V+Edge KGI, Weight ontology edges by inv. frequency AUPRC Running Time (min) Opt. Terms NELL KGI M baseline M Ontology M O+Vertex M O+V+Edge M

71 Evolving Models

72 Problem: Incremental Updates to KG? How do we add new extractions to the Knowledge Graph?

73 Naïve Approach: Full KGI over extractions

74 Improving the naïve approach Intuition: Much of previous KG does not change Online collective inference: Selectively update the MAP state Bound the regret of partial updates Efficiently determine which variables to infer

75 Key Idea: fix some variables, infer others

76 Approximation: KGI over subset of graph

77 Theory: Regret of approximating update R n (x, y S ; ẇ) apple O s Bkwk 2 n w p ky S ŷ S k 1!

78 Practice: Regret and Approximation Algo inference regret Do Nothing Random 50% Value 50% WLM 50% Relational 50% # epochs

79 Conclusion Knowledge Graph Identification is a powerful technique for producing knowledge graphs from noisy IE system output Using PSL we are able to enforce global ontological constraints and capture uncertainty in our model Unlike previous work, our approach infers complete knowledge graphs for datasets with millions of extractions Code available on GitHub:

80 Key Collaborators