Joe Raad [1,2], Wouter Beek [3], Frank van Harmelen [3], Nathalie Pernelle [2], Fatiha Saïs [2]

Transcription

1 DETECTING ERRONEOUS IDENTITY LINKS IN THE WEB OF DATA Joe Raad [1,2], Wouter Beek [3], Frank van Harmelen [3], Nathalie Pernelle [2], Fatiha Saïs [2] [1] INRA, Paris France [2] LRI, Orsay France [3] VU, Amsterdam Netherlands

2 MOTIVATION

3 5 LINKED OPEN DATA make your data available on the Web make it available as structured data make it available in a non-proprietary format use open standards from the W3C link your data to other data Tim Berners-Lee, 2010

4 WHY LINKING YOUR DATA? My Pharmacy Knowledge Graph mykg:drug33 rdfs:label "Dolasetron" ; mykg:treats mykg:nausea ; mykg:location geonames:paris. Freebase Knowledge Graph freebase:m.02jq1 freebase:objectname "Dolasetronum" ; freebase:objecttype freebase:medicine. Ok Google... where to find a medicine for Nausea in Paris?

5 HOW TO LINK YOUR DATA? owl:sameas (the Semantic Web Identity Predicate) x, owl:sameas, y means that: x = y ( P)(Px Py) there is one thing which has two names: x and y

6 "THE SAMEAS PROBLEM" (1/2) Like the WWW, the Semantic Web contains a number of incorrect (identity) statements Oversimplified heuristic generation Classical notion of identity is problematic Lack of formal standardized alternatives Concept dri of identity links targets

7 "THE SAMEAS PROBLEM" (2/2) The largest identity set contains 177,794 terms that 'should' refer to the same real world entity However: full list at:

8 HOW CAN WE DETECT ERRONEOUS SAMEAS LINKS? Source Trustworthiness [Cudre-Mauroux et al. 2009] UNA or Ontology Axioms Violation [de Melo 2013; Valdestilhas et al. 2017; Hogan et al. 2012; Papaleo et al. 2014] Content-based [Acosta et al. 2013; Paulheim et al. 2014; Cuzzola et al.,2015] Network Metrics [Guéret et al. 2012; Sarasua et al.,2017]

9 REQUIREMENTS High accuracy and recall Scalable to the LOD Tested on real world data No (or minimal) assumptions on the data (e.g. UNA, textual description, source trustworthiness) (No existing approach combines all these criteria)

10 APPROACH & EXPERIMENTS

11 OVERALL IDEA Use the community structure of the network containing solely owl:sameas links to assign an error degree for each link 4 MAIN STEPS

12 DATASET LOD-a-lot dataset: 28.3B unique triples [Fernandez et al. 2017]

13 1. EXTRACT THE EXPLICIT IDENTITY STATEMENTS prefix owl: < select distinct?s?p?o { bind (owl:sameas?p)?s?p?o } OUTPUT: 558.9M owl:sameas (179.7M terms)

14 2. PARTITION TO EQUALITY SETS :a owl:sameas :b :a owl:sameas :c :c owl:sameas :a :d owl:sameas :e :f owl:sameas :f Eq Set 1 Eq Set 2 OUTPUT: 48.9M Non-Singleton Equality Sets [Beek et al. 2018]

15 'BARACK OBAMA' EQUALITY SET These 440 identifiers denote the same thing (EqSet 5723)

16 3. DETECT THE COMMUNITY STRUCTURE IN EACH EQ SET We use the Louvain algorithm Detects non-overlapping communities Adapted to weighted networks Low computational complexity Outperforms other algorithms [Lancichinetti and Fortunato. 2009; Yang et al. 2016] [Blondel et al. 2008]

17 4. ASSIGN ERROR DEGREES Intra-Community Link Inter-Community Link Between 0 and 1 based on the weight of the link and the density of the community(ies)

18 COMMUNITIES - 'BARACK OBAMA' C0: person; C1: president; C2: government; C3: senator

19 EVALUATION

20 ERROR DEGREE DISTRIBUTION OF 556M OWL:SAMEAS

21 FINDING THE THRESHOLD (1/3) MANUAL EVALUATION OF 200 SAMEAS LINKS 1. All evaluated links with an error degree < 0.4 are correct

22 FINDING THE THRESHOLD (2/3) MANUAL EVALUATION OF 200 SAMEAS LINKS 2. The higher an error degree is, the more likely an owl:sameas link is erroneous

23 FINDING THE THRESHOLD (3/3) MANUAL EVALUATION OF 60 SAMEAS WITH ERR > Links with an err > 0.99 and belonging to large equality sets are more likely to be incorrect

24 EVALUATION / ACCURACY (1/2) Erroneous sameas: err > 0.99 (threshold) TP TN FP FN Total Table Table 'Obama' EqSet Total % Accuracy TP + TN Total

25 EVALUATION / ACCURACY (2/2) Erroneous sameas: err > 0.99 (threshold) 95 correct DBpedia-Freebase owl:sameas 78 / 95 links exist in our dataset 77 / 78 links have an err 0.99 (error degree ranging from 0.52 to 0.94) 98.7% Accuracy [Acosta et al. 2013]

26 EVALUATION / RECALL Manually chose 40 random different terms (e.g. dbr:facebook, dbr:strawberry, dbr:chair) Made sure they are not explicitly sameas (but some which are in the same equality set) Added all the possible 780 links between them 93% Recall 725/780 have an error degree > 0.99 (error degree ranging from 0.87 to )

27 ERROR DEGREES Intra-Community Link Inter-Community Link [0, 1] based on the weight of the link and the density of the community(ies)

28 SYMMETRY IMPACT (1/2) 1. Symmetrical owl:sameas links are more likely to be correct

29 SYMMETRY IMPACT (2/2) DISCARDED THE WEIGHT FROM THE ERROR DEGREE Manual evaluation of 30 sameas links with error degree > in the largest equality set Only 2 erroneous owl:sameas (17 correct and 11 can't tell) Accuracy drops from 88% to 11% (in the largest equality set)

30 WHO IS MESSING UP THE LOD? C0: person; C1: president; C2: government; C3: senator

31 WHO IS MESSING UP THE LOD? freebase:m.05b6w1g owl:sameas dbr:president_barack_obama freebase:m.05b6w1g owl:sameas dbr:president_obama freebase:m.05b6w1g freebase:object.name "Presidency of Barack Both owl:sameas links have are error degree of the only two links in the 'Obama' equality set with err > 0.99

32 HOW MESSED UP IS THE LOD? Our approach can give an approximation based on: 1. nbr of symmetrical sameas (450M with 98% chance of correctness) 2. nbr of non-symmetrical sameas with err degree 0.99 (105M with a 88.6% chance of correctness) 3. nbr of non-symmetrical sameas with err degree >0.99 (1.2M with an erroneous probability between 40 and 88%) We estimate that 4% of owl:sameas links in the LOD are erroneous (around 22.5M links) previous estimations: 2.8% by [Hogan et al. 2012] and 21% by [Halpin et al. 2010]

33 CONCLUSION & PERSPECTIVES

34 SUMMARY Approach for detecting erroneous identity links Uses the sameas network's community structure No assumptions on the data Scale to the Web (total runtime of 11 hours) High Accuracy (86%) and Recall (93%) Links, Error Degrees, and Eq Sets:

35 PERSPECTIVES Combine multiple community detection algorithms Consider the Equality Set size in the error degree Include duplicate owl:sameas triples Combine with state-of-the-art approaches Replace erroneous owl:sameas links with contextual identity links [Beek et al. 2016; Raad et al. 2017]

36 THANK YOU! J.Raad, W.Beek, F.van Harmelen, N.Pernelle, and F.Saïs Detecting Erroneous Identity Links on the Web using Network Metrics, ISWC W.Beek, J.Raad, J.Wielemaker, and F.van Harmelen sameas.cc: The Closure of 500M owl:sameas Statements, ESWC. 2018