Building the Multilingual Web of Data. Integrating NLP with Linked Data and RDF using the NLP Interchange Format

Transcription

1 Building the Multilingual Web of Data Integrating NLP with Linked Data and RDF using the NLP Interchange Format Presenter name 1

2 Outline 1. Introduction 2. NIF Basics 3. NIF corpora 4. NIF tools & services 2

3 Introduction 3

4 LOD-aware NLP services Not only data, but also LOD-aware services using: Lexica and dictionaries in RDF (lemon model) Training data for NLP in RDF (NIF model) Service metadata descriptions in RDF Combination with real world facts (i.e. DBpedia or Geoname) Long Term Goal index of tools and data users can easily produce ready-made, preconfigured NLP services and pipelines freemium / pay-per use business models 4

5 NLP2RDF project Realize this long-term goal Sustainably maintain and consolidate results from short-term projects Bootstrap the Eco-System 5

6 NLP2RDF: NLP Interchange Format The NLP Interchange Format (NIF) is an RDF/OWL-based format that aims to achieve interoperability between Natural Language Processing (NLP) tools, language resources and annotations. 6

7 NLP2RDF: NLP Interchange Format In a nutshell: Way to mint URIs for arbitrary strings and content of documents on the web 7

8 NLP2RDF: NLP Interchange Format In a nutshell: Way to mint URIs for arbitrary strings and content of documents on the web Logical formalisation of strings and annotations via an ontology 8

9 NLP2RDF: NLP Interchange Format In a nutshell: Way to mint URIs for arbitrary strings and content of documents on the web Logical formalisation of strings and annotations via an ontology Quick and easy format 9

10 NLP2RDF: NLP Interchange Format In a nutshell: Way to mint URIs for arbitrary strings and content of documents on the web Logical formalisation of strings and annotations via an ontology Quick and easy format Builds on existing standards, e.g. RDF, LAF/GrAF, RFC

11 NLP2RDF: NLP Interchange Format In a nutshell: Way to mint URIs for arbitrary strings and content of documents on the web Logical formalisation of strings and annotations via an ontology Quick and easy format Builds on existing standards, e.g. RDF, LAF/GrAF, RFC 5147 Reusability of RDF tools and implementation 11

12 NLP2RDF: NLP Interchange Format In a nutshell: Way to mint URIs for arbitrary strings and content of documents on the web Logical formalisation of strings and annotations via an ontology Quick and easy format Builds on existing standards, e.g. RDF, LAF/GrAF, RFC 5147 Reusability of RDF tools and implementation Decreases development cost for integration 12

13 NIF: Motivation Developers nightmare: All tools belong to similar class of NLP tools Fulfill similar functions but often not interoperable ecosystems 13

14 NIF: Motivation Developers nightmare: All tools belong to similar class of NLP tools Fulfill similar functions but often not interoperable ecosystems All have: Heterogeneous output formats (JSON, XML, ) 14

15 NIF: Motivation Developers nightmare: All tools belong to similar class of NLP tools Fulfill similar functions but often not interoperable ecosystems All have: Heterogeneous output formats (JSON, XML, ) Heterogeneous API parameters 15

16 NIF: Motivation Developers nightmare: All tools belong to similar class of NLP tools Fulfill similar functions but often not interoperable ecosystems All have: Heterogeneous output formats (JSON, XML, ) Heterogeneous API parameters Heterogeneous way of annotating text: 16

17 NIF: Motivation Developers nightmare: All tools belong to similar class of NLP tools Fulfill similar functions but often not interoperable ecosystems All have: Heterogeneous output formats (JSON, XML, ) Heterogeneous API parameters Heterogeneous way of annotating text: Some remove HTML internally, offsets not usable 17

18 NIF: Motivation Developers nightmare: All tools belong to similar class of NLP tools Fulfill similar functions but often not interoperable ecosystems All have: Heterogeneous output formats (JSON, XML, ) Heterogeneous API parameters Heterogeneous way of annotating text: Some remove HTML internally, offsets not usable Some use byte offset instead of char offset 18

19 Outline 1. Introduction 2.NIF Basics 3. NIF corpora 4. NIF tools & services 19

20 NIF architecture 20

21 NIF architecture 21

22 NIF architecture 22

23 NIF architecture 23

24 NIF architecture 24

25 NIF architecture 25

26 NIF architecture 26

27 NIF architecture 27

28 NIF architecture 28

29 NIF architecture 29

30 NIF architecture 30

31 Annotations 31

32 Annotations 32

33 Annotations 33

34 Annotations 34

35 Example: Tripadvisor Corpus Corpus contains hotel reviews and review metadata 1760 semi-structured files In NIF: every file's content becomes one nif:context resource Strings in the file can be adressed via URIs 35

36 Context 36

37 Context Adress the content of a document nif:isstring contains document content 37

38 Context Adress the content of a document nif:isstring contains document content Note that in NIF the document is!= content of the document two different documents can have the same content => must not have the same URI 38

39 other Strings 39

40 other Strings Adress arbitrary strings in the document Use string offsets in relation to context to adress strings nif:anchorof contains the string Additional properties can now be added to the string 40

41 other Strings a tripadvisor:review ; Adress arbitrary strings in the document Use string offsets in relation to context to adress strings nif:anchorof contains the string Additional properties can now be added to the string 41

42 Words and Phrases Sentiment values, POS tags and other annotations etc can now be added to words and phrases 42

43 String counting 43

44 Ontology 44

45 Ontology 45

46 Ontology 46

47 Ontology 47

48 Ontology 48

49 Demo: 49

50 Demo: 50

51 Outline 1. Introduction 2. NIF Basics 3.NIF corpora 4. NIF tools & services 51

52 NIF corpora overview Name Size (Triple) Wikilinks 500M News K RSS K Reuters-128 7K Spotlight 3K KORE50 2K Brown 500K URL Wikipedia abstract corpus in progress : Tag nif on datahub 52

53 Wikilinks corpus Overview Large scale coreference resolution corpus by Umass / Google Over 10M crawled websites that contain text (Named Entities) that link to Wikipedia Converted to the NIF format and published as LOD: Additional processing done to extract relevant text snippets, add Dbpedia ontology classes and coarse-grained classes (entity types) Over 500 million triples, 79GB LOD, 12GB gzipped dumps Over 30 million links to over 3 million entities 53

54 Brown corpus Overview Converted to the NIF format and published as LOD: Corpus showcases handling of POS tags in NIF POS tags: mapped via OliA to predefined categories <#char=643,647> a nif:string, nif:word, nif:rfc5147string ; nif:anchorof "Jury"^^xsd:string ; nif:referencecontext <#char=0,> ; nif:olialink brown:nn ; nif:sentence <#char=619,777> ; nif:beginindex "643"^^xsd:nonNegativeInteger ; nif:endindex "647"^^xsd:nonNegativeInteger. Categories can be used to query all resources of a certain POS regardless of tagset used in the corpus 54

55 Brown corpus POS tags 55

56 Brown corpus POS tags Querying all nouns using the OliA mapping 56

57 Brown corpus POS tags Querying all nouns using the OliA mapping 57

58 Outline 1. Introduction 2. NIF Basics 3. NIF corpora 4.NIF tools & services 58

59 NIF tools Available NIF tools : Stanford Core NLP OpenNLP RDFace Validator ConLL converter... 59

60 NIF tools: DBpedia Spotlight 60

61 NIF tools: Stanford Core 61

62 NIF tools: Stanford Core 62

63 NIF tools: Stanford Core 63

64 NIF tools: Stanford Core 64

65 NIF tools: Stanford Core 65

66 NIF tools: Stanford Core 66

67 NIF tools: Stanford Core 67

68 Done! Thank you very much!