TODO. LLOD and corpora

Transcription

1 Ch. Chiarcos, ACoLi CO, 2016, July 22 LLOD and corpora If you have Un*x environment (Linux, BSD, Mac, Cygwin): - make sure you have JAVA installed and wifi works - lab members: svn checkout /intern/incubator/conll-rdf - others: download and unzip others: - find a neighbor who does TODO

2 LLOD and corpora 1. Linguistic Linked Open Data and interoperability challenges wrt. corpora 2. State of the Art: NIF & POWLA... and why neither is used in NLP nor corpus linguistics 3. Yet another format: CoNLL-RDF Breaking the usage barrier? 4. Working with CoNLL-RDF

3 Linked Open Data RDF, RDF vocabularies, Linking

4 Resource Description Framework (RDF) W3C standard (1999) generic data model: directed labeled graph nodes, edges, labels originally developed to provide metadata about resources e.g., journals in a bookstore and ebooks in an online shop resources are unambiguously identified in the web of data by Uniform Resource Identifiers URIs) 4

5 Resource Description Framework (RDF) a (labeled directed multi-) graph nodes ( RDF resources ) anything we want to provide information about edges ( RDF properties ) assigns a source node ( subject ) a target node ( object ) or a value ( literal ) nodes and edges are unambiguously identified Uniform Resource Identifiers (URIs), e.g., URLs 5

6 RDF thesaurus:bronstijd rdf:type thesaurus:period. thesaurus: bronstijd rdf:type thesaurus: period (the concept) bronstijd is an (instance of concept) period 6

7 RDF thesaurus:bronstijd rdf:type thesaurus:period. thesaurus: bronstijd rdf:type thesaurus: period abbreviated for URI < could be opened in a browser resolvable URIs may provide further information 7

8 RDF thesaurus:bronstijd rdf:type thesaurus:period. thesaurus:bronstijd skos:preflabel thesaurus: bronstijd rdf:type skos:preflabel thesaurus: period in German (de), the preferred label for (the concept) bronstijd is Bronzezeit 8

9 RDF thesaurus:bronstijd rdf:type thesaurus:period. thesaurus:bronstijd skos:preflabel thesaurus:bronstijd skos:preflabel "bronze thesaurus: bronstijd rdf:type thesaurus: period skos:preflabel skos:preflabel bronze in English (en), the preferred label for (the concept) bronstijd is bronze age 9

10 RDF thesaurus:bronstijd rdf:type thesaurus:period. thesaurus:bronstijd skos:preflabel thesaurus:bronstijd skos:preflabel "bronze thesaurus:bronstijd skos:preflabel thesaurus: bronstijd rdf:type thesaurus: period skos:preflabel skos:preflabel skos:preflabel bronze in English (en), the preferred label for (the concept) bronstijd is bronstijd 10

11 RDF thesaurus:bronstijd rdf:type thesaurus:period. thesaurus:bronstijd skos:preflabel thesaurus:bronstijd skos:preflabel "bronze thesaurus:bronstijd skos:preflabel triple notation (Turtle) thesaurus: bronstijd rdf:type thesaurus: period graphical notation skos:preflabel skos:preflabel skos:preflabel bronze 11

12 RDF Querying thesaurus:bronstijd rdf:type thesaurus:period. thesaurus:bronstijd skos:preflabel thesaurus:bronstijd skos:preflabel "bronze thesaurus:bronstijd skos:preflabel PREFIX skos: < SELECT?de?en WHERE { }?a skos:preflabel?de.?a skos:preflabel?en. FILTER(langMatches(lang(?de), de")) FILTER(langMatches(lang(?en), en )) SPARQL SQL meets Turte triple notation (Turtle) SPARQL 12

13 RDF Querying thesaurus:bronstijd rdf:type thesaurus:period. thesaurus:bronstijd skos:preflabel thesaurus:bronstijd skos:preflabel "bronze thesaurus:bronstijd skos:preflabel triple notation (Turtle) PREFIX skos: < SELECT?de?en WHERE { }?a skos:preflabel?de.?a skos:preflabel?en. FILTER(langMatches(lang(?de), de")) FILTER(langMatches(lang(?en), en )) triples with variables FILTERS with XPath-like functions SPARQL 13

14 RDF Querying thesaurus:bronstijd rdf:type thesaurus:period. thesaurus:bronstijd skos:preflabel thesaurus:bronstijd skos:preflabel "bronze thesaurus:bronstijd skos:preflabel triple notation (Turtle) thesaurus: bronstijd rdf:type thesaurus: period graphical notation skos:preflabel skos:preflabel skos:preflabel bronze => list with two cols 14

15 RDF RDF graphs can represent arbitrarily complex structures, can be freely extended with additional nodes, links (edges pointing to external resources), etc. thesaurus: arch_concept thesaurus: bronstijd rdf:type thesaurus: period rdfs:subclassof skos:preflabel skos:preflabel skos:preflabel bronze (every) period is an arch_concept period is subclass of arch_concept 15

16 RDF vocabularies Specialized vocabularies for different kinds of information do exist and can (and should) be re-used => uniform information representation thesaurus: arch_concept thesaurus: bronstijd rdf:type thesaurus: period rdfs:subclassof skos:preflabel skos:preflabel skos:preflabel RDF und RDF Schema (RDFS) basic vocabulary for taxonomies bronze 16

17 RDF vocabularies Specialized vocabularies for different kinds of information do exist and can (and should) be re-used Additional vocabularies (SKOS, OWL, etc.) extended vocabulary for taxonomies and ontologies thesaurus: bronstijd rdf:type thesaurus: period thesaurus: arch_concept rdfs:subclassof skos:preflabel skos:preflabel skos:preflabel bronze 17

18 RDF vocabularies Specialized vocabularies for different kinds of information do exist Domain-specific vocabularies can be specified as required thesaurus: bronstijd rdf:type thesaurus: period thesaurus: arch_concept rdfs:subclassof skos:preflabel skos:preflabel thesaurus bronze toy vocabulary for archeology 18

19 RDF vocabularies graphical browsers and editors e.g. kos-play (SKOS) d.edu/ (RDF, OWL, SKOS) 19

20 RDF vocabularies graphical browsers and editors do exist as well as data bases, W3C-standardized query language, APIs, reasoners, usw. 20

21 RDF vocabularies RDF resources and edges are defined by URIs as a URL, they may be accessed remotely re-usable vocabularies & knowledge bases link to community-maintained data types/terms instead of defining your own Linked Open Data Cloud (LOD) a great variety of resources that are linked with each other and released under an open license 21

22 Linked Data Linked Data rules of best practice for publishing data on the web use URIs as names for things (1) links to external URIs (links) allow us to retrieve more information from these sites if they can be resolved via HTTP (2) and provide information as RDF* (3) and they include links to other URIs (4) then, this is Linked Data (informal) 22

23 Linked Data Linked Data rules of best practice for publishing data on the web => Information integration Structural interoperability comparable formats and protocols to access data => the same query language for different data sets 23

24 Linked Data Linked Data rules of best practice for publishing data on the web => Information integration Structural interoperability Conceptual interoperability develop and (re-)use a shared vocabularies for equivalent concepts => the same query on different data sets 24

25 Linked Data Linked Data rules of best practice for publishing data on the web => Information integration Structural interoperability Conceptual interoperability Federation data published on the web with a query interface (SPARQL end point) => a single query to query different datasets simultaneously 25

26 Linked Open Data (LOD) 26

27 LOD cloud: Aug 2014 media linguistic LOD bibliography government data geo information social networks life sciences 27

28 Linking Corpora with other schemes with terminology repositories with lexical-semantic networks corpora as Linked Open Data => network effects

29 Structurally Interoperable Language Resources NLP Interchange Format (NIF): Interoperability for NLP pipelines 29

30 Linked Data Corpus Creation with NIF (NIF Reference Card) Best practices to follow for the generation of Linked Data text corpora, using the NLP Interchange Format (NIF). Target audience Corpus creators and users seeking to make corpora interoperable and to publish them as linked data. Basic knowledge of RDF is mandatory for conversion. Basic knowledge of linked data and web server access is needed for publication. Scope Conversion of existing corpora into RDF using NIF, as well as creation of linked data corpora from textual data. Website: Github: Example corpus: 30

31 Core concepts Corpus We understand a corpus as a collection of documents. Documents contain text, represented as strings of characters and annotations that provide more information about these strings. NIF provides a way to identify strings using URIs and annotate them using an ontology. String identification via URI: Strings are identified using a URI scheme consisting of: the prefix of the corpus URI; the character indices of beginning and end of the string; and a scheme identifier between document URI and string position identifier. Character indices in NIF are counted offset based, starting at zero before the first character and counting the gaps between the characters until after the last character of the referenced string: This URI scheme is valid for text/plain. Other mime types may require different URI schemes. String annotation After assigning URIs to meaningful strings of the corpus, these URIs can be annotated using the NIF core ontology. Website: Github: Example corpus: 31

32 Example String URIs Strings are the basis of analysis nif:anchorof => string value < <#char=3,12> < Context Contains document text in nif:isstring nif:beginindex is always 0 Strings refer to Context with nif:referencecontext Website: Github: Example corpus: 32

33 Example Pre-defined categories Word, Phrase, Sentence, Paragraph rdfs:subclassof String hierarchy => substring Pre-defined properties head, lemma, stem, postag, oriented towards industrial applications primarily used for Entity Linking (taldentref) limited in scalability limited support of dependency annotations, no support for relational semantics 33

34 Example Information Integration Annotations are attached to strings Implicit unification of divergent annotations If different tools annotate the same string, this refers to the same URI 34

35 NIF Core ontology Rich vocabulary OWL-based redundant properties transitive inverse before ~ previousword 35

36 NIF and computational linguistics NIF is used in EU projects community projects (e.g., Apache Stanbol) industrial applications Why not in NLP and linguistics? ignorance Standard data formats I'm not sure these are important: if I someone can use a parser, they can probably also write a Python wrapper Mark Johnson (2012), Computational Linguistics. Where do we go from here?, invited plenary talk at the 50th Annual Meeting of the ACL, Jeju 37

37 NIF and computational linguistics NIF is used in EU projects community projects (e.g., Apache Stanbol) industrial applications Why not in NLP and linguistics? ignorance readability human-readable? CoNLL The DT _ Semantic Web NNP is VB _ a DT _ good JJ _ idea NN _ < a nif:string, nif:phrase, nif:offsetbasedstring ; nif:anchorof "Semantic Web"@en ; nif:beginindex "4"^^xsd:int ; nif:endindex "16"^^xsd:int ; nif:olialink < ; itsrdf:taidentref < ; nif:referencecontext < NIF 38

38 NIF and computational linguistics NIF is used in EU projects community projects (e.g., Apache Stanbol) industrial applications Why not in NLP and linguistics? ignorance readability limited expressivity non-string annotations / empty strings? non-phrasal MWEs? hard-wired properties and concepts limited to morphosyntax, NER and dependency syntax 39

39 NIF and computational linguistics NIF is used in EU projects community projects (e.g., Apache Stanbol) industrial applications Why not in NLP and linguistics? ignorance readability limited expressivity established formats for basic annotations Do we get anything that CoNLL-TSV doesn t give us yet? 40

40 NIF and computational linguistics NIF is used in EU projects community projects (e.g., Apache Stanbol) industrial applications Why not in NLP and linguistics? ignorance readability limited expressivity established formats for basic annotations developed with a focus on Semantic Web applications neither linguistics nor NLP 41

41 NIF Alternatives TELIX motivation similar as NIF RDFa annotations, added to XML documents popularity decreases with the popularity of XML OpenAnnotation intended for expressing metadata over content elements in HTML limited support for linguistic annotations 42

42 NIF Alternatives POWLA RDF/OWL reconstruction of an XML standoff format capable to represent any annotation faithfully comes from a small and specialized sub-community in NLP and linguistics (multi-layer corpora, discourse annotation) as unreadable as the original XML standoff format but better to process 43

43 CoNLL-RDF Yet another corpus formalism

44 Technical motivations for corpora in RDF use off-the-shelf technologies for data storing (RDF triple/quad stores), querying (SPARQL), manipulation (SPARQL Update), and access (SPARQL end points) structurally interoperable with NLP output (NIF) and other RDF-based corpus formats, dictionaries (lemon), and terminology bases => flexible information integration 45

45 CoNLL-RDF RDF-based formalism rather than OWL-based (NIF, POWLA) minimal: no transitive and inverse properties generic: no pre-defined categories nor properties Grounded in an established and widely used formats in the field tab-separated values: CoNLL format family comfortable import and export to human-readable representations 46

46 47

47 48

48 CoNLL-X format (2006) 49

49 Tab-Separated Values 50

50 CoNLL-2009 additions: SRL 51

51 CoNLL-U: Universal Dependencies 52

52 Tool-specific CoNLL variants: SENNA 53

53 CoNLL formats comments begin with # sentences separated by empty line one word per line annotations separated by tab (=> columns) empty columns left empty, contain -, _ or O HEAD points to another word in the same sentence foreign key, identified by sentential position/id all other annotations assign string values 54

54 CoNLL-RDF generate URIs word: base URI + s + sentence nr +. + word ID sentence: word with ID 0 ( CoNLL root) base URI should refer to the original document in a corpus, e.g., 55

55 CoNLL-RDF generate URIs word: base URI + s + sentence nr +. + word ID supply labels for all columns datatype property in conll namespace or conll:<colname> 56

56 CoNLL-RDF generate URIs word: base URI + s + sentence nr +. + word ID supply labels for all columns datatype property in conll namespace special treatment of HEAD object property pointing to head URI 57

57 CoNLL-RDF generate URIs word: base URI + s + sentence nr +. + word ID supply labels for all columns datatype property in conll namespace special treatment of HEAD object property pointing to head URI minimal use of NIF vocabulary nif:word, nif:nextword, nif:nextsentence 58

58 CoNLL-RDF generate URIs word: base URI + s + sentence nr +. + word ID supply labels for all columns datatype property in conll namespace special treatment of HEAD object property pointing to head URI minimal use of NIF vocabulary conventionally formatted to resemble CoNLL 59

59 CoNLL vs. CoNLL-RDF... 60

60 CoNLL-RDF processing three simple java programs svn:/intern/incubator/conll-rdf/ CoNLL2RDF basic converter read CoNLL file, write Turtle 61

61 CoNLL-RDF processing three simple java programs svn:/intern/incubator/conll-rdf/ CoNLL2RDF basic converter CoNLLStreamExtractor graph manipulation read CoNLL file sentence by sentence, for every sentence, apply SPARQL Update statements, write Turtle 62

62 CoNLL-RDF processing three simple java programs svn:/intern/incubator/conll-rdf/ CoNLL2RDF basic converter CoNLLStreamExtractor graph manipulation CoNLLRDFFormatter basic visualization read CoNLL-RDF, output: CoNLL-like visualization special treatment of depencencies limited to conll namespace coloring on Unix shells imposes some naming conventions 63

63 CoNLL-RDF processing three simple java programs svn:/intern/incubator/conll-rdf/ CoNLL2RDF basic converter CoNLLStreamExtractor graph manipulation CoNLLRDFFormatter basic visualization anything beyond this is handled by SPARQL update queries modular pipeline (examples in *.sh) re-usable modules (given appropriate documentation) maybe hard to read, but easy to write your own! 64

64 An example Requires a Unix shell Please look into the code! 65

65 An example: acoli-example.sh 66

66 An example: acoli-example.sh read one or several files if you have one file only, say FILE.conll, write cat FILE.conll \ instead 67

67 An example: acoli-example.sh call CoNLLStreamExtractor (run.sh just adds the classpath) column labels in the order they occur in the CoNLL file base URI (can be any URL, etc.) 68

68 An example: acoli-example.sh do some layouting $* means that command-line arguments are passed to CoNLLRDFFormatter but these are not necessary 69

69 An example: acoli-example.sh do some layouting Please, try for yourself $* means that command-line arguments are passed to CoNLLRDFFormatter but these are not necessary 70

70 Manipulating the data Between reading and writing, manipulations can be applied to the graph. u => next are some SPARQL Update queries these queries are read from the files in the argument optionally, a number of iterations can be supplied in {0} 71

71 sparql/remove-id.sparql For more complicated manipulations, we also require insertions Use INSERT {... } before DELETE 72

72 shift-reduce/initialize-shift.sparql This is from another pipeline, see shift-reduceexample.sh 73

73 Task: Write a SPARQL update query Given the UD annotations write a query that extracts the main verb (predicate) of a sentence add this information to the graph?verb a conll:predicate. add this query to acoli-example.sh and run it use this as a template. 74

74 Example, extended Linking with OLiA 75

75 Linking Corpora OLiA External Reference Models (Terminology Repositories) (Schmidt et al. 2006, Chiarcos 2008, 2010, 2012) OLiA Annotation Models GOLD ISOcat (morphosyntax) OntoTag (morphosyntax) Reference Model STTS TIGER Penn TüBa- D/Z Susanne German Connexor Brown etc. Annotation Models for German English Linking: given a POWLA individual i if annotations of i match OLiA annotation model specs TDS ontology MULTEXT/ MULTEXT/ MULTEXT/ East East East EAGLES EAGLES EAGLES 11 European languages 15 (mostly) Eastern European languages then declare i an instance of the corresponding OLiA class

76 Integrating external information link English POS / dependencies with OLiA 77

77 sparql/link-penn-pos-simple.sparql 78

78 Task II: Extend this such that?a is also an instance of superclasses of?concept 79

79 Example, further extended Consult an online dictionary 80

80 Getting German glosses for English text SPARQL permits accessing endpoints and web services on the web in the SELECT part of the query SERVICE <URL> {... } SPARQL permits accessing endpoints and web services on the web e.g. DBnary, an RDF edition of various Wiktionaries 81

81 Exploring an end point What datasets are in here? There seems to be a German DBnary 82

82 Exploring an end point What does the German dictionary contain? lemon:lexicalentry may be what we re looking for 83

83 Exploring an end point Examples for LexicalEntries Ok, let s look into Abenteuer 84

84 Exploring an end point Examples for LexicalEntries Indeed, this contains dbnary:istranslationof 85

85 Exploring an end point targetlanguage => language dbnary:writtenform => String Check an example translation 86

86 Explore an end point etc., until we have pairs of strings from DBnary 87

87 Explore an end point 88

88 Adding German translations sparql/gloss-en-to-de-dbnary.sparql 89

89 Sample output 90

90 Task III: return a word-by-word German translation (and nothing else) hints: {... } UNION {...} => logical or FILTER(NOT EXISTS {... }) => negation 91