NLP resources: construc.on, standardiza.on, exploita.on & API. Karim Bouzoubaa

Transcription

1 NLP resources: construc.on, standardiza.on, exploita.on & API Karim Bouzoubaa

2 outline Exploita.on NLP resources Construc.on Standardiza.on API

3 Exploita.on

4 Exploitation LRs are used in various NLP so7ware tools: morphological, and analysis of texts spell- checking and paraphrasing search and text mining 4

5 outline Exploita.on NLP resources Construc.on Standardiza.on API

6 NLP Resources

7 Resources Introduction Definition Types Examples Evaluation criteria

8 Introduc.on - Defini.on q The key to NLT development is the Language Resource q Resource produc@on takes a lot of effort and is very expensive Example: The Arabic standard LC- STAR phone@c lexicon of the European Linguis@c Resource Associa@on (ELRA) with 110,271 entries costs EUR (for use in academic research) Language resources are language-related data, accessible in an electronic format, and used for the development of NLP systems 8

9

10 Types 2 categories 1. Corpus writen: monolingual texts, mul@lingual texts, annoted texts, treebanks speech: reading texts aloud, speeches, dialogues, radio and television broadcasts Mul@media: images, sounds and videos 2. Lexicon monolingual and mul@lingual Dic@onaries Gaze@ers (geographical dic@onary) Terminologies ontologies

11 Content of a lexicon An entry in the lexicon may contain : morphological, syntac@c, seman@c and pragma@c informa@on the gramma@cal category (noun, verb, etc.), o subcategory proper@es (transi@ve verb or not, masculine or feminine) seman@c informa@on (animated name, verb requiring a human subject

12 Examples 12

13 Oxford dic.onary

14 verbnet

15 criteria q Formal (regardless of content) Size Maintenance (durability, scalability) Compa@bility q Func.onal (language criteria) Lexicographic annota@on (existence and relevance) Intrinsic rules

16 outline Exploita.on NLP resources Construc.on Standardiza.on API

17 Construc.on

18 Produc.on cycle resources Example (Contempory Arabic) Reusing ressources Example of free resources Good prac.ces Interoperability Viability

19 crea.ng resources two approaches for developing LRs: q creating new resources q tuning existing resources 19

20 crea.ng resources Collect data, of a general nature or belonging to a par@cular sector of ac@vity, directly in digital form or, in some cases, by digi@zing them. 20

21 Example of creating resources Contemporary Arabic

22 Resources Reuse q The opera@on of making changes to a resource for the purpose of performing certain func@ons and improving it in a different usage environment from the original one q Example:... 22

23 Corpus q q q q Lexicon q Example of free resources Corpus of Contemporary Arabic Khoja POS tagged corpus Quranic Arabic Collec@on of free arabic texts and books: - Almeshkat - Al- Eman Buckwalter s list of Arabic roots q Al- Baheth Al- Arabi 23

24 Good In order to contribute to the of a set of sustainable RLs, some principles must be respected: Resource documenta@on Interoperability of resources 24

25 Documenta.on of resources LRs are o7en poorly documented or undocumented at all. should be as comprehensive as possible, and include on: the format of the data the content of the data the context the possible uses 25

26 Resources interoperability q The interoperability of LRs is the ability to operate in different systems q The formats of the LRs must be standard 26

27 Interoperability documentation - reuse Many difficul@es are encountered when reusing available LRs

28 Interoperability documentation - reuse Contribute to the development of LRs respec@ng interoperability rules Availability Portability Reusability normaliza@on

29 outline Exploita.on NLP resources Construc.on Standardiza.on API

30 Standardiza.on

31 why? q How to integrate exis@ng resources into one's own contexts? q How to separate the resources from the tools that manage them?

32 Panorama standardisation agencies: CNIS: China National Institute of Standardization FNOR: Agence Française de Normalisation DIN: Deutsches Institut für Normung ANSI: American National Standards Institute W3C: World Wide Web Consortium TEI: Text Encoding Initiative ISO: the International Organization for Standardization projects: LIRICS :Linguistic Infrastructure for Interoperable Resources and Systems EAGLES: Expert Advisory Group on Language Engineering Standards Multext : Multilingual Text Tools and Corpora research structures: CLARIN: Common Language Resources and Technology Infrastructure FLaReNet : Fostering Language Resources Network Alpage : Analyse Linguistique Profonde A Grande Echelle.

33 Organization

34 standards proposition Préliminaire Proposition Preliminary Work Item (PWI) Préparatoire new project of the WG New Work Item Proposal (NP) Commission Committee Draft (CD) Enquête Approbation Final Draft International Standard (FDIS) Draft International Standard (DIS) Publication International Standard (IS)

35 LMF Modeling Arabic paradigms according to the LMF standard Aïda Khemakhem et al conversion of editorial to LMF Feten Baccar et al. 2008, Aïda Khemakhem et al Domain ontology from LMF Feten Baccar et al Proposed standardized of standard Arabic lexicons Susanne Salmon- Alt et al 2013 of anomalies and of the content of LMF Wafa WALI et al of a system of produc@on of Arabic dic@onaries respec@ng the LMF standard Mohammed Reqqass et al. 2014

36 LMF Example

37 LMF Example

38 TEI <TEI> <teiheader> <name> NAFIS Arabic Stemming Gold Standard</name>... </teiheader> <text> <phr> ععللييككمم ببااللججدد ففا إننهه ا أسسااسس< val > < val />االلننججااحح <"ععللييككمم"= rend w> <choice n="14"> <seg> <m type="prefix"></m> <form type="base"> m> < m />ععلليي<" type="root m> < m />عع لل يي<" type="stem </form> m> < m />ككمم<" type="suffix </seg> <seg> <m type="prefix"></m> <form type="base"> m> < m />ععلليي<" type="root m> < m />ععلل يي <" type="stem </form> m> < m></seg />ككمم<" type="suffix... </choice> </w> </phr>... </text> </TEI>