Whole World OLIF. Version 3.0 of the Versatile Language Data Format

Transcription

1 Whole World OLIF Version 3.0 of the Versatile Language Data Format Christian Lieske, SAP Language Services, SAP AG Susan McCormick, PhD, Linguistic Consultant tekom Annual Conference, November 2007 Caveat: This presentation pertains to the current draft for OLIF 3.0; the actual OLIF 3.0 may differ

2 Authors Christian Lieske Knowledge Architect Susan McCormick Linguistic Consultant Natural Language Processing (term extraction and checking, controlled language authoring, ) Content Engineering and Processing (including evaluation, prototyping and piloting) Contributor to W3C, OASIS and OLIF activities Degree in Computer Science with focus on Natural Language Processing and Artificial Intelligence Natural Language Processing development (MT parsing/semantics; text-to-speech; lexicography) Lexical/terminology management, standards development (SAP; OTELO; OLIF) Currently lead linguistic developer for OLIF; coordinator of OLIF activities PhD in Linguistics from Cornell University tekom Annual Conference 2007 / Page 2

3 Agenda 1. Language Data 2. OLIF in a Nutshell 3. Version Enhancements to the Format 3.2. Broader Tool Support 3.3. User Base 4. Conclusions/Outlook tekom Annual Conference 2007 / Page 3

4 Agenda 1. Language Data 2. OLIF in a Nutshell 3. Version Enhancements to the Format 3.2. Broader Tool Support 3.3. User Base 4. Conclusions/Outlook tekom Annual Conference 2007 / Page 4

5 Language Data Scenarios reference definition human use Machine Translation System tool use generate glossary Quality assurance in technical documentation XML Editor author Terminology Management System generate Terminology Extraction Markup-based Extraction tekom Annual Conference 2007 / Page 5

6 Language Data Observations Different purposes data import/export migration between tools data validation avoidance of redundancy Different content terminological data, different for each system lexicographical data, different for each system varying complexity (possibly even incompatibility) Different formats anything from CSV to highly complex XML vocabularies tekom Annual Conference 2007 / Page 6

7 Agenda 1. Language Data 2. OLIF in a Nutshell 3. Version Enhancements to the Format 3.2. Broader Tool Support 3.3. User Base 4. Conclusions/Outlook tekom Annual Conference 2007 / Page 7

8 OLIF Bird s-eye-view Versatile, open format for language data XML-based Specification consists of XML schema and prose document Sensitive to other formats and standards Lexical and terminological data for human and machine use Basic data as well as more complex cases such as MT lexicons Options for modeling administrative, morphological, syntactic and semantic data Specific support for 6 European languages, and 1 Asian language tekom Annual Conference 2007 / Page 8

9 OLIF Evolution and Drivers OLIF Prototype developed in EC-funded OTELO project OLIF Version 2 additional input from tool vendors and other language standards OLIF Version 2.1 five categories of changes (e.g. switch to XML Schema Definition) OLIF Version 3.0 Watch this show Caveat: This presentation pertains to the current draft for OLIF 3.0; the actual OLIF 3.0 may differ OLIF is driven by a group of language technology companies and organizations interested in issues related to language data OLIF information is available from the OLIF2 Consortium web site tekom Annual Conference 2007 / Page 9

10 The OLIF File The header: provides meta-data The body: carries entries, the real payload tekom Annual Conference 2007 / Page 10

11 The (Basic) OLIF Entry Canonical form Language Part of speech Subject field Semantic reading (optional) tekom Annual Conference 2007 / Page 11

12 The (General) OLIF Entry Monolingual data Cross-references (optional) Transfers/translations (optional) tekom Annual Conference 2007 / Page 12

13 OLIF Myths and Truths OLIF is only for machine-use In fact, OLIF can be written by authors and consumed by readers OLIF is only for lexicology OLIF is only for Machine Translation OLIF is only for some languages In fact, OLIF supports two important paradigms for organizing language data (concept- and lemma-orientation) In fact, OLIF is for any application which needs language data as resources In fact, OLIF is a general linguistic and representation model tekom Annual Conference 2007 / Page 13

14 Agenda 1. Language Data 2. OLIF in a Nutshell 3. Version Enhancements to the Format 3.2. Broader Tool Support 3.3. User Base 4. Conclusions/Outlook tekom Annual Conference 2007 / Page 14

15 Agenda 1. Language Data 2. OLIF in a Nutshell 3. Version Enhancements to the Format 3.2. Broader Tool Support 3.3. User Base 4. Conclusions/Outlook tekom Annual Conference 2007 / Page 15

16 Single-Sourcing Support Use original data in OLIF without conversion 1. Markup allowed in several elements (e.g. definition ) 2. Recognizable proprietary values allowed in several places (e.g. related to subject field information) tekom Annual Conference 2007 / Page 16

17 Initial Support for W3C ITS Standard Use the Internationalization Tag Set (ITS) from the World Wide Web Consortium to enhance internationalization localization/translation OLIF experimentally provides an ITS rules file allows ITS markup tekom Annual Conference 2007 / Page 17

18 Specific Support for Japanese Use data categories and values targeted at Japanese Morphological Syntactic Semantic Stylistic OLIF support for Japanese is modeled after the JMdict Japanese source lexical format ( tekom Annual Conference 2007 / Page 18

19 Encoding of Capitalization/Case Variation Encode capitalization variants Type Scope allowed or disallowed term bank level vs. Individual terms Context beginning of sentence, Class all uppercase, tekom Annual Conference 2007 / Page 19

20 Enhanced Usability for End-Users Ensure correct data entry information (e.g. inflection types) for the 7 languages with specific support usability enhancement for German inflection codes tekom Annual Conference 2007 / Page 20

21 Enhanced Usability for Engineers Tailor OLIF to your needs subset OLIF for a specific purposemorphological schema modified to allow reliable creation of OLIF subsets tekom Annual Conference 2007 / Page 21

22 Agenda 1. Language Data 2. OLIF in a Nutshell 3. Version Enhancements to the Format 3.2. Broader Tool Support 3.3. User Base 4. Conclusions/Outlook tekom Annual Conference 2007 / Page 22

23 acrolinx OLIF is central to terminology component of checking, and the term harvesting/extraction capabilities Among the uses of OLIF for acrolinx are: terminology import terminology export output from term harvesting (bidirectional) conversions related to other formats (acrolinx internal AXT, TBX, MTF, plain text) tekom Annual Conference 2007 / Page 23

24 SDL Technologies OLIF is supported as import format for the SDL terminology management solution The conversion plug-in developed for SDL MultiTerm 2007 (version 7.5) allows conversion of bilingual OLIF files into MultiTerm XML format. is available to any MultiTerm 2007 customer has already enabled corporate customers to turn data sets of more than 100,000 OLIF entries into MultiTerm databases tekom Annual Conference 2007 / Page 24

25 Other Tool Support (1/2) 1. Several companies have implemented exports from their termbases 2. MT developers have been creating converters 3. Students have worked on applications such as a Keyword Enhancement Service (KES) creates meaningful metadata for a given resource enhances a list of keywords by using OLIF converts expanded list to standard format to produce enhanced document metadata

26 Other Tool Support (2/2) 4. The user community has developed tools e.g. for general editing removal of noise (from term extraction) Implementation approaches vary configuration or resources (e.g. stylesheet) proprietary programs tekom Annual Conference 2007 / Page 26

27 Agenda 1. Language Data 2. OLIF in a Nutshell 3. Version Enhancements to the Format 3.2. Broader Tool Support 3.3. User Base 4. Conclusions/Outlook tekom Annual Conference 2007 / Page 27

28 European Patent Office Machine Translation (MT) for patents/abstracts/communications to/from English MT system chosen by a public call for tender further call for tender addressed the need for patent-specific MT dictionaries bilingual term extraction was chosen as approach for semi-automatic creation of dictionaries Decision for OLIF due to features such as Openness XML and Unicode orientation Support for features of MT engines (e.g. conditional translations) tekom Annual Conference 2007 / Page 28

29 SAP AG Central terminology database (SAPterm) more than 30 languages, approximately 2.5 million terms, glossary definitions, 850 different domains based on SAP technology, single sourcing environment for a variety of lexicon and dictionary offshoots. OLIF serves as a bridge between SAPterm and derived resources MT lexicons, acrolinx technology. glossaries for User Assistance/Online Help term lookup tools and termbases used by translation tools tekom Annual Conference 2007 / Page 29

30 Community The Yahoo Group for OLIF currently has approx. 120 users serves as a medium for exchange allows discussion of changes enables sharing of data group/olifconsortium/ The OLIF Web site counts thousands of hits per month serves as the official source for all OLIF material, incl. specifications and supporting documentation tekom Annual Conference 2007 / Page 30

31 Agenda 1. Language Data 2. OLIF in a Nutshell 3. Version Enhancements to the Format 3.2. Broader Tool Support 3.3. User Base 4. Conclusions/Outlook tekom Annual Conference 2007 / Page 31

32 Conclusions Enhanced world readiness Whole World OLIF Stronger support from tool vendors and uptake by users OLIF is open if something isn't right, get involved to fix it Thank you tekom Annual Conference 2007 / Page 32

33 Disclaimer All product and service names mentioned and associated logos displayed are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary. This document may contain only intended strategies, developments, and is not intended to be binding upon the authors or their employers to any particular course of business, product strategy, and/or development. The authors or their employers assume no responsibility for errors or omissions in this document. The authors or their employers do not warrant the accuracy or completeness of the information, text, graphics, links, or other items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. The authors or their employers shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitation shall not apply in cases of intent or gross negligence. The authors have no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages tekom Annual Conference 2007 / Page 33