Internationalization of uportal Overview of Internationalization & Localization

Transcription

1 Internationalization of uportal Overview of Internationalization & Localization Once upon a time, in the dim, primordial past, software could only "speak" one human language at a time. Each country or region needed it's own version. In some cases, internationally relevant features were retrofitted onto the shipping English language product. In other cases, the English and international versions might be completely separate products, perhaps sharing some basic code but often sharing little more than the product's name. In either case, whether making such changes retroactively to a shipping product or developing parallel versions, the result was usually a lengthy, expensive product cycle. Even if a piece of software was functionally suitable for use in more than one country, producing an appropriate language version of that product was difficult. This is because the menus, dialog boxes and messages which make up that product's user interface (UI) were often written directly in the program source code. Successfully translating source files can be quite difficult, expensive and time-consuming for a number of reasons. It requires a combination of linguistic and engineering knowledge that is not commonly available; mistakes are easy to make; and the translated code often does not function as expected, because its authors made assumptions about things like the language of the UI, the length or position of English words (which change when translated to other languages), etc. Luckily, many software developers have realized that if they design their products from the beginning to understand the requirements of multi-locale computing, these products will reach global markets sooner, for less expense, and probably be much more successful than products designed the "old fashioned way." Such multi-locale products can be called "globally enabled". Globally enabled software is software that supports a wide range of languages, human cultural conventions, fonts, encoding and other features that make it useful, not just in one country or region, but around the world. Additionally, the user interface for globally enabled products is separate from the core instruction code, allowing the software to be translated without requiring recompilation. Since globally enabled software doesn't make assumptions about language of the user interface, the translated programs are more robust, requiring less "fixing" or special enhancements to support individual languages. This speeds the release of translated programs. The uportal is in the process of being globally enabled. The core uportal framework will require minor adjustments to meet this goal. The user interface is rendered by transformations (XSLT) and, for the most part, completely separate from the framework. Having the UI disconnected from the framework in this way means uportal doesn t need multiple versions of the framework, just language libraries for the transformations. The remainder of this document is intended to convey enough information so that we can continue to make uportal a globally enabled project, and can be aware of some of the complexity of this process. As we move forward, the goal is to enable uportal to support many more languages and other globally relevant feature sets. While we pursue this endeavor, let s keep the following in mind: One code base for the world, and English is just another language.

2 Definitions Before proceeding, let's establish some definitions of the major terms and concepts used herein: Internationalization: (a.k.a. Globalization, a.k.a. Enabling) Designing and developing a software product to function in multiple locales. This process involves identifying the locales that must be supported, designing features which support those locales, and writing code/transformations that functions equally well in any of the supported locales. Localization: Modifying or adapting product to fit the requirements of a particular locale. This process includes (but may not be limited to) translating the user interface, documentation and packaging, changing UI geometries, customizing features (if necessary), and testing the translated product to ensure that it still works (at least as well as the original). Localizability: The degree to which a software product can be localized. Localizable products separate data from code, correctly display the target language and function properly after being localized. i18n: Acronym for "internationalization" ("i" + 18 letters + "n"; lower case i is used to distinguish it from the numeral 1 (one)). L10n: Acronym for "localization" ("L" + 10 letters + "n"; upper case L is used to distinguish it from the numeral 1 (one)). L12y: Acronym for "localizability" ("L" + 12 letters + "y"; upper case L is used to distinguish it from the numeral 1 (one)). Locale: A set of conventions affected or determined by human language and customs, as defined within a particular geo-political region. These conventions include (but are not necessarily limited to) the written language, formats for dates, numbers and currency, sorting orders, etc. Resource: 1) Any part of a project, which can appear to the user or be changed or configured by the user. 2) Any piece of the projects data, as opposed to it's code. Core product - framework: The language independent portion of a project (as distinct from any particular localized version of that project) Current Discussion of i18n Enhancements of uportal The following discussion took place at the Educause conference in Atlanta. The topics are at the same time both specific to an individual need, and general enough to benefit the entire project. A preliminary task list is included as an outcome of the meeting. Desired function: allow uportal users to select one language preference for the entire portal rendering. Desired function: allow uportal users to select a language preference for individual channels that would override any global language preference. Outcome: 1) Use the java.util.locale class (language and country codes) to define a language preference. See 2) Add interfaces and classes to retrieve a set of possible locales. The reference implementation could read from a new UP_LOCALE table with the columns: Justin Tilton: instructional media + magic, inc page 2 of 9

3 LANGUAGE_CODE, COUNTRY_CODE, SUPPORTED (true or false) 3) Add two properties to portal.properties: [locale_aware=true (or false)], [default_locale=en_us, (or ja_jp, sv_se, etc.)]. The locale_aware property lets the portal know whether or not it should make any considerations for supporting multiple languages. For example, should it prompt channel publishers for channel titles in more than one language? The default_locale property would specify a string which defines the default language and country code for the particular portal installation. 4) Add a GUI to the preferences area that lets a user choose from a list of available (supported) locales to be used as the default locale for all of his/her channels. The storage for this preference must have plugable implementations since some schools might want to store it in the portal db while others will want to store it in LDAP. 5) Enhance channel publishing so that it asks the publisher to enter channel names, titles, descriptions, etc for one or more locales. The locale information for a published channel will require a new database column: UP_CHANNEL.CHAN_LOCALE. The values for that column would have the format LANGUAGECODE_COUNTRYCODE. Examples: en_us, ja_jp. 6) Add new attribute to the channel element in userlayout.dtd called locale. Example: <channel name="cartoon of the day" locale="en_us"... /> 7) Add method to ChannelRuntimeData: [public java.util.locale getlocale();] 8) Change character encoding in construction of portal serializer by adding "UTF-8" 9) Add new option in user preferences to set a locale for a particular channel instance which overrides the default locale for a layout. Store the channel instance locale as either a stylesheet attribute or another column in in UP_LAYOUT_STRUCT. 10) Come up with "best practices" for a channel to manage data for multiple locales. For example, any text prompts or labels can be stored in ResourceBundles. Also, XSLT stylesheets could be created for each locale and stored in a locale-based directory structure. Alternatively, locale-based portfixes could be appended to the end of each stylesheet name such as main_en_us.xsl and main_ja_jn.xsl. 11) Make a utiltity that takes the locale into consideration to find the right XSLT stylesheet. Enabling the portal The most important aspect will be to ensure that the data can be processed correctly regardless of language. Transformations of the data must take into account locale specific requirements: Extended characters (non-ascii) must be rendered appropriately (including ligatures, composite characters, direction of writing horizontal and vertical, etc.) Users must be able to input text in any supported language, and the results must be processed and stored without corruption Users should be able to select multiple language outputs based on a preference hierarchy Numbers (and numbered lists etc.) are generated correctly Ordered lists and tables are are sorted according to the rules of a particular locale Different languages displayed in the same UI are each displayed properly and consistently Text is wrapped properly (even if the language does not use spaces) The textual content of the portal can be searched and indexed correctly regardless of the Justin Tilton: instructional media + magic, inc page 3 of 9

4 language Automated text (quotation marks, captions, etc.) are generated according to the locale of the text Numbers, dates, and times are formatted according to the rules of a locale Use a system of cross-referenced translations to facilitate re-use Localization of the portal The second aspect of i18n, assuming the enabling above is done, is to ensure the content can be localized. Terminology for the translations is established Special requirements such as maximum length or a forced locale or subset of characters are identified A list of the components to be localized (stylesheets, cpd files, etc.) is created, updated, and tracking possibly automated into a localization kit with some administration tools Methods developed to test the localization in progress Analysis A tool to scope the process of localization of a selected set of content would be useful (i.e. an administration tool that could do things like phrase count, word count, etc. to provide a means for estimating the task) Preparation A tool to build a packaged list of content to be localized (i.e. an administration tool that would compile a list of the components/channels, etc. that needed to be localized, and a status for each in the process) Translation and Editing A tool to enable simple translation and editing of content whereby the linguists do not have to be familiar with the underlying structure of the uportal i18n implementation. (i.e. an administration tool that might present the phrase in one of the languages already in the library and allow the linguist to simply select the language they are translating to, and a enter the translated text. The tool would be responsible for writing the content back to the library in the appropriate XML format). Language Challenges Languages are often grouped based on rendering properties: character set, writing direction, etc. Additional issues are equally important: punctuation and marks, cultural issues for graphics and icons, abbreviations, collation and sorting, word separation, digits, date and time, etc. Encoding XML In the last 10 years two large character sets have been developed: ISO and Unicode. These two sets effectively merged their efforts with the release of the 1.1 version of Unicode in Version 3.2 of the Unicode standard was released in March of The set of characters defined by Unicode is usually referred to as the Universal Character Set (UCS). The Justin Tilton: instructional media + magic, inc page 4 of 9

5 UCS is organized in blocks grouped by symbol collections and scripts. A script is the set of characters needed to write a given set of languages. For example the Latin script is used to write English, French, German, and many others. Cyrillic is used for Russian, Bulgarian, Ukrainian, and others. UTF The Unicode Transformation Format (UTF) 16 bit encoding form (UTF-16) and the UTF-8 encoding form, are two means to the same end. Each identifies a code point (think character). All characters in Unicode are represented by a 16 bit code even Latin characters that could be represented in 7-bit codes. Therefore, when text is composed of ASCII only characters and encoded in UTF-16, the file will be twice as large as an equivalent document encoded in ASCII. Enter UTF-8 a multi-byte 8-bit encoding in which each code point is mapped to a sequence of one to four bytes (the main advantage of this being compatibility with ASCII). There is no difference between a document encoded in ASCII and one encoded with UTF-8. Since the XML tags found in the uportal are almost always represented with ASCII characters, and make up a large part of the documents total size, It is recommended the uportal i18n project use UTF-8 as the encoding. Special Aspects of Rendering There are a few specific issues of rendering that pertain directly to internationalization. Most a related to how text is organized; writing direction, character layout, contextual shaping, text wrapping, and ordering. Writing Direction Modern languages use three types of writing systems: horizontal left to right (lines from top to bottom), horizontal right to left (lines from top to bottom), and Vertical top to bottom (columns running right to left). Most modern applications already implement the first system (horizontal, left to right), so there is no need to discuss it. Text written right to left is used in scripts such as Hebrew, Arabic, and Syriac. Text written in Chinese, Japanese, and Korean utilize both vertical and horizontal styles. Bi-Directional Text Scripts written right to left are called Bi-directional, because they often incorporate text that reads left to right, such as foreign words and quotations, symbols, and numbers. The rules for rendering this text are outlined in the Unicode Technical Report 9. They are typically implemented at the OS level, and the OS will be referred to as Bidi-enabled. There is no specific construct in XML to indicate that data is stored in, and should be represented as, bi-directional. This is a rendering issue, and therefore has to be coded at the rendering level (in the stylesheets). CSS2 and XSL do offer an initial set of properties (direction:, and unicode-bidi:) but the implementation in the browsers is spotty and inconsistent. Vertical Writing Vertical can be found in literature, newspapers, and other media in the languages of Chinese, Japanese, Korean, and others. The CSS3 specification is designed to request the display of Justin Tilton: instructional media + magic, inc page 5 of 9

6 vertical writing through the use of the writing-mode property, and examples have been rendered using Adobe's Scalable Vector Graphics (SVG) viewer. Text Layout In addition to writing direction, variations exist in the way some scripts arrange characters, and formatting styles. Some of the following have been addressed by CSS3and XSL, but again the implementation of these specifications in browsing devices is unpredictable. Ruby Text The term Ruby designates a sequence of annotation text placed close to a base text, sometimes used to indicate the pronunciation of the base text. The ruby module defined by the W3C is used as a block enclosed in a <ruby> element with an <rb> (base) child and a <rt> (Ruby text) child. Combining Text Another special layout (specific to Japanese) is the grouping of characters in blocks (kumimoji) or in-line annotations (Warichu). Picture a string of characters with a short block in the middle of half sized characters in two lines enclosed by parenthesis. Context Shaping Some languages, such as Arabic, use different letter forms depending on their position in a word, or a preceding or following letter. UCS provides precomposed forms to represent many different characters, but these should not be used. The text will store the same code point for that particular character, and the rendering engine will pick the proper glyph depending on the context. Text Wrapping Many scripts do not use spaces to separate words. Consequently, wrapping of such text is unpredictable according to Roman-oriented algorithms. UCS provides a special character for cases when the application does not support language specific wrapping The character the Zero Width Space indicates to the engine that a break is needed. This solution is far from perfect, since the author must decide where to break the text, but is useful for languages like Thai. Sorting XSL offers a dedicated method for sorting data which can be called within the <xsl:apply templates> and <xsl:call-templates> elements. Attributes within the <xsl:sort> element can be set to order the list in different ways (i.e ascending, descending), and the element should adhere to the rules of sorting for the lang attribute if set. The implementation of lang specific sorting should be tested with each individual processor as not all of them work. Justin Tilton: instructional media + magic, inc page 6 of 9

7 Creating i18n Ready XML and XSLT Files Currently there exist a number of applications and workflows that have been developed to assist linguists to translate projects. These applications were typically developed to read in the Resource bundles of an application and expose to the linguist each of the translatable messages. Some examples of this might be the use of TRADOS to translate Microsoft RC files or the use of KBabel to translate UNIX based PO files. Some of these translation assistant applications have even been made XML aware in their latest versions, and some have built in capability for interoperability with other translation applications using an XML based interchange format called XLIFF. XLIFF is an initiative within OASIS to define the XML Localization Interchange File Format. The purpose of this format is to store localizable data and carry it from one step of the localization process to the other, while allowing interoperability between tools. While these tools could be helpful in the uporal localization process, and can provide an end-toend solution for traditional application development localization, they will not completely solve the localization needs for uportal. If however, the format by which we decide to store our language libraries is compatible with existing tools, and if channel developers provide a template for the translatable portions of their XSLT, we can make the process much smoother, and allow linguists to use the tools with which they are familiar. This additional effort should greatly increase the speed with which the uportal becomes localized. I suggest the use of the XLIFF standard as the means of storing our language libraries (our resource format). Transformations XLIFF en-us.xlf template.xlf html.xsl fr-fr.xsl Channel wml.xsl template.xlf en-us.xlf fr-fr.xsl en-us.xlf template.xlf voicexml.xsl fr-fr.xsl Figure 1 A channel with device dependent transformations and XLIFF language resource files Justin Tilton: instructional media + magic, inc page 7 of 9

8 Illustrated by Figure 1, each channel developer would build transformations with the forethought of replacing all rendered text with strings from the XLIFF resource libraries, identifying each with an ID unique to that transformation. After constructing the transformation, a template XLIFF file would be created so that linguists could easily and quickly create additional localizations. <xsl:when test="document($doc-file)"> <xsl:value-of select="document($docfile)/file/body/trans-unit[@id= 1 ]/target"/> </xsl:when> <xsl:otherwise>cannot find the file.</xsl:otherwise> Listing 1 Example excerpt of XSLT to call XLIFF resource library for localized text <?xml version="1.0"?> <!DOCTYPE xliff PUBLIC "-//XLIFF//DTD XLIFF//EN" " > <xliff version="1.0"> <file source-language="en" datatype="plaintext"> <header></header> <body> <trans-unit id="1"> <source xml:lang="en-us">cannot find the file.</source> <target xml:lang="fr-fr">fichier non trouvé.</target> </trans-unit> </body> </file> </xliff> Listing 2 Example simplified XLIFF file used as a resource library As an alternative to hand editing the XLIFF XML or using a third party tool, the uportal might eventually incorporate its own set of portal based translation tools (and translation memories), similar to the screen in Figure 2, from RWS Groups JCAT (a JAVA/Web based tool for computer assisted translations). Figure 2 A screen shot from JCAT a computer assisted translation tool from RWS Group Justin Tilton: instructional media + magic, inc page 8 of 9

9 Notes and References Much of the overview section of this document was excerpted from the Mozilla.org documentation on i18n. Additionally, sections of the analysis of Enabling the uportal were abstracted from Yves Savourel s book XML Internationalization and Localization. I have listed the following URL s as references for additional information. 1. XML Internationalization and Localization FAQ 2. Localization and Translation Tools 3. Internationalization resource directory: 4. The XML Localisation Interchange File Format Discussion list for the Internationalization Tag Set 6. The Okapi Framework 7. A common reference for interoperable text manipulation on the World Wide Web 8. Why a standard markup method is needed for working with multilingual documents 9. Mozilla Internationalization & Localization Guidelines ASPN XSL template examples for localization RWS Group JCAT Justin Tilton: instructional media + magic, inc page 9 of 9