The Internationalization Tag Set

Transcription

1 Richard Ishida 1

2 Richard Ishida 2

3 Richard Ishida 3

4 Richard Ishida 4

5 A schema (with a small 's') describes the structure of an XML document. Some formats in which h people write schemas include DTDs (Document Type Defintions), the W3C's XML Schema (with a capital 'S'), and RelaxNG. The ITS Best Practises document contains examples in each of these formats of how to implement the ITS tagset. Richard Ishida 5

6 Richard Ishida 6

7 Richard Ishida 7

8 This is an example of the need for special markup to support features of certain languages or scripts. At the top we have a quotation in Hebrew embedded in an English sentence. (The characters are shown in the order they appear in memory.) The second line shows how we would expect the text to be displayed with the characters 'W3C' to the left of the embedded quote. Unfortunately, the Unicode bidirectional algorithm alone is not able to produce this result. The lower line shows the result you would get from the code at the top. Richard Ishida 8

9 To achieve the desired result in XHTML you need to add the dir attribute around the quotation, ti with the value set to rtl. This establishes the directional context t for the Hebrew quote, which enables the Unicode bidirectional algorithm to correctly align the words. Richard Ishida 9

10 This next example shows a need for markup to assist in the localization process. Here is some documentation that refers to a hard panel. The text on the hard panel will not be translated. This means that the words START and MAKE READY/RUN should not be translated in the documentation. Richard Ishida 10

11 This is what a German translation might look like. Note that the uitext element content t has not been translated. t Richard Ishida 11

12 If the markup looks like this, there is nothing to indicate to a translator, translation tool or machine translation ti system that t the words in the uitext t elements should be left in English. Richard Ishida 12

13 It would help significantly if the translator or the tools were able to see something like a translate t attribute t with the value set to no to say that t the uitext t elements should not be translated. Richard Ishida 13

14 Our final example in this section relates to a best practice rather than additional markup. The caption for the photo has been captured in an attribute t value. The problem with this is that it is now impossible to identify the Japanese text using xml:lang information, nor to apply the kind of bidirectional markup we saw earlier for Middle Eastern languages. Richard Ishida 14

15 A much better approach is to use an element for the caption. You now have the possibility to apply metadata t to the text, t such as that t shown above. You are also, by the way, able to apply abbreviation markup, additional styling, etc. Richard Ishida 15

16 One way of addressing i18n needs in XML design is to follow the ITS specification. Here we introduce that t spec and some of the key ideas associated with it. Richard Ishida 16

17 Richard Ishida 17

18 ITS has a 'data category' called Translate. It can take the values 'yes' and 'no'. A data category is a kind of abstract t concept, which h is implemented using markup. Richard Ishida 18

19 To illustrate how to apply the values of this data category in markup we will use our previous example of hard panel text t that t must be left untranslated. t Richard Ishida 19

20 We can express that we don't want to translate the content of a particular element by using a local attribute, t on the element itself, ITS suggests that t you incorporate the its:translate attribute into your schema so that it can be used in this way in valid XML. The attribute can only have the values 'yes' and 'no'. Richard Ishida 20

21 This example says that all title attributes on acronym elements in the document should be considered d for translation. ti Remember that t the default is that t attributes t are left untranslated. (Note that acronyms can change during translation, so title attributes that contain the expanded version of the acronym may also need to be translated.) It is very useful to be able to say whether specific attributes are to be translated or not. This can only be done in ITS using global rules. Richard Ishida 21

22 Here's how we would apply that to the previous example. Richard Ishida 22

23 Richard Ishida 23

24 We may be faced with a slightly different situation, where the schema already has an attribute t that t indicates that t content t should not be translated, t but that t uses its own attributes or elements to express that. It is not obvious to a processor that the change attribute in the example above fulfills the same role as our its:translate attribute. Using global rules, however, we can associate that attribute with the translate data category. The markup at the top of the slide shows how you could do this. It selects all elements in the document with a change attribute with the value set to false, and using the translate attribute in the rule, asserts that this is equivalent to its:translate="no". There is then a similar declaration for change="true". This can be useful for describing legacy content so that a translation process can understand what text to lock for translation. Richard Ishida 24

25 Richard Ishida 25

26 Richard Ishida 26

27 Richard Ishida 27

28 Richard Ishida 28

29 Richard Ishida 29

30 First, you should check if this is really necessary. The "Best Practices for XML Internationalization" ti ti document contains already a lot of "ready-to-use" " integrations ti of ITS markup into existing schemas. Richard Ishida 30

31 Second, make yourself familiar with how the schema language works. In the "Best Practices for XML Internationalization" ti ti document, we provide only simple examples like on this slide. Note that even these examples are highly specific to the schema language. Richard Ishida 31

32 Third, examine the way your schema is designed. There are a lot of design patterns which h also differ between schema languages. It is not possible to be more detailed d or even exhaustive here, and be aware that we only provide examples in the "Best Practices for XML Internationalization" document. Richard Ishida 32

33 ITS 1.0 defines the precedence of ITS information for data categories. The precedence order is as described d on this slide. The following examples show the usage of local and global ITS markup and how precedence comes into play. Richard Ishida 33

34 Richard Ishida 34

35 Richard Ishida 35

36 Richard Ishida 36

37 Richard Ishida 37

38 Richard Ishida 38

39 Richard Ishida 39

40 We will now look at all the data categories that made it into the first version of ITS. We will look at how the concepts associated with that t data category can be applied locally, then globally. We will list the various elements and/or attributes that ITS defines to handle that data category, then provide some examples of their use. Richard Ishida 40

41 Richard Ishida 41

42 Richard Ishida 42

43 There is no local ITS usage for this data category. You should simply use xml:lang for this purpose. This example says that for any element in the content, the language attribute plays the same role as, and uses the same values as, the xml:lang attribute would. Make sure that the definition of the xml:lang attribute allows for empty values. That is: In a DTD you must not use NMTOKEN as the data type, instead use CDATA. In XML Schema the built-in data type language does not allow empty values. However, the declaration for xml:lang in the XML Schema document for the XML namespace at does allow for empty values and therefore can be used. It is not recommended to use your own attribute or element to specify the language of the content. The xml:lang attribute is supported by various XML technologies such as XPath and XSLT (e.g. the lang() function). Using something different would diminish the interoperability of your documents and reduce your ability to take advantage of some XML applications. Richard Ishida 43

44 If you have different languages in the attribute values and content of an element, consider nesting elements, if possible. See Handling attribute t values and element content in different languages. Richard Ishida 44

45 If you need to specify language as data or meta-data about something external to the document, do it with an attribute t different from xml:lang. l For more information see the article xml:lang in XML document schemas. The reason you would choose to create your own element (or attribute) is to convey the language as a value (as part of a data structure or as metadata about an external document) rather than to indicate the language of a specific piece of content. Avoiding the use of xml:lang to describe external language values avoids creating problems for content authors who need to label content for text-processing purposes. Richard Ishida 45

46 The global rules allow you to map legacy markup onto ITS categories. Richard Ishida 46

47 In this document the langcode element is used to specify the language of the text element. The langcode element has no inheritance it behavior equivalent to the one of xml:lang. Note: This example is a multilingual document, which has its own set of issues (see Best Practice 12: Working with multilingual documents). The corresponding ITS Rules document contains an its:langrule element that specifies that the langcode element holds the same values as the xml:lang attribute and applies to the text element. Richard Ishida 47

48 Richard Ishida 48

49 Richard Ishida 49

50 We have so far been looking at data categories that provide information that is particularly l useful for localization li of content. t This data category should be included d in schemas so that content authors can create content in languages such as those used in the Middle or Far East. Richard Ishida 50

51 This is an example of the need for special markup to support features of certain languages or scripts. At the top we have a quotation in Hebrew embedded in an English sentence. (The characters are shown in the order they appear in memory.) The second line shows how we would expect the text to be displayed with the characters 'W3C' to the left of the embedded quote. Unfortunately, the Unicode bidirectional algorithm alone is not able to produce this result. The lower line shows the result you would get from the code at the top. The bidi algorithm can also handle simple cases where right-to-left and left-to-right text are mixed. However, situations commonly arise where higher level contextual information is needed to achieve the desired layout of bidirectional text. This contextual information can be provided by markup in XML. Such markup also affects page layout behavior. For example, in a right-to-left context, table columns are ordered right-to-left, list bullets appear to the right of text, the page is right-aligned, and so forth. The desired effect can be achieved using Unicode control characters, but this is not recommended (See Unicode in XML and other Markup Languages [Unicode in XML]). Markup is needed to establish the default directionality of a document, and to change that where appropriate by creating nested embedding levels. Markup is also occasionally needed to disable the effects of the bidirectional algorithm for a specified range of text. Richard Ishida 51

52 The its:dir attribute is used here locally to express that the directional context of the quote is right-to-left. t This information is necessary if we want the text t to be displayed correctly. We have already seen this example earlier in the presentation. Richard Ishida 52

53 Note: Directionality information cannot be deduced from language markup: There is not necessarily a one-to-one match between a given language and what directionality to use. For example, Azerbaijani can be written using both right-to-left and left-to-right scripts, and the language code az is relevant for either. The values of inline directional markup are not necessarily aligned with the values of markup about the language. For example, a part of a document might be declared as having right-to-left directionality, but there might be only a general language declaration for a left-to-right script language available, like fr. Markup used to indicate directionality has values that indicate that the normal directionality should be overridden; it is not possible to indicate that using language related values. Richard Ishida 53

54 In the example on this slide, the content actually has markup to perform the same role as the its:dir attribute. t We use a global l rule to indicate that t all direction attributes with a value of rtltext are equivalent to its:dir="rtl". (In an actual implementation, you would probably want to define rules for all possible values of the direction attribute we kept things simple for the example.) Richard Ishida 54

55 If you are working with an existing schema where there is a way to specify text directionality that t is not implemented using the its:dir attribute, t you should provide an ITS Rules document where you use the its:dirrule element to associate the different directionality indicators with their equivalents in ITS. You must ensure that the legacy markup provides for the semantics as the ITS markup. Richard Ishida 55

56 Richard Ishida 56

57 Note: In XML documents, using markup is more appropriate than using Unicode Bidi Embedding Controls. See Bidi formatting codes vs. markup for bidi support for a more detailed explanation. You also need to use dedicated markup to apply directional information, rather than just applying CSS direction properties to ordinary elements. See CSS vs. markup for bidi support for further information. Richard Ishida 57

58 Richard Ishida 58

59 In this document the elements term and b should be treated as part of their parent, and dthe element tfn should ldbe treated t as an independent d run of ftext. t Richard Ishida 59

60 Many applications that process content for linguistic-related tasks need to be able to perform a basic segmentation ti of the text t content. t They need to be able to do this without knowing the semantics of the elements. While in many cases it is possible to detect mixed content automatically, there are some situations where the structure of an element makes it impossible for tools to know for sure where appropriate segmentation boundaries fall. For example, the boundaries of some inline elements, such as emphasis, do not typically correspond to segmentation boundaries; on the other hand, some inline elements embedded in a parent element, such as footnotes or quotations, may define segments that should be handled separately from the text in which they are embedded. Intelligent segmentation is particularly important in translation to successfully match source text against translation-memory databases. Richard Ishida 60

61 Whether you are creating a new schema or documenting legacy markup, provide an ITS Rules document where you use <withintextrule> tr elements to indicate which elements should be treated as either part of their parents, or as a nested but independent run of text. By default, element boundaries are assumed to correspond to segmentation boundaries. Richard Ishida 61

62 In the document the elements term and b should be treated as part of their parent, and dthe element tfn should ldbe treated t as an independent d run of ftext. t The <withintextrule> element is used to specify the behavior of three elements, all other elements are assumed to have the value "no". First <withintextrule>: The elements term and b are defined as part of the text flow. Second <withintextrule>: The element fn is defined as a separate bit of content nested inside its parent element. Richard Ishida 62

63 Richard Ishida 63

64 Ruby (or furigana in Japanese) is commonly used in Far Eastern scripts to provide phonetic transcriptions ti of the complex ideographic characters where such characters are obscure, or where the reader is assumed to not yet know how to read the character. An approach to implementing ruby markup in a schema is described by the W3C's Ruby Annotation Specification. Richard Ishida 64

65 Richard Ishida 65

66 The ITS specification provides elements that cover the usage described in the Ruby Annotation ti specification at The example of use shows how you would use ITS markup to associate a run of hiragana text with kanji base text in Japanese. It includes markup for ruby parentheses, which provide a useful fallback approach if an application doesn't support ruby text display. Richard Ishida 66

67 There are also a set of attributes and an element to point to or provide ruby markup in legacy content. t The rubytext element allows you to specify the ruby for an identified range of text in the document. This may be used in an internal set of global rules. Richard Ishida 67

68 If you are working with an existing schema where there is a way to specify text directionality that t is not implemented using the its:dir attribute, t you should provide an ITS Rules document where you use the its:dirrule element to associate the different directionality indicators with their equivalents in ITS. You must ensure that the legacy markup provides for the semantics as the ITS markup. Richard Ishida 68

69 Richard Ishida 69

70 Richard Ishida 70

71 Note: In XML documents, using markup is more appropriate than using Unicode Bidi Embedding Controls. See Bidi formatting codes vs. markup for bidi support for a more detailed explanation. You also need to use dedicated markup to apply directional information, rather than just applying CSS direction properties to ordinary elements. See CSS vs. markup for bidi support for further information. Richard Ishida 71

72 Richard Ishida 72

73 Richard Ishida 73

74 In the example on this slide we locally provide a note about the content of the first data element, and declare that t note to be of type 'alert'. The effect of labelling lli a note with a type is not normatively defined, but as an example, a translation tool could use the 'alert' attribute value to ensure that the translator sees the note before completing the translation. Further on in the example a note is provided for the second data element. This is just an informative note, so no type information has been given. (The default is 'description'.) A translation tool may make the translator aware that there is a note for this piece of content, but leave it up to the translator to decide whether or not they read it. Richard Ishida 74

75 Richard Ishida 75

76 Note that the content author can do this. Richard Ishida 76

77 Note that the content author can do this. Richard Ishida 77

78 This example shows global rules that select all msg elements with the class attribute t value set to "DisableInfo" " and associate them with a note that t describes how the message is to be used. It also gives this note a type of 'alert'. Note, also, that we used an its:translate="no" attribute locally on the ITS global rules markup to indicate that the note itself should not be translated. [Note that using text such as "The X has been disabled" in this way is very bad practice! For more information about that see the W3C I18n article "Using Composite Messages".] Richard Ishida 78

79 In this example we use the locnoterule element to select all data elements that have a parent element msg, and point to the location of the note information for that data element. In this case we use a relative XPath expression to say that the note is contained in the notes element that has the same msg parent. You can see the notes and text for translation in the lower part of the example. Note, also, that we use another global rule to state that all notes elements are to be left untranslated. Richard Ishida 79

80 In this example we use the locnoterule element to select all data elements that have a parent element msg, and point to the location of the note information for that data element. In this case we use a relative XPath expression to say that the note is contained in the notes element that has the same msg parent. You can see the notes and text for translation in the lower part of the example. Note, also, that we use another global rule to state that all notes elements are to be left untranslated. Richard Ishida 80

81 Richard Ishida 81

82 Note: In XML documents, using markup is more appropriate than using Unicode Bidi Embedding Controls. See Bidi formatting codes vs. markup for bidi support for a more detailed explanation. You also need to use dedicated markup to apply directional information, rather than just applying CSS direction properties to ordinary elements. See CSS vs. markup for bidi support for further information. Richard Ishida 82

83 Richard Ishida 83

84 In order to most effectively reuse translated text where content is reused (for example across updates) it is necessary to have a unique and persistent t identifier associated with the element. This identifier allows the translation tools to correctly track an item from one version or location to the next. After ensuring that this is the same item, the content can be examined for changes, and if no change has taken place the potential for reuse of the previous translation is very high. Change analysis of this kind constitutes an extremely powerful productivity tool for translation when compared to typical source matching techniques (a.k.a. translation memory). These techniques simply look for similar source text in a multilingual database without, most of the time, being able to tell whether the context of its use is the same. Identifiers can also be helpful to track displayed text back to its underlying source. For example, when reviewing a translated user interface, the identifiers can be used as temporary prefixes to the text so that any correction can be efficiently done on the proper strings. Richard Ishida 84

85 Your schema should provide xml:id (or an equivalent mechanism) to allow you to assign unique identifiers to elements, e.g. to <footnote> t elements. In XML documents, you should use unique identifiers where appropriate. Finally, use the "Elements within Text" data category to provide additional segmentation information. The purpose of the <withintextrule> element on this slide is to identify footnotes as nested chunks of text. Richard Ishida 85

86 Richard Ishida 86

87 Richard Ishida 87

88 Richard Ishida 88

89 Richard Ishida 89

90 You should not separate the article from the noun. If "hydraulic lift" is independently d replaced din the future by some other term, you may need dto change the article to 'an' or remove it. The article/noun separation also causes trouble for the translators. Without any easy way to see the actual term when translating the paragraph, they may not be able to decide the gender or number of the article. If it is used at the beginning of a different sentence, the term would need to be capitalized. The term is singular in the termbase, but it may need to be plural somewhere else in the document. In inflected languages the form required in the text may be different from the form stored in the termbase. For example, in Polish the term would be stored in its nominative form ("dźwignia hydrauliczna"), while it should be in its instrumental form once inserted in this context: "Używając dźwignię hydrauliczną podnieś pojazd z ziemi." Richard Ishida 90

91 Richard Ishida 91

92 Richard Ishida 92

93 Richard Ishida 93

94 Richard Ishida 94

95 Richard Ishida 95

96 Richard Ishida 96

97 If you do not indicate what words are terms of interest in the content, the translators t will not know that t these terms need to be translated t consistently. tl Often, multiple translators are working on different files in a given project, and the way they choose to translate specific words can be inconsistent with the way that other translators have translated them. If important terms are marked in the content, they can extract these terms from the content before the content is translated, and pre-translate them in the form of a shared electronic dictionary. This ensures consistency of translation of important terms. While markup denoting terms for a given schema level should be specified in a set of ITS rules provided with the schema, there are cases where these general rules need to be overridden or complemented for specific elements, in specific documents. It is up to the author of the content to provide such overriding markup. Richard Ishida 97

98 In this example we use local ITS markup to say that the word "motherboard" is a term, and point to its definition. iti Richard Ishida 98

99 This slide summarizes the global <termrule> element. Richard Ishida 99

100 This example uses a global rule to say that the content term elements are terms, and that t a pointer to the information describing that t term is to be found in the target attribute on these elements. Richard Ishida 100

101 Richard Ishida 101

102 Richard Ishida 102

103 It is difficult to manage concurrent translations in all languages. It is very likely that each translation ti will be done by a different translator, t in a different location. To facilitate this, the document will have to be broken down into separate parts and reconstructed later on. This adds processing time, increases cost and provides more opportunities for the introduction of errors. Depending on the point in the document's lifecycle, such a document may already contain translations, some up-to-date and some outdated (because the source material may have changed). In order to identify what parts need to be localized and what parts should be left alone, the document would then also need to contain custom information about localization state, which may or may not be supported by localization tools. Richard Ishida 103

104 Richard Ishida 104

105 Richard Ishida 105

106 Richard Ishida 106

107 Richard Ishida 107

108 The name of an element should indicate what its function is, not how its content will be presented, because presentation ti may vary depending di on different factors such as language, script, medium, or accessibility. Using documents where elements or attributes do not follow a predictable naming pattern may cause problems when using XSLT-driven processes. It may also be an issue for translation tools. This is especially true if not all parts of the document are to be translated, since it would be more difficult to specify rules to distinguish the translatable nodes from the non-translatable ones. Richard Ishida 108

109 Richard Ishida 109

110 Richard Ishida 110

111 Richard Ishida 111

112 ITS data categories are defined independent of each other. However, sometimes it makes sense to use them together. th We will provide an example on the following slides. Richard Ishida 112

113 The example demonstrates interaction between the "Translate" and the "Elements Within Text" data categories. A translator can benefit from having separate chunks at a block-level translatable text. However, inline elements should not constitute a separate chunk. This requirement can be achieved by the usage of "Elements Within Text". Take a look at the rules at this slide and think about it: what chunks of text are separate units? Richard Ishida 113

114 The answer is on this slide. Richard Ishida 114

115 This slide shows an output as an XLIFF document. The translatable chunk which is in the input document part of another translatable t chunk is here part of a separate <trans-unit> element. Richard Ishida 115

116 A question from a foot view: What can you expect from an ITS implementation? ti Richard Ishida 116

117 ITS defines two conformance types. This slide shows conformance of schemas who make use of ITS markup declarations. Richard Ishida 117

118 ITS describes also conformance related to processing of ITS information. The necessary conformance criteria i are listed here. Richard Ishida 118

119 Richard Ishida 119

120 Richard Ishida 120

121 Always remember that community involvement is crucial to development of W3C specifications. The W3C does not simply decide in an ivory tower to develop specifications such as ITS and impose them on the public. The process only starts when we have support from the W3C member companies, experts and industry participants who will compose the Working Group. If you feel that this work is valuable, please consider participating in the Interest Group. Richard Ishida 121