BiDi in the Wild. Challenges of the Unicode Bidirectional algorithm. Moriel Schottlender Software Engineer

Transcription

1 BiDi in the Wild Challenges of the Unicode Bidirectional algorithm Moriel Schottlender Software Engineer

2

3

4 Wikipedia s Right-to-Left support

5 Right-to-Left Wikipedias ~260 Wikipedias in Left-to-Right ~17 Wikipedias in Right-to-Left

6 Right-to-Left Wikipedias ~260 Wikipedias in Left-to-Right ~17 Wikipedias in Right-to-Left Arabic Wikipedia ~1,000,000 users ~375,000 articles

7 Right-to-Left Wikipedias ~260 Wikipedias in Left-to-Right ~17 Wikipedias in Right-to-Left Arabic Wikipedia ~1,000,000 users ~375,000 articles Persian Wikipedia ~514,000 users ~460,000 articles

8 Right-to-Left Wikipedias ~260 Wikipedias in Left-to-Right ~17 Wikipedias in Right-to-Left Arabic Wikipedia ~1,000,000 users ~375,000 articles Persian Wikipedia ~514,000 users ~460,000 articles Hebrew Wikipedia ~277,000 users ~175,000 articles

9

10

11 Editing Right-to-Left Wikipedias

12 A brief history of Right-to-Left support online

13 Long long ago Computers mostly only knew Left to Right Supporting non-latin scripts required special fonts There was no real Right-to-Left support

14 Long long ago Computers mostly only knew Left to Right Supporting non-latin scripts required special fonts There was no real Right-to-Left support Solution:

15 Long long ago Computers mostly only knew Left to Right Supporting non-latin scripts required special fonts There was no real Right-to-Left support Solution: Writing backwards

16 Long long ago Computers mostly only knew Left to Right Supporting non-latin scripts required special fonts There was no real Right-to-Left support Solution: Writing backwards

17 Long long ago Computers mostly only knew Left to Right Supporting non-latin scripts required special fonts There was no real Right-to-Left support Solution: Writing backwards enod eb ot dah gnihtemos

18 Long long ago Computers mostly only knew Left to Right Supporting non-latin scripts required special fonts There was no real Right-to-Left support Solution: Writing backwards Something had to be done

19 Pre-BiDi Solution: Visual and Logical encoding Visual: שלום עולם Char order: Logical: שלום עולם Char order: 432 1

20 Pre-BiDi Solution: Visual and Logical encoding Visual: שלום עולם Char order: (Someone had to type this backwards!) Logical: שלום עולם Char order: 432 1

21 Pre-BiDi Solution: Visual and Logical encoding Visual: שלום עולם Char order: (Someone had to type this backwards!) Logical: שלום עולם Char order: 432 1

22 Unicode Bidirectional Algorithm

23 Unicode Bidirectional Algorithm If all text on a page is uniform (all RTL or all LTR) the ordering of the display text is unambiguous.

24 Unicode Bidirectional Algorithm If all text on a page is uniform (all RTL or all LTR) the ordering of the display text is unambiguous. RTL content can include digits (written LTR) RTL content can be mixed with LTR content

25 Unicode Bidirectional Algorithm If all text on a page is uniform (all RTL or all LTR) the ordering of the display text is unambiguous. RTL content can include digits (written LTR) RTL content can be mixed with LTR content Examples: Santa Clara ב Unicode Conference אני הולכת להרצות ב צריך להתקשר ל

26 Unicode Bidirectional Algorithm If all text on a page is uniform (all RTL or all LTR) the ordering of the display text is unambiguous. RTL content can include digits (written LTR) RTL content can be mixed with LTR content The Bidirectional Algorithm is meant to solve ambiguity in rendering order. Examples: Santa Clara ב Unicode Conference אני הולכת להרצות ב צריך להתקשר ל

27 Quick primer to BiDi entity types Strong Affect the directionality of entities around them Alphabet Weak Do not affect the directionality of entities around them Punctuation*, digits Neutral Take the directionality of the context they re in Space, newline, tab, etc

28 Numbers 123 עברית

29 Numbers LTR 123 עברית RTL

30 Numbers LTR 123 עברית RTL LTR (Whitespace is neutral) 23 1 עברית RTL

31 Numbers LTR 123 עברית RTL LTR (Whitespace is neutral) 23 1 עברית RTL עברית RTL

32 Text and numbers English Hebrew English

33 Text and numbers English Hebrew English English עברית English

34 Text and numbers Strong Weak Strong Weak Strong English Hebrew English English עברית English

35 Text and numbers Strong Weak Strong Weak Strong English Hebrew English English עברית English LTR

36 Text and numbers Strong Weak Strong Weak Strong English Hebrew English English עברית English LTR RTL

37 Text and numbers Strong Weak Strong Weak Strong English Hebrew English English עברית English LTR RTL LTR

38 The confusing issue of the parentheses

39 The confusing issue of the parentheses Demo

40 Parentheses

41 Parentheses (hello)

42 Parentheses (hello)

43 Parentheses (hello) ( )שלום

44 Parentheses (hello) ( )שלום

45 Parentheses

46 Parentheses Good luck with HTML

47 Parentheses Good luck with HTML Or math comparisons

48 <a href=" title="foo">bar</a> LTR

49 <a href=" title="foo">bar</a> LTR RTL <a href=" title="foo"> <שלום /a> LTR LTR

50 <a href=" title="foo">bar</a> LTR RTL <a href=" title="foo"> <שלום /a> LTR LTR RTL <a href=" title=" <אהלן"<שלום /a> LTR LTR

51 <a href=" title="foo">bar</a> LTR RTL <a href=" title="foo"> <שלום /a> LTR LTR RTL <a href=" title=" <אהלן"<שלום /a> LTR LTR

52 RTL [[! שמאל זו אני Moriel schottlender.jpg 250px: ]]קובץ

53 RTL [[! שמאל זו אני Moriel schottlender.jpg 250px: ]]קובץ

54 RTL [[! שמאל זו אני Moriel schottlender.jpg 250px: ]]קובץ RTL RTL LTR RTL [[! שמאל זו אני Moriel schottlender.jpg 250px: ]]קובץ

55 RTL [[! שמאל זו אני Moriel schottlender.jpg 250px: ]]קובץ RTL RTL LTR RTL [[! שמאל זו אני Moriel schottlender.jpg 250px: ]]קובץ

56 Your brain on BiDi Credit: U.S. Navy photo by Photographer s Mate 2nd Class Aaron Peterson. Public Domain.

57 The tale of an animated bitmap

58 fig.bmp

59 \u202epmb.gif fig.bmp

60 \u202epmb.gif fig.bmp #!/usr/bin/env python import shutil shutil.copy("animated.gif", u"\u202epmb.gif") Code by David Chan

61 \u202epmb.gif fig.bmp #!/usr/bin/env python import shutil shutil.copy("animated.gif", u"\u202epmb.gif") Code by David Chan

62 \u202epmb.gif fig.bmp #!/usr/bin/env python import shutil shutil.copy("animated.gif", u"\u202epmb.gif") Code by David Chan

63 \u202epmb.gif fig.bmp #!/usr/bin/env python import shutil shutil.copy("animated.gif", u"\u202epmb.gif") Code by David Chan

64 \u202epmb.gif fig.bmp #!/usr/bin/env python import shutil shutil.copy("animated.gif", u"\u202epmb.gif") Code by David Chan

65 \u202epmb.gif fig.bmp #!/usr/bin/env python import shutil shutil.copy("animated.gif", u"\u202epmb.gif") Code by David Chan

66 \u202epmb.gif fig.bmp #!/usr/bin/env python import shutil shutil.copy("animated.gif", u"\u202epmb.gif") Code by David Chan

67 Control characters Implicit directional formatting U+200E: LEFT-TO-RIGHT MARK (LRM) U+200F: RIGHT-TO-LEFT MARK (RLM) Explicit directional embedding U+202A: LEFT-TO-RIGHT EMBEDDING (LRE) U+202B: RIGHT-TO-LEFT EMBEDDING (RLE) U+202C: POP DIRECTIONAL FORMATTING (PDF) Explicit directional override U+202D: LEFT-TO-RIGHT OVERRIDE (LRO) U+202E: RIGHT-TO-LEFT OVERRIDE (RLO) Explicit directional isolate U+2066: LEFT-TO-RIGHT ISOLATE U+2067: RIGHT-TO-LEFT ISOLATE U+2068: FIRST STRONG ISOLATE U+2069: POP DIRECTIONAL ISOLATE

68 When BiDi is technically correct but practically wrong

69 Solution: Force Isolation

70 Solution: Force Isolation

71 Solution: Force Isolation

72 New topic created on [board name]: <bdi>[topic title]</bdi> Solution: Force Isolation

73 New topic created on [board name]: <bdi>[topic title]</bdi> Solution: Force Isolation

74 New topic created on [board name]: <bdi>[topic title]</bdi> Solution: Force Isolation

75 Dates

76 Dates

77 8:40, במאי IST 28 אל TLV (TLV to IST 28 May, 8:40) Dates

78 8:40, במאי IST 28 אל TLV RTL LTR RTL LTR (TLV to IST 28 May, 8:40) Dates

79 8:40, במאי IST 28 אל TLV RTL LTR RTL LTR (TLV to IST 28 May, 8:40) Dates

80 LTR in RTL clients

81 LTR in RTL clients

82 LTR in RTL clients

83 LTR client LTR in RTL clients

84 RTL client LTR client LTR in RTL clients

85 RTL client LTR client LTR in RTL clients

86 RTL client 2 1 LTR in RTL clients 2 1 LTR client

87 RTL client 2 1 LTR in RTL clients 2 1 LTR client

88 RTL client Solution: Always define content directionality 1 LTR client

89 Applications implement BiDi inconsistently

90 Web Inconsistent implementation of BiDi (Facebook)

91 Web Maximum 2 Terms Inconsistent implementation of BiDi (Facebook)

92 Web Maximum 2 Terms Mobile Inconsistent implementation of BiDi (Facebook)

93 Web Maximum 2 Terms Terms 2 Maximum Mobile Inconsistent implementation of BiDi (Facebook)

94 Web Maximum 2 Terms Terms 2 Maximum Mobile Inconsistent implementation of BiDi (Facebook) BiDi not implemented???

95 desktop Inconsistent automatic detection of direction (Google Hangounts) mobile

96 desktop Inconsistent automatic detection of direction (Google Hangounts) mobile

97 desktop No auto-flip auto-flip Inconsistent automatic detection of direction (Google Hangounts) mobile auto-flip auto-flip

98 Even real life ignores BiDi (and Unicode) a lot

99

100

101 מסורת

102 ﺳﻧت מסורת

103

104 اﻟﺗﻘﻠﯾد

105 When BiDi itself is confusing

106 Numbers, math and phone numbers Numbers are rendered Left-to-Right even in (most) Right-to-Left contexts

107 Numbers, math and phone numbers Numbers are rendered Left-to-Right even in (most) Right-to-Left contexts LTR Phone number

108 Numbers, math and phone numbers Numbers are rendered Left-to-Right even in (most) Right-to-Left contexts LTR Phone number RTL מספר טלפון

109 Numbers, math and phone numbers Numbers are rendered Left-to-Right even in (most) Right-to-Left contexts LTR Phone number RTL מספר טלפון LTR Phone number

110 Numbers, math and phone numbers Numbers are rendered Left-to-Right even in (most) Right-to-Left contexts LTR Phone number LTR Phone number RTL RTL מספר טלפון Plus / Minus signs are weak מספר טלפון

111 Numbers, math and phone numbers Numbers are rendered Left-to-Right even in (most) Right-to-Left contexts LTR Phone number LTR Phone number RTL RTL מספר טלפון מספר טלפון Plus / Minus signs are weak LTR There are 1-2 things but 4-5 others

112 Numbers, math and phone numbers Numbers are rendered Left-to-Right even in (most) Right-to-Left contexts LTR Phone number LTR Phone number RTL RTL מספר טלפון Plus / Minus signs are weak מספר טלפון spaces LTR There are 1-2 things but 4-5 others

113 Numbers, math and phone numbers Numbers are rendered Left-to-Right even in (most) Right-to-Left contexts LTR Phone number LTR Phone number RTL RTL מספר טלפון מספר טלפון spaces Plus / Minus signs are weak LTR There are 1-2 things but 4-5 others RTL אחרים 5-4 דברים אבל 1-2 יש spaces (flipped)

114

115

116

117

118

119 Printed abroad Printed in Israel

120 Bonus: Emoticons

121 Emoticons LTR :) RTL (:

122 Emoticons LTR :) LTR :( RTL RTL (: ):

123 Emoticons LTR :) LTR :( RTL RTL (: ): LTR :D LTR :P RTL RTL D: P:

124 (Bonus fail) Web

125 (Bonus fail) Web Android: Google Drive Android: Google Slides

126 RTL Users Expect nothing good

127 RTL Users are used to their computer not quite cooperating

128

129

130

131 Now what?

132 Now what? Consistency in implementing the Bidirectional algorithm

133 Now what? Consistency in implementing the Bidirectional algorithm Standard in predicting directionality while typing

134 Now what? Consistency in implementing the Bidirectional algorithm Standard in predicting directionality while typing Improving isolation of numbers and dates

135 Now what? Consistency in implementing the Bidirectional algorithm Standard in predicting directionality while typing Improving isolation of numbers and dates Consistent punctuation within sentences (We solved jumping parentheses; let s solve periods, commas, and colons!)

136 Remember Parentheses... LTR MSchottlender (WMF)

137 Remember Parentheses... LTR MSchottlender (WMF) RTL (MSchottlender (WMF

138 Remember Parentheses... LTR MSchottlender (WMF) RTL (MSchottlender (WMF

139 Remember Parentheses... LTR MSchottlender (WMF) RTL (MSchottlender (WMF

140 Remember Parentheses... LTR MSchottlender (WMF) RTL (MSchottlender (WMF

141 Keep RTLing

142 Keep RTLing

143 Keep RTLing