Infrastructure for High-Quality Arabic

Transcription

1 TUG 06 Marrakech Infrastructure for High-Quality Arabic Yannis Haralambous École Nationale Supérieure des Télécommunications de Bretagne Technopôle Brest Iroise, CS 83818, Brest Cedex TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 1/39

2 About Characters and Glyphs The Incessant Ballet TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 2/39

3 Characters and Glyphs A A AA 0041 latin capital letter a TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 3/39

4 Characters and Glyphs A A AA 0041 latin capital letter a A 0391 greek capital letter alpha TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 3/39

5 Characters and Glyphs 0643 arabic letter kaf, 062A arabic letter teh, 0627 arabic letter alef, 0628 arabic letter beh TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 4/39

6 Characters and Glyphs 0939 devanagari letter ha, 093F devanagari vowel sign i, 0928 devanagari letter na, [virama], 0926 devanagari letter da, 0940 devanagari vowel sign ii h E³ n [virama]d ³F Eh df TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 5/39

7 The TEX Way TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 6/39

8 From Input to DVI Input: character a (code 97); token: a11 (code 97, category 11); char_node: <fontcmr10, glyph index 97>; opcode: set_char_97 (implicit glyph index 97); result: a. TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 7/39

9 From Input to DVI Input: characters fi (codes 102, 105); tokens f11 i 11 (codes 102 and 105, category 12); lig_node: <font cmr10, glyph index 12, hlist: f i >; opcode: set_char_12 (implicit glyph index 12); result: fi. TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 8/39

10 From Input to DVI Input: characters \discretionary{k-}{k}{ck}; tokens: discretionary {1 k } 2 { 1 k 11 } 2 { 1 c 11 k 11 } 2 ; disc_node: <pre-break hlist: k-, post-break hlist: k, nodes to remove: 2>, char_nodes: <fontcmr10, 63>, <fontcmr10, 107>; opcodes: set_char_63, set_char_107 if no hyphenation, set_char_107, set_char_45, line break, set_char_45 otherwise; result: ck or k- + line break + k. TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 9/39

11 TEX Reads Characters, Writes Glyphs T EX reads characters; a character token is a character and a catcode; a char_node is a font ID and a glyph index; a DVI set_char opcode is a glyph index (in the current font). Example: in the DVI file, fi is glyph 12 of fontcmr10 (no reference to characters f, i). TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 10/39

12 Arabic Typeset With Omega 1 Input: characters ; tokens: ; context analysis OTP reads, replaces by and pushes INSIDE mode, reads, replaces by, reads, replaces by, pops mode, reads followed by EOB, leaves unchanged; tokens: ; char_nodes and DVI opcodes as usual, no reference to initial characters. TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 11/39

13 Our Vision Through mighty textemes and wonders TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 12/39

14 Textemes I m a character TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 13/39

15 Textemes I m a character I m your glyph TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 13/39

16 Textemes Let us be a texteme! TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 14/39

17 Textemes Let us be a texteme!... and they lived happily ever after and had many properties... TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 14/39

18 The Texteme Notion A texteme is a set {c,p1 = v 1,...,p n = v n,g} where: c is a Unicode character (can be void); for 0 i n, pi is a property name and v i a property value (n can be 0); g is a glyph reference (can be void); all texteme components can be inserted, modified and/or deleted by the user, by OTPs, and by modules; they can also be locked and unlocked by the user and by OTPs. TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 15/39

19 Notation c=w kern=-0.28pt g= w c=a hyph=1 g= a c=t color=yellow g= t c=e g= e c=r g= r to obtain: water c=f gdef=f g= fi c=i gdef=i g= c=l g= l c=l g= l c=e g= e to obtain: fille TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 16/39

20 Typesetting Arabic With Textemes 1) c=0643 g= c=062a color=red g= c=0627 g= c=0628 g= 2) c=0643 form=1 g= c=062a color=red form=2 g= c=0627 form=3 g= c=0628 form=0 g= 3) c=0643 form=1 g= c=062a color=red form=2 g= c=0627 form=3 g= c=0628 form=0 g= and the result: TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 17/39

21 Typesetting Hindi With Textemes 1) c=0939h g= c=093fe³ g= c=0928n virama=1 g= c=0926d g= c=0940 ³F g= 2) c= link E³ g=e c=0939h g=h c=093fe³ g= c=0928n virama=1 g= c=0926d g=d c=0940 ³F g=f and the result:eh df TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 18/39

22 Dynamic Typesetting T EX uses character, ligature, discretionary nodes; Gutenberg used ligatures and variant forms for optimizing justification: Omnia per ip um facta unt: et ine ip o factum e t n Oµia «iπm facta unt: & ~ne iπo factum eá ni il. Arabic can also take advantage of dynamic typesetting to minimize the use of black glue (keshideh). TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 19/39

23 Extended Graph T EX builds a directed graph of all feasible line breaks of a given paragraph. Arcs of this graph carry a metric: the badness of lines. The best choice of line breaks is the shortest path for that metric. Ω 2 will use an extended TEX graph where there will be more than one arcs between two nodes, depending on variant glyphs and activation of ligatures. TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 20/39

24 Dynamicity of Arabic Script Arabic text justification can be obtained by (in order of priority): using blank spaces of variable width (as in other scripts); enabling or disabling ligatures; choosing between alternative forms of glyphs; inserting keshideh connections between letters (preferably curvilinear ones), also called black glue. TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 21/39

25 The Typesetting Process TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 22/39

26 Step 0: Arabic forms most systems consider that forms are contextually calculated; ZWJ is a clumsy solution; Arabic form can become a texteme property. TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 23/39

27 Step 0: Short vowels very useful for NLP but not written; Ahmed Lakhdar Ghazal said:... ; there is software for vowelizing (Sakhr Diacritizer, etc.); short vowels can become texteme properties; visibility of short vowels also; same problem for all-caps Greek. TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 24/39

28 Step 0: Semitic roots and dots Semitic roots are very useful for indexation; there are also prefixes and postfixes; old manuscripts do not have dots on letters; this information can be inserted in texteme properties. TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 25/39

29 Step 0: Semantic keshideh Keshideh (black glue) can be semantic; variant forms can have semantic overload; this information can be inserted in texteme properties. TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 26/39

30 Step 0: Color color can be very useful, but it must not interfere with contextual analysis; this information can be inserted in texteme properties; the mandatory ligature lam-alif is a problem: TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 27/39

31 Step 1: Hyphenation Arabic language is not hyphenated; Arabic script can be hyphenated! TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 28/39

32 Step 2: Contextual analysis Calculate forms when the property is not locked; can be done either at OTP or at module level. TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 29/39

33 Step 2 : Hamza rules the idea belongs to Klaus Lagally; in fact, the Unicode Arabic table is not grammatical enough; Arabic grammar mentions letter hamza; hamza can be: arabic letter hamza, arabic letter alef with hamza, arabic letter yeh with hamza or arabic letter waw with hamza, depending on context (including short vowel information). TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 30/39

34 Step 3: Bidi My friend said! or My friend said! ; RLE (RIGHT-TO-LEFT EMBEDDING), LRE (LEFT-TO-RIGHT EMBEDDING): push a level ; RLO (RIGHT-TO-LEFT OVERRIDE), LRO (LEFT-TO-RIGHT OVERRIDE): push a behaviour ; PDF (POP DIRECTIONAL FORMAT): pop ; RLM (RIGHT-TO-LEFT MARK), LRM (LEFT-TO-RIGHT MARK): cheat. TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 31/39

35 Step 4: OpenType and Arabic 1. supply the glyph corresponding to the pair (character, contextual form), the form being provided as an OpenType feature [GSUB table, lookup of type 1 single substitution ]; 2. supply grammatical (lam-alif) and esthetic ligatures [GSUB table, lookup of type 4 ligature ]; 3. supply alternative forms for glyphs [GSUB table, lookup of type 3 variant selection ]; 4. kerning between single or ligatured glyphs [GPOS table, lookup of type 2 positioning of a pair of glyphs ]; 5. place short vowels and other diacritics on isolated glyphs or on ligature components [GPOS table, lookups of types 4 diacritical marks and 5 diacritical marks on ligatures ]. TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 32/39

36 Step 4: OpenType vs. Super-OpenType once the longest glyph string is found, the substitution or positioning is applied and we move on; special case: backtrack and lookahead; in Super-OpenType we store all cases. TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 33/39

37 Step 5: Fine-Tuning move apart touching glyphs: letters, short vowels, diacritics. TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 34/39

38 Step 6: Extended Graph Knuth dixit (p. 143 of Digital Typography ): It is interesting to consider how to extend the total-fit algorithm so that it could handle cases like the dropping of m s and n s in Figure 22. The badness function of a line would then depend not only on its natural width, stretchability, and shrinkability; it would also depend on the number of m s and the number of n s on that line. A similar technique could be used to typeset biblical Hebrew, which is never hyphenated; Hebrew fonts intended for sacred texts usually include wide variants of several letters, so that individual characters on a line can be replaced by their wider counterparts in order to avoid wide spaces between words. TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 35/39

39 Step 6: Extended Graph an extended T EX graph is a graph where every arc is itself a graph with respect to variants and ligatures activations; this can lead to combinatorial explosion: ten lines of 60 glyphs which three forms each = combinations; we need a strateg y TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 36/39

40 Step 6: Strategies 1. Classify glyphs by width (the number of classes is inverse proportional to amount of white and black glue authorized); 2. give an order to classes; 3. calculate badnesses of random combinations until one is less than our treshold; 4. other ideas... TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 37/39

41 Step 7: Post-processing 1. Mainly for keshideh: change glyphs surrounding it and draw a nice curve (see paper by Elyaakoubi et al. 2. sometimes keshideh may carry a glyph: TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 38/39

42 Conclusion arabic is a beautiful script let us hope that some day computer science will be worthy of it! TUG 06 Marrakech Infrastructure for High-Quality Arabic p. 39/39