Character Encodings. Fabian M. Suchanek

Transcription

1 Character Encodings Fabian M. Suchanek 22

2 Semantic IE Reasoning Fact Extraction You are here Instance Extraction singer Entity Disambiguation singer Elvis Entity Recognition Source Selection and Preparation

3 Scripts Thanks for all the fish Google Translate. No warranty. 3

4 Scripts Thanks for all the fish (Latin) ( Simplified Chinese) (Hebrew) (Arabic) (Korean) (Thai) Google Translate. No warranty. 4

5 How to map characters to bytes? y A a ß? 1 byte = 8 bit = 255 e é numbers ,000 different characters from 90 scripts 5

6 Def: Character encoding A character encoding (also: char encoding) is a bijective mapping from characters to (sequences of) bytes. A -> 65 B -> 66 characters ü -> 99, 42 bytes... -> 99, 99, 2 6

7 Def: ASCII encoding The ASCII encoding is a particular character encoding maps certain chars to single bytes, and ignores the others. A -> 65 B -> 66 C -> > ü -> 26 letters + 26 lowercase letters + punctuation 100 characters Disadvantage: works only for English 7

8 Def: Code pages A code page is a character encoding that maps script-specific characters to single bytes. Greek code page: A -> 65 B -> 66 Western code page: A -> 65 B -> 66 α -> à -> (0-127 are usually mapped as in ASCII) Disadvantages: We have to know the code page We cannot mix scripts We cannot represent more than 256 characters Example (View -> Encoding) 8

9 Def: HTML entities HTML entities are a particular character encoding where particular strings (as defined by W3C) represent characters. à -> à ü -> ü ß -> ß... These are sequences of bytes if encoded in ASCII Advantage: Works in all browsers Disadvantage: Very clumsy Example List 9

10 Def: Unicode Unicode is a character encoding that maps each character to [0, ], i.e., to 4 bytes. A -> 65 -> 0, 0, 0, 65 B -> 66 -> 0, 0, 0, 66 α-> > 0, 0, 3, 42 -> > 0, 0, 4, (not the real mappings) Characters are as in ASCII Advantage: Maps all known characters Disadvantage: Takes much space Example1 Example2 10

11 Def: UTF-8 UTF-8 is a particular character encoding that maps Unicode characters to sequences of bytes of different lengths. A -> 65 B -> 66 α-> 128, 42 -> 128, 128, 32 11

12 UTF-8: Chars 0-0x7F Unicode chars 0-0x7F are mapped like in ASCII (i.e., to a single byte). A -> 65 B -> a -> 96 b -> $ -> 36! -> Advantages: Compatibility with ASCII and code pages Space efficiency for English docs 12

13 13 UTF-8: Chars 0x80-0x7FF Unicode chars 0x80-0x7FF (11 bits) are mapped to two bytes as follows: xxxxxxxxxxx 110xxxxx 10xxxxxx 2 bytes Unicode 9x80-0x7FF are Greek, Arabic, Hebrew etc.

14 14 UTF-8: Chars 0x80-0x7FF Example Unicode 11 bit representation ç = 0xE7 =

15 UTF-8: Chars 0x80-0x7FF Example Example: Encoding façade f a ç a... 0x66 0x61 0xE7 0x Example 15

16 UTF-8: Chars 0x800-0x7FFF Unicode chars 0x800-0x7FFF (16 bits) are mapped to three bytes as follows: xxxxxxxxxxxxxxxx 1110xxxx 10xxxxxx 10xxxxxx 3 bytes This character range concerns mainly Chinese. 16

17 17 Decoding UTF f a ç a... if the byte starts with 0xxxxxxx if the byte starts with 110xxxxx if the byte starts with 1110xxxx if the byte starts with 10xxxxxx

18 Decoding UTF f a ç a... if the byte starts with 0xxxxxxx => it s a normal ASCII character if the byte starts with 110xxxxx => it s an extended char, 1 byte follows if the byte starts with 1110xxxx => it s a Chinese char, 2 bytes follows if the byte starts with 10xxxxxx => it s a follower byte, you messed it up! 18

19 Summary: UTF-8 UTF-8 maps Unicode chars to 1-4 bytes. Advantages: common Western chars are only 1 byte backwards compatibility with ASCII stream readability (follower bytes cannot be confused with marker bytes) sorting compliance 19

20 UTF-16 UTF-8 is inefficient if the text contains many characters between 0x0800 and 0xFFFF: It needs 3 bytes. For this reason UTF-16 has been proposed. It encodes every Unicode character in either 16 or 32 bits. Advantages: less space consumption in the range 0x0800-0xFFFF: 2 bytes as opposed to 3 in UTF-8. Disadvantages: more space consumption in the range 0x0000-0x007F: 2 bytes instead of 1 in UTF-8. not backwards compatible to ASCII 20

21 Example: Char encodings in Python Reading: with open( text.txt, encoding= utf-8 ) as file: for line in file: print(line) If omitted, this uses the default encoding of the operating system, which might be different from UTF-8! Writing: with open( text.txt, w, encoding= utf-8 ) as file: file.write( Bună ziua! ) 21

22 22 Example: Char encodings in Java File f = new File(...); InputStream s = new FileInputStream(f); Reader r = new InputStreamReader(s, UTF-8 ); f a ç

23 Summary: Char encodings ASCII: only English chars Code pages: one page per script HTML entities: work in browsers Unicode: maps all chars UTF-8: maps chars to variable # bytes In most applications, UTF-8 is the encoding of choice. ->archiving ->named-entity-recognition 23