Faster Segmentation-Free Handwritten Chinese Text Recognition with Character Decompositions. Théodore Bluche & Ronaldo Messina

Transcription

1 Faster Segmentation-Free Handwritten Chinese Text Recognition with Character Decompositions Théodore Bluche & Ronaldo Messina

2 Handwritten Chinese Text Recognition Main difficulty: large number of characters (4000+) in Chinese (and they are complex in shape) Has long been and still is mostly character-based (character segmentation, then recognition) that approach is now deprecated for most scripts (Latin, Arabic, ): we recognize text lines directly

3 Segmentation-free Chinese Text Recognition Messina, Ronaldo, and Jerome Louradour. "Segmentation-free handwritten Chinese text recognition with LSTM-RNN." Document Analysis and Recognition (ICDAR), th International Conference on. IEEE, Recognition from line images with MDLSTM-RNNs Training with CTC (4000+ labels)

4 Segmentation-free Chinese Text Recognition The size of the character set is an issue! - The last linear layer involves a product with a huge matrix The softmax should normalize a lot of activations It makes this model very slow for Chinese text, compared to Latin or Arabic scripts!

5 Usual Time Distribution Most of the time is spent in the first LSTM - >50% of total time ~20ms / line

6 Processing Time for Chinese HWR The first LSTM still takes ~20ms / line but over 60% of the processing time is spent in the last linear layer (big matrix multiplication), collapse and even softmax takes 25%!! Prohibitive in production models

7 How would a human do? The model transcribes lines of text - looks at the image - types its contents - by choosing each character among a list of That d be quite long for a human too to type a transcript on a 4000-key keyboard...

8 Not everyone can afford (or would even want to use) that keyboard!

9 Outline of this talk Chinese Input Methods - character decompositions Method Overview Results Conclusion

10 Outline of this talk Chinese Input Methods - character decompositions Method Overview Results Conclusion

11 Input Methods for Chinese Input method = simplify entering Chinese text on QWERTY keyboards Sequences of keys are mapped to Chinese characters i.e. reduce the alphabet from several thousands to a few dozens Two main categories: Phonetic-based (e.g. pinyin) ma: 傌, 马, 亇,么 Graphic-based each key (more or less) represents a component of a character (e.g. Cangjie, Wubi)

12 Cangjie Graphical decomposition 24 basic code units X for collisions or "difficult to decompose" parts Z auxiliar for entering punctuation Rules Most have 4 codes Direction Connected forms Un-connected forms...and Exceptions Fixed decompositions Arbitrary codes for characters that cannot be decomposed (卍, 姊, 臼, )

13 Wubi Graphical decomposition 25 codes Keyboard is divided according to the type of stroke (H,V, At most five, but many with less than 4 codes Most characters are uniquely defined Rules for disambiguation /, \, hook)

14 Outline of this talk Chinese Input Methods - character decompositions Method Overview Results Conclusion

15 Proposed Method Instead of recognizing Chinese characters, we propose to recognize their decomposition according to an input method (i.e. the neural network output the sequences of keys you should write to obtain the right transcription) Training: The ground-truth is converted into sequences of codes A character delimiter symbol is inserted between the codes MDLSTM trained with CTC to predict the sequences of codes (we added two BLSTM layers on top of the network to help it capture the code dependencies) Recognition: the codes should be mapped back to characters The sub-sequences between delimiters may be mapped to characters with the input methods (wrong codes sequences are mapped to an error) (more robust) a transducer accepting valid code sequences and outputting characters

16 Arbitrary decompositions Goal: Assess whether the network's internal representations should be related to the graphical aspect of the characters Fixed length: we just adapt the number of "codes" wrt charset 522 = = = 4096 Random assignments Similar characters do not necessarily have a similar encoding (unlike the chosen input methods)

17 Examples 加快步伐 Arb-2: Qr Xy Mi ag qz Arb-3: EML DJC MGL MID DEI Arb-4: GGGF EFGE DAGA FHBF CECG cangjie: K S R P D K Y L M H O I wubi: lkg nakg hir wat

18 Outline of this talk Chinese Input Methods - character decompositions Method Overview Results Conclusion

19 Database CASIA off-line handwriting database HWDB2.0 - HWDB2.2 (segmented into lines) characters (85/15)% random split for (train/valid) 2666 character classes in training set Evaluation set: ICDAR 2013 Task 4 Held-out part of CASIA 3397 lines characters 1379 character classes Line-segmented

20 Per-layer timings

21 Results (validation) Model Codes (%ED) Baseline Character (%ED) 5.1 Cangjie Wubi Arb Arb Arb ED: edit distance, or character error rate Train/valid = 85/15% CASIA-2.x

22 Results (test) Model NN+Map (%CER) NN+LM (%CER) Baseline Cangjie Wubi Arb Arb Arb ICDAR CER: character error rate - LM: here a character 3-gram test = ICDAR 2013 Task 4

23 加快步伐 (RNN) l k g n n w y h i i u h a t 加 快 Correct codes 步: h i r 伐: w a t "Near" codes 淼: i i i u 越: f h a t 戏: c a t 找: r a t (RNN+LM) wubi sample 加快步伐

24 加快步伐 (RNN) K S R P D K Y M V H O I 加 快 迓 伐 NB: "O I" maps to 3 different "characters" (亽, 仏, 伐) 步: Y M L H (RNN+LM) 加 快 步 伐 cangjie sample

25 加快步伐 (RNN) Q r X y A z b f q Z 加 快 烛 线 NB: 步 : M i 伐 : a G (RNN+LM) 加 快 速 线 Arb-2 sample

26 Outline of this talk Chinese Input Methods - character decompositions Method Overview Results Conclusion

27 Conclusion RNNs can predict sequences instead of characters Shape matters for decompositions We achieved a 4x speedup with a mild degradation wrt baseline Processing speed is now at par with Latin script languages Smaller footprint models

28 Future research directions Optimize decompositions for recognition performance wubi/cangjie were designed for ease of use (humans) Mixed traditional/simplified Chinese models More data re-use Alternative encodings Universal multi-lingual model with Unicode points

29 Thank you! Questions?