Character Recognition Internals

Transcription

1 SSIP 2005, Szeged Character Recognition Internals Dr. István Marosi Scansoft-Recognita, Inc., Hungary

2 OCR Internals Main tasks of an OCR system: Image acquisition Layout recognition Text recognition User assisted correction Result exportation

3 OCR Internals Main tasks of an OCR system: Image acquisition Get image B/W Scanning Gray Scanning Color Scanning Load from image file Preprocess image Layout recognition Text recognition User assisted correction Result exportation

4 OCR Internals Main tasks of an OCR system: Image acquisition Get image Preprocess image Color separation Thresholding Despeckling Rotation Deskewing Layout recognition Text recognition User assisted correction Result exportation Color Separation De-speckle, de-skew

5 The Preprocessed Image Joined chars

6 The Preprocessed Image Joined chars

7 The Preprocessed Image Joined chars

8 The Preprocessed Image Broken chars

9 The Preprocessed Image Broken chars

10 The Preprocessed Image Broken chars

11 OCR Internals Main tasks of an OCR system: Image acquisition Layout recognition Text zones Columns of flowed text Tables Inverse text Graphic zones Text recognition User assisted correction Result exportation

12 OCR Internals Main tasks of an OCR system: Image acquisition Layout recognition Text zones Graphic zones Line Art Photo Text recognition User assisted correction Result exportation

13 OCR Internals Main tasks of an OCR system: Image acquisition Layout recognition Text recognition Segmentation Calculation of Feature Vector Elements Classification Language Analysis Voting User assisted correction Result exportation

14 Segmentation What are those pixel groups belonging to a single letter?

15 Segmentation What are those pixel groups belonging to a single letter?

16 Segmentation What are those pixel groups belonging to a single letter?

17 Segmentation What are those pixel groups belonging to a single letter?

18 Segmentation What are those pixel groups belonging to a single letter?

19 Segmentation What are those pixel groups belonging to a single letter?

20 Segmentation What are those pixel groups belonging to a single letter?

21 OCR Internals Main tasks of an OCR system: Image acquisition Layout recognition Text recognition Segmentation Calculation of Feature Vector Elements Classification Language Analysis Voting User assisted correction Result exportation

22 Calculation of FV Elements: Contour Tracing Find a (new) white-black transition Follow the edge of the pixels using the MIN or MAX rule Administrate the already traced white-black transitions Collect information while going around And repeat the process on new shapes...

23 Contour Tracing Find a (new) white-black transition Follow the edge of the pixels using the MIN or MAX rule Administrate the already traced white-black transitions Collect information while going around And repeat the process on new shapes...

24 Contour Tracing Find a (new) white-black transition Follow the edge of the pixels using the MIN or MAX rule a b if black(a) then turn(ccw) else if black(b) then forward else turn(cw)

25 Contour Tracing Find a (new) white-black transition Follow the edge of the pixels using the MIN or MAX rule a b a b if black(a) then turn(ccw) else if black(b) then forward else turn(cw) if white(b) then turn(cw) else if white(a) then forward else turn(ccw)

26 Contour Tracing Find a (new) white-black transition Follow the edge of the pixels using the MIN or MAX rule Administrate the already traced white-black transitions Collect information while going around And repeat the process on new shapes...

27 Some Easily Calculatable Data Problem #1 Turning CW: I n =I n-1 +1 Turning CCW: I n =I n-1-1 Going Forward: I n =I n-1

28 Some Easily Calculatable Data Problem #2 Turning CW: I n =I n-1 +1 Turning CCW: I n =I n-1-1 Going Forward: I n =I n-1

29 Some Easily Calculatable Data Problem #3 Going Up: I n =I n-1 -X n Going Down: I n =I n-1 +X n Going Right: I n =I n-1 Going Left: I n =I n-1

30 Some Easily Calculatable Data Problem #4 Going Up: I n =I n-1 -X n Going Down: I n =I n-1 +X n Going Right: I n =I n-1 Going Left: I n =I n-1

31 OCR Internals Main tasks of an OCR system: Image acquisition Layout recognition Text recognition Segmentation Calculation of Feature Vector Elements Classification Language Analysis Voting User assisted correction Result exportation A B A B

32 Classification; Training models Restricted Coulomb Energy (RCE) Network (Dr. Leon Cooper, Dr. Charles Elbaum) B A B A

33 Classification; Training models Restricted Coulomb Energy (RCE) Network (Dr. Leon Cooper, Dr. Charles Elbaum) B A B A

34 Classification; Training models Nestor Learning System (NLS)

35 Classification; Training models Nestor Learning System (NLS) Default radius R max

36 Classification; Training models Nestor Learning System (NLS)

37 Classification; Training models Nestor Learning System (NLS) Default radius R max

38 Classification; Training models Nestor Learning System (NLS)

39 Classification; Training models Nestor Learning System (NLS)

40 Classification; Training models Nestor Learning System (NLS) Default radius R max

41 Classification; Training models Nestor Learning System (NLS)

42 Classification; Training models Nestor Learning System (NLS) Decreased radius

43 Classification; Training models Nestor Learning System (NLS)

44 Classification; Training models Nestor Learning System (NLS) Decreased radius R min

45 Classification; Training models Nestor Learning System (NLS) Pass 2 Decreased radius

46 OCR Internals Main tasks of an OCR system: Image acquisition Layout recognition Text recognition Segmentation Calculation of Feature Vector Elements Classification Language Analysis Voting User assisted correction Result exportation

47 Voting Text recognition in OmniPage Pro OCR Engines available: Caere s engine (codename: Salt & Pepper) Recognita s engine (codename: Paprika) ScanSoft s engine (codename: Fireworx)

48 Voting Text recognition in OmniPage Pro OCR Engines available: Caere s engine (Salt & Pepper) Uses a Matrix Matching based algorithm feature set: 40 cells of an 8x5 grid good overall description of a shape weaker at detailed structure Recognita s engine (Paprika) Uses a Contour Tracing based algorithm feture set: convex and concave arcs on the contour good detailed description of a shape weaker at overall structure

49 Voting Text recognition in OmniPage Pro OCR Engines available: Caere s engine (Salt & Pepper) Recognita s engine (Paprika) ScanSoft s engine (Fireworx) Segmentation algorithms:

50 Voting Text recognition in OmniPage Pro OCR Engines available: Caere s engine (Salt & Pepper) Recognita s engine (Paprika) ScanSoft s engine (Fireworx) Segmentation algorithms: Developed by independent groups Have different strengths and weaknesses

51 Text recognition in OmniPage Pro OCR Engines available Segmentation algorithms Conclusion: Voting They are complementary Let s create a voting system

52 Voting strategies Voting External Black box voting Paprika Fireworx Image Salt & Pepper Txt 3 Txt 2 Txt 1 Vote Dict ~20% gain Final Txt

53 Voting strategies External Black box voting Internal Shape voting Voting Paprika Image Fireworx Txt 2 Salt & Pepper Txt 1 Txt 3 Bronze Dict Final Txt

54 Voting Image Paprika K.B. Recognize original segmentation Original segmentation: Every independent connected component is a character Good segmentation: recognize Bad segmentation: reject

55 Voting Image Paprika K.B. Recognize original segmentation Train adaptive classifier from original shapes Txt 1 Txt 2 Adaptive K.B.

56 Voting Image Paprika K.B. Recognize original segmentation Train adaptive classifier from original shapes Txt 1 Txt 2 Adaptive K.B. Recognize broken and joined shapes Dict Try several segmentations Loop if unrecognizable

57 Voting Image Paprika K.B. Recognize original segmentation Train adaptive classifier from original shapes Txt 1 Txt 2 Adaptive K.B. Recognize broken and joined shapes Dict Train adaptive classifier from ugly shapes

58 Voting Image Paprika K.B. Recognize original segmentation Train adaptive classifier from original shapes Txt 1 Txt 2 Adaptive K.B. Recognize broken and joined shapes Dict Train adaptive classifier from ugly shapes Try several segmentations Loop if unrecognizable Recognize more broken and joined shapes Txt 3

59 Voting strategies Voting Image Fireworx Salt & Pepper Paprika Txt 1 Txt 1 ~60% gain Txt 3 Bronze Final Txt Dict

60 OCR Internals Main tasks of an OCR system: Image acquisition Layout recognition Text recognition User assisted correction By the user s random editing... Pop-up verifier Manual Training By proofreading of doubtful words Result exportation

61 OCR Internals Main tasks of an OCR system: Image acquisition Layout recognition Text recognition User assisted correction By the user s random editing... By proofreading of doubtful words Correct: User dictionary Changed: IntelliTrain Remember trained characters Apply them on following pages Result exportation

62 IntelliTrain Recognized word: sorneuüng

63 IntelliTrain Recognized word: Fixed word: sorneuüng something

64 IntelliTrain Recognized word: Fixed word: sorneuüng something

65 IntelliTrain Recognized word: Fixed word: Substitutions found: sorneuüng something m rn thi Uü

66 IntelliTrain Recognized word: sorneuüng Fixed word: something Substitutions found: m rn thi Uü Perform automatically: Learn image pattern and substitution info Find similar substituted ( blue ) text on actual page Match against pattern of substitution and correct Find such errors on following pages, too

67 OCR Internals Main tasks of an OCR system: Image acquisition Layout recognition Text recognition User assisted correction Result exportation Combine pages into a Document Header / Footer recognition Page numbers Hyperlinks (e.g. See Table 20 ) Save results

68 OCR Internals Main tasks of an OCR system: Image acquisition Layout recognition Text recognition User assisted correction Result exportation Combine pages into a Document Save results doc file Speech synthesizer

69