Multiple Frame Integration for OCR on Mobile Devices

Transcription

1 Multiple Frame Integration for OCR on Mobile Devices Master s Thesis Georg Krispel Advisor: Horst Bischof December 12, 2016 Institute for Computer Graphics and Vision Anyline GmbH

2 Scene Text Recognition on Mobile Devices

3 Scene Text Recognition on Mobile Devices Scene Text Recognition Use Cases 3

4 Scene Text Recognition on Mobile Devices Cont d An almost orthogonal view is assumed A search window is introduced to improve user experience and spare searching for the text Sophisticated preprocessing steps Text recognition Possible repetition for validation 4

5 Scene Text Recognition on Mobile Devices Cont d An almost orthogonal view is assumed A search window is introduced to improve user experience and spare searching for the text Sophisticated preprocessing steps Text recognition Possible repetition for validation 4

6 Problems Low resolution images from outdated mobile phones Reflections and glares Poor lighting conditions 5

7 Problems Low resolution images from outdated mobile phones Reflections and glares Poor lighting conditions 5

8 Objectives Evaluate the possibilities of mitigating these effects to improve overall text recognition results Exploit multiple frames available in the camera stream and their redundant information (Multiple Frame Integration) Implement the resulting pipeline on mobile hardware 6

9 Scene Text Processing Pipeline

10 Assumptions Text is written on a nearly planar surface The surface is well textured Sufficiently smooth motion of the camera 8

11 Overview Detect text in keyframes and track it (respectively the underlying plane) over time Keyframe selection according to blurriness and text detection result Before text detection we rectified the underlying plane Utilizing multiple threads to outsource expensive tasks Asynchronous plane rectification and scene text detection Tracking of dominant plane in order to propagate text detection results to remaining frames Reinitialization after certain time respectively degeneration of tracking 9

12 Initialization Process Tracking Tracking Tracking Tracking & MFI Tracking & MFI Main Thread #0 #1 #8 #9 #10 #0 #0 Text Detection Thread Plane Rectification Text Detection Pipeline Initialization Process 10

13 Initialization Process Cont d Pipeline Processing Example 11

14 Initialization Process Cont d Pipeline Processing Example 12

15 Modules A modular design ensures the possibility of exchanging the different parts of the pipeline: Visual Tracking Rectification Text Detection Multiple Frame Integration 13

16 Visual Tracking Feature based Good features to track and Kanade-Lucas-Tomasi (8, 13, 14) AKAZE features (1, 2) and FLANN matching (9) Intensity based refinement just for text patches Parametric image alignment using ECC (6) 14

17 Rectification Rectangular region localization and extraction (LocEx) module by Andreas Hartl et al. (7) M-Estimator Sample Consensus (MSAC) based vanishing point detection by Nieto et al. (12) 15

18 Rectification Rectangular region localization and extraction (LocEx) module by Andreas Hartl et al. (7) M-Estimator Sample Consensus (MSAC) based vanishing point detection by Nieto et al. (12) 15

19 Rectification Rectangular region localization and extraction (LocEx) module by Andreas Hartl et al. (7) M-Estimator Sample Consensus (MSAC) based vanishing point detection by Nieto et al. (12) 15

20 Scene Text Detection TextSpotter (TS) by Neumann et al. (11) Based on classification and grouping of Extremal Regions Stroke-Width-Transformation (SWT) by Epshtein et al. (5) 16

21 Multiple Frame Integration Text Recognition Recognition Result Fusion Image Enhancement Text Recognition MFI approaches 17

22 Multiple Frame Integration Cont d Image Enhancement Minimum Operator Integration method by Yi et al. (15) Result Fusion Voting for most frequent recognition 18

23 Impact of MFI Approaches on Overall Recognition Results

24 Datasets We assumed the use case of energy meter readings We tailored our pipeline to solely detect the respective numbers Just bright text on dark background Additional histogram based verification step Constrained bounding box dimensions 20

25 Datasets Exemplary frames of the evaluation datasets showing different types of energy meters and ground truth annotation 21

26 Datasets Cont d Video ID Light source max. Resolution No. of Frames Duration 1 Tungsten 768x :07 2 Daylight 768x :07 3 Flash 768x :19 4 Daylight 768x :42 5 Tungsten 768x :16 6 Tungsten 768x :07 22

27 Detection and Tracking Accuracy We utilized CLEAR-MOT Evaluation Framework (4) Multiple Object Tracking Precision (MOTP) Multiple Object Tracking Accuracy (MOTA) We compared our method with full tracking-by-detection approaches Thereby, subsequently occurring bounding boxes are associated by their overlap utilizing Munkres algorithm (10). 23

28 Detection and Tracking Accuracy Cont d Res. Method MOTP Misses FP rate MM MOTA 768x1366 NATIVE TS x854 NATIVE TS NATIVE SWT TS MSAC&TS LOCEX&TS KLT&MSAC&TS AK&MSAC&TS Hybrid KLT&MSAC&TS Multiple Object Tracking Precision and Accuracy 24

29 Runtime Device Laptop Shield Tablet Resolution Tracking Method Rectification, Detection Total 480x854 AKAZE x854 KLT Hybrid KLT x720 KLT Hybrid KLT Average time performance measurements in milliseconds 25

30 Reading Accuracy We extracted the text patches and utilized the Anyline Energy module 1 to read the meter readings from the current patch and the currently available integrated counterpart respectively we fused the preceding results. These recognition rates are compared

31 Reading Accuracy Cont d Single extracted frames sampled during a sequence of 62 frames compared to respective integration results. 27

32 Reading Accuracy Cont d Degenerated Multi-frame Integration over Time 28

33 Reading Accuracy Cont d Recognition rate ECC Hybrid SF MIN YI HIST Method The recognition rates using the single extracted frames and the different MFI methods 29

34 Reading Accuracy Cont d Resolution Single frame Minimum operator Yi integration Histogram voting 768x x x Hybrid Recognition rates 30

35 Conclusion & Outlook

36 Conclusion & Outlook We showed that our MFI approach is capable of achieving real-time performance with little optimization on mobile hardware The multi-thread detection and tracking approach can keep up with full detection approaches A distinct improvement of the recognition rates is possible Generally image enhancement integration methods require almost perfect image registration If text recognition is fast enough, result fusion methods should be preferred over the evaluated image enhancement approaches 32

37 Questions? 33

38 References I References [1] P. F. Alcantarilla, A. Bartoli, and A. J. Davison. KAZE features. In European Conference on Computer Vision, [2] P. F. Alcantarilla, J. Nuevo, and A. Bartoli. Fast explicit diffusion for accelerated features in nonlinear scale spaces. In British Machine Vision Conference, [3] D. L. Baggio, S. Emami, D. M. Escriva, K. Ievgen, N. Mahmood, J. Saragih, and R. Shilkrot. Mastering OpenCV with Practical Computer Vision Projects. Packt Publishing, Limited, [4] K. Bernardin and R. Stiefelhagen. Evaluating multiple object tracking performance: the CLEAR MOT metrics. EURASIP Journal on Image and Video Processing, 2008(1):1 10,

39 References II [5] B. Epshtein, E. Ofek, and Y. Wexler. Detecting text in natural scenes with stroke width transform. In Conference on Computer Vision and Pattern Recognition, pages IEEE, [6] G. D. Evangelidis and E. Z. Psarakis. Parametric image alignment using enhanced correlation coefficient maximization. Transactions on Pattern Analysis and Machine Intelligence, 30(10): , [7] A. Hartl and G. Reitmayr. Rectangular target extraction for mobile augmented reality applications. In International Conference on Pattern Recognition, pages IEEE, [8] B. D. Lucas, T. Kanade, et al. An iterative image registration technique with an application to stereo vision. In International Joint Conference on Artificial Intelligence, volume 81, pages ,

40 References III [9] M. Muja and D. G. Lowe. Fast approximate nearest neighbors with automatic algorithm configuration. International Conference on Computer Vision Theory and Applications, 2( ):2, [10] J. Munkres. Algorithms for the assignment and transportation problems. Journal of the Society of Industrial and Applied Mathematics, 5(1):32 38, March [11] L. Neumann and J. Matas. Real-time scene text localization and recognition. In Conference on Computer Vision and Pattern Recognition, pages IEEE, [12] M. Nieto and L. Salgado. Real-time robust estimation of vanishing points through nonlinear optimization. In SPIE Photonics Europe, pages International Society for Optics and Photonics,

41 References IV [13] J. Shi and C. Tomasi. Good features to track. In Computer Society Conference on Computer Vision and Pattern Recognition, pages IEEE, [14] C. Tomasi and T. Kanade. Detection and tracking of point features. School of Computer Science, Carnegie Mellon Univ. Pittsburgh, [15] J. Yi, Y. Peng, and J. Xiao. Using multiple frame integration for the text recognition of video. In International Conference on Document Analysis and Recognition, pages IEEE,