Xiaolin Li and Dit-Yan Yeung. Hong Kong University of Science and Technology. Clear Water Bay, Kowloon, HONG KONG

Transcription

1 On-line Handwritten Alphanumeric Character Recognition Using Feature Sequences Xiaolin Li and Dit-Yan Yeung Department of Computer Science Hong Kong University of Science and Technology Clear Water Bay, Kowloon, HONG KONG Internet: Fax: Abstract. In this paper we present an approach in which an on-line handwritten character is characterized by a sequence of dominant points in strokes and a sequence of writing directions between consecutive dominant points. The directional information is used for character preclassication and the positional information is used for ne classication. Both preclassication and ne classication are based on dynamic programming matching. A recognition experiment has been conducted with 62 character classes of dierent writing styles and 21 people as data contributors. The recognition rate of this experiment is 91%, with 7.9% substitution rate and 1.1% rejection rate. The average processing time is 0.35 second per character on a MHz personal computer. 1 Introduction With the development of digitizing tablets and microcomputers, on-line handwriting recognition has become an area of active research since the 1960s [3]. One reason for this is that on-line handwriting recognition promises to provide a dynamic means of communication with computers through a pen-like stylus, not just a keyboard. This seems to be a more natural way of entering data into computers. The problem of on-line handwriting recognition can be dened in various ways. Variables in the problem denition include character set, writing style, and desired recognition rate. In general, each problem denition lends itself to dierent algorithmic approaches, which in turn make use of dierent features for classication [1, 2, 3, 4]. In this paper, we present an approach to on-line handwritten alphanumeric character recognition based on sequential features. In particular, points in strokes corresponding to local extrema of curvature are detected. These points correspond to the minima of curvilinear velocity of the pen-tip movement in the delta log-normal theory [6]. In addition, the mid-point between two consecutive points that correspond to curvature extrema or pen-down/pen-up locations is also used to represent a local minimum of angular velocity. We refer to all these points as dominant points in strokes. In our system, an on-line handwritten character is characterized by a sequence of dominant points in strokes and a sequence of writing directions be-

2 tween consecutive dominant points. The directional information of the dominant points is used for character preclassication and the positional information is used for ne classication. Both preclassication and ne classication are based on dynamic programming matching using the idea of band-limited time warping. Our recognition process consists of several stages: 1) data preprocessing, 2) feature extraction, and 3) character classication. Details of these stages are discussed in the following sections. 2 Data Preprocessing and Feature Extraction 2.1 Data Preprocessing In the current context, a handwritten stroke refers to the locus of the pen tip from its pen-down to the next pen-up position. It can therefore be described as a sequence of consecutive points on the x?y plane: S = p 1 p 2 pl, where p 1 and pl are the pen-down and pen-up points, respectively. Based on this representation, a handwritten character can then be described as a sequence of strokes C = S 1 S 2 SN. Since handwritten characters often have large variations in size and position, it is necessary to normalize the input data to facilitate subsequent processing. In our system, data normalization is performed by scaling each character both horizontally and vertically such that it is tightly bounded by a W H box with the top-left corner of the box at the origin o(0; 0). Following this normalization step, stroke smoothing and linear interpolation are performed so that each stroke can be represented by a chain code sequence D = d 1 d 2 dl?1. Figure 1(a) and (b) illustrate these ideas. o 5 NW 6 N 7 NE x d l?1 d l?k 4 W E 0 p l a l v l y SW 3 S SE 1 2 (a) direction code d l d l+k?1 (b) exterior angle and contour angle (c) example of feature extraction Fig. 1. Data preprocessing and feature extraction 2.2 Feature Extraction Feature Types The stroke-based features used in our system are dominant points in strokes and direction primitives between dominant points.

3 Dominant points refer to points of the following types: (a) pen-down and penup points; (b) points corresponding to local extrema of curvature; and (c) midpoints between two consecutive points of type (a) or (b). A direction primitive refers to one of the eight chain-code directions: E, SE, S, SW, W, NW, N, and NE (see Figure 1(a)). It represents the writing direction from a dominant point to the next one. Dominant Points Based on the chain coding scheme, consecutive exterior angles and contour angles formed by pairs of arrows along the stroke can be de- ned as shown in Figure 1(b). In Figure 1(b), the exterior angle al at point pl is formed by the pair of arrows dl?1 and dl, and is located on the left-hand side of the arrows. The value of al can be obtained easily by table lookup. Denoting the sequence of exterior angles in a stroke as A = a 2 a 3 al?1 and performing low-pass ltering on A, one can segment the stroke into a sequence of convex/concave/plain regions. The contour angle vl at pl is dened within a support region and its value is estimated by averaging angles alk, where for k = 1; 2; : : : ; K, alk is formed by the pair of arrows dl?k and dl+k?1. Denoting the sequence of contour angles in the stroke as V = v 2 v 3 vl?1, one can easily obtain the maximum within a convex region and the minimum within a concave region. All such maxima and minima constitute the local extrema of curvature along a stroke. More details of the above technique can be found in [5]. After detecting the extreme points, a mid-point between two consecutive points of type (a) or (b) along a stroke is then located to approximate the point of local minimum in angular velocity. The pen-down and pen-up points, the local extrema of curvature and these mid-points together then constitute the dominant points in a stroke. Direction Primitives Based on the dominant points extracted, a direction primitive can be dened as a vector from a dominant point to the following one, after quantization into one of the eight directions E, SE, S, SW, W, NW, N, and NE. Feature Sequences After feature extraction, a character C can be represented as a sequence of dominant points and a sequence of direction primitives. 2.3 Other Considerations There are a few other special considerations in our system design. First, dierent writing areas are designated for dierent character sets (Chinese, English, and numerals). Second, ledger regions are used for English ascender-descender judgement. Third, if a large portion of a stroke is part of a circle or a very smooth arc (i.e. no curvature extreme in that portion), we identify the mid-point of that region as a pseudo extreme of curvature.

4 2.4 An Example Figure 1(c) shows an example of the letter `B'. The original handwriting is located in the designated writing area with its discrete data sampled by the digitizing tablet displayed in the top-right region. In the lower region, the two preprocessed strokes are displayed, with the local extrema of curvature marked. Each stroke is also characterized by a sequence of direction primitives. The rst stroke has three dominant points (no curvature extrema) with sequence `22'. The second stroke has nine dominant points (three extreme points) with sequence ` '. Since this character is written in the area designated for (lowercase or uppercase) English letters, and the upper portion of the rst stroke is above the mid-level of the top ledger region, it is therefore categorized as an English letter with ascender. 3 Character Classication 3.1 Dynamic Programming for Elastic Matching Band-limited Time Warping Time warping is a useful technique for nding the correspondence between two strings (sequences). Given two strings, many time warps are possible. A cost (or gain) function can be dened to evaluate each warp. If we assume that there is only local variation and deformation, we can limit the extent of possible positional shift of each symbol. This results in a reduction in the number of possible warps that need to be investigated using dynamic programming. Dynamic Programming Algorithm Dynamic programming is a useful technique for nding the optimal path from one node to another in a graph. Suppose G is a matching graph with nodes in N levels (such as the graph in Figure 2(b)) and each node in the graph is associated with an index and a cost value. Let v 1 be the source and vt be the sink. Starting from the source there are a number of paths leading to the sink, with each path associated with a total cost value. Denoting the cost value of node vi as cvi and the minimum partial cost (over all possible partial paths) from node vi to the sink vt as pvi, the minimum total cost from the source to the sink can then be determined as follows: 1. Initialization step: 2. Induction step: pvi = cvi + pvt = cvt (1) min hvi;vji2g where hvi; vji denotes a directed edge in G. fpvj g (2) The algorithm for dynamic programming based on a gain measure can be derived in a similar manner.

5 virtual source N-1 source 1 2 N M-1 virtual sink. band M node (p,q) denotes matching di p with dr q (a) Matching graph Gs(DI ; DR) node (p,q) denotes matching pi p with pr q (b) Matching graph Gd(PI; PR) sink Fig. 2. Matching graphs 3.2 Preclassication Similarity Between Two Sequences Let DI and DR be the sequences of direction primitives of an input character CI and a reference character CR, respectively: DI = di 1 di 2 dim?1 (3) DR = dr 1 dr 2 drn?1 (4) Without loss of generality, assume that M N. Using the idea of band-limited time warping, one can construct a matching graph Gs(DI ; DR) with nodes in N+1 levels, as shown in Figure 2(a). As shown in the gure, a direction primitive in DR is allowed to have only one match in DI, but a direction primitive in DI may have multiple matches in DR. Furthermore, virtual source and sink nodes are introduced so that matching of the two ends is not enforced. The gain at a node (p; q) can be interpreted as the similarity between dip and drq, which is determined as (see Figure 1(a)) s(dip ; d rq) = 8 < : 1 dip = drq 0:6 jdip? d rq j = 1 or 7 0 otherwise The gains at the source and the sink are set to zero. The maximum total gain Gmax from the source to the sink can be found by dynamic programming as discussed above. The similarity between DI and DR is then dened as S(DI ; DR) = G max (6) N? 1 Candidate Classes Suppose CI and CR belong to the same character set, and they have the same ascender-descender property if the character set is the set of English letters. CR is said to be a candidate class for CI if where TS is a threshold. (5) S(DI ; DR) TS (7)

6 3.3 Fine Classication Distance Between Two Sequences Let PI and PR be the sequences of dominant points of the input character CI and a reference character CR, respectively: PI = pi 1 pi 2 pim (8) PR = pr 1 pr 2 prn (9) Using the idea of band-limited time warping, a matching graph Gd(PI ; PR) with nodes in N levels can be constructed as shown in Figure 2(b). Unlike in the matching graph Gs(DI ; DR), a dominant point in either sequence may have multiple matches in the other sequence. The cost at a node (p; q) is dened as the Euclidean distance between dominant points pip and prq. The minimum total cost Cmin from the source to the sink can be found by dynamic programming. With this cost, the distance (or dissimilarity) between PI and PR is dened as D(PI ; PR) = C min N (10) Character Classication Let fprg be the set of dominant point sequences corresponding to the set of candidate classes fcrg obtained from the preclassication step. The input character CI is classied as C R 2 fc Rg if P R corresponding to C R satises P R = arg minfd(pi ; PR)g (11) PR and D(PI ; P R) TD (12) where TD is a threshold. Otherwise, the input character will be rejected. 4 Recognition Experiment Our system has been implemented on a MHz personal computer running Microsoft Windows 3.1, with a WACOM digitizing tablet as input device. The width and height of each normalized character are set to W = 108 and H = 128, and the thresholds for preclassication and ne classication are set to TS = 0:6 and TD = 32. Character Classes and Writing Styles We use 62 character classes with dierent writing styles in our experiment, i.e., numerals 0-9, uppercase letters A-Z, and lowercase letters a-z. Our writing templates are derived from the Italian manuscript style and some other writing styles.

7 Data Collection and Reference Set In order to evaluate the performance of our system, 21 participants were invited to contribute handwriting data for our recognition experiment. The data for each participant are stored in three data les, one each for numerals, uppercase and lowercase letters written in Italian manuscript style and some other styles. The rst le contains 20 numeral instances (0-9, 0-9). The second les contains 52 uppercase letters (A-Z, A- Z). The third les contains 52 lowercase letters (a-z, a-z). Figure 3 shows two examples of such data les. Fig. 3. Uppercase and lowercase letters written by participants All the 124 characters written by the rst participant were used as reference patterns to form the initial reference set. Reference set evolution was then performed by using the data from ve randomly chosen participants. After running this procedure, 56 new reference patterns were selected to give a total of 180 patterns in the reference set. This reference set was then used in the recognition of the characters written by the other 15 participants. Recognition Results To collect more statistics about the performance, the data from 15 participants were tested. The recognition results are summarized in Table 1. From the table, one can see that the recognition rate for numerals is the highest while that for lowercase letters is the lowest. The overall recognition rate is 91.0%, with 7.9% substitution rate and 1.1% rejection rate. If we consider the three best candidates, the hit rate is 97.1%. The average time for processing one data le of 20 numerals is about 6 seconds, while that for 52 English letters is about 18 seconds. Thus, the average processing time is estimated to be about 0.35 second per character on a MHz personal computer. 5 Conclusion and Remarks In this paper, we have presented an approach to on-line handwritten alphanumeric character recognition. In our approach, an on-line handwritten character

8 Numeral Uppercase Lowercase Overall Total % % % % 1st % % % % 2nd 8 2.7% % % % 3rd 0 0.0% 1 0.1% % % Others 1 0.3% % % % Rejected 2 0.7% % 5 0.6% % Table 1. Recognition results is represented by a sequence of dominant points in strokes and a sequence of writing directions between consecutive dominant points, which form the basis for character classication. An advantage of this approach over other methods comes from the fact that our character matching processes are elastic and hence can tolerate local variation and deformation. Besides, dominant points are quite easy to extract using the technique described in [5]. Moreover, our approach can handle large alphabets (such as Chinese characters) due to its fast preclassication. However, since our approach is based on sequential handwriting signals, it is intrinsically stroke-order dependent. Acknowledgement The research work reported in this paper has been supported by a Sino Software Research Centre (SSRC) Research Award (SSRC 94/95.EG11) and a Research Infrastructure Grant (RI 92/93.EG08) of the Hong Kong University of Science and Technology. References 1. D.D. Kerrick and A.C. Bovik. Microprocessor-based recognition of handprinted characters from a tablet input. Pattern Recognition, vol.21, no.5, pp.525{537, M.S. El-Wakil and A.A. Shoukry. On-line recognition of handwritten isolated Arabic characters. Pattern Recognition, vol.22, no.2, pp.97{105, C.C. Tappert, C.Y. Suen and T. Wakahara. The state of the art in on-line handwriting recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.12, no.8, pp.787{808, C.K. Lin, K.C. Fan and F.T. Lee. On-line recognition by deviation-expansion model and dynamic programming matching. Pattern Recognition, vol.26, no.2, pp.259{ 268, X. Li and N.S. Hall. Corner detection and shape classication of on-line handprinted Kanji strokes. Pattern Recognition, vol.26, no.9, pp.1315{1334, R. Plamondon. Handwriting generation: the delta lognormal theory. Proceedings of the Fourth International Workshop on Frontiers in Handwriting Recognition, pp.1{10, This article was processed using the LaT E X macro package with LLNCS style