On-line handwritten digit recognition based on trajectory and velocity modeling

Transcription

1 Available online at Pattern Recognition Letters 29 (2008) On-line handwritten digit recognition based on trajectory and velocity modeling Monji Kherallah a, *, Lobna Haddad a, Adel M. Alimi a, Amar Mitiche b,1 a Research Group on Intelligent Machines (REGIM), University of Sfax, ENIS, BP W 3038, Sfax, Tunisia b Telecommunications (INRS), University of Quebec, 800, de la Gauchetière Ouest, Suite 6900, Montreal, Quebec, Canada H5A 1K6 Received 11 February 2007; received in revised form 19 September 2007 Available online 8 December 2007 Communicated by L. Heutte Abstract The handwriting is one of the most familiar communication media. Pen based interface combined with automatic handwriting recognition offers a very easy and natural input method. The handwritten signal is on-line collected via a digitizing device, and it is classified as one pre-specified set of characters. The main techniques applied in our work include two fields of research. The first one consists of the modeling system of handwriting. In this area, we developed a novel method of the handwritten trajectory modeling based on elliptic and Beta representation. The second part of our work shows the implementation of a classifier consisting of the Multi-Layers Perception of Neural Networks (MLPNN) developed in a fuzzy concept. The training process of the recognition system is based on an association of the Self Organization Maps (SOM) with Fuzzy K-Nearest Neighbor Algorithms (FKNNA). To test the performance of our system we build 30,000 Arabic digits. The global recognition rate obtained by our recognition system is about 95.08%. Ó 2007 Elsevier B.V. All rights reserved. Keywords: Handwriting modeling; Stroke overlapping; Elliptic trajectory modeling; Beta velocity modeling; Digit recognition 1. Introduction * Corresponding author. Tel.: ; fax: addresses: monji.kherallah@enis.rnu.tn (M. Kherallah), haddad.lobna@webmails.com (L. Haddad), adel.alimi@enis.rnu.tn (A.M. Alimi), Amar.Mitiche@inrs-emt.uquebec.ca (A. Mitiche). 1 Tel.: Poste 2010; fax: The automatic recognition of handwritten characters is of paramount importance in applications where handwriting is the desirable input channel, such as in form filling. This importance presents a technological revolution in man machine interfaces (keyboard, mouse, etc.). The field of handwriting recognition can be split into two different approaches. The first one deals with the recognition of handwriting in the form of an image and it is termed offline. In this instance, only the completed character or word is available. The second approach called on-line, concentrates on the recognition of handwriting captured by a tablet or similar touch-sensitive device, and uses the digitized trace of the pen to recognize the symbol. In this area, the recognizer will have access to the x and y coordinates as a function of time which has temporal information about how the symbol was formed. It is the on-line approach that has been taken into account in this work. In most handwriting recognition systems existing in PDA (personal digital assistant), the processing steps of segmentation, recognition, decision-making and post processing are serially taken. These systems usually use the resources exhaustively in each stage of the serial engine. Every stage is tuned to maximize global performance. Pre-processing is primarily related to character processing operations such as normalization to remove irregularities of handwriting. Recognition is the application of classification algorithms. Independent contextual information is used as a /$ - see front matter Ó 2007 Elsevier B.V. All rights reserved. doi: /j.patrec

2 M. Kherallah et al. / Pattern Recognition Letters 29 (2008) post-processing step to recognize and enhance the handwritten words. Most architectures of handwriting recognition system are monotonically linear and based on stochastic models like Hidden Markov models (Bellegarda et al., 1994; Ben Amara and Belaid, 1996; Cho and Kim, 2004; Hafsa et al., 2004). These models are probabilistic and need powerful calculators and a considerable calculation time. However, in the structural method, a set of basic are usually selected as primitives (El-Sheik and El-Taweel, 1990; Hafsa et al., 2004; Heutte et al., 1998; Kherallah et al., 2002, 2004; Morasso et al., 1993; Simard et al., 1993), and stroke recognition is based on the use of certain geometrical features like line segment directions, length, order, number, relation, etc. These are also found to be useful for character recognition. Several properties of human handwriting movements were taken. Size and speed can be involuntarily varied without changing the shape of the velocity profile of the handwritten script (Uno et al., 1989; Viviani and Schneider, 1991). The originality of this paper deals with two fields of research. The first one presents a novel approach of the handwriting modeling system based on Beta elliptic approach. The second topic of our contribution deals with a hierarchical recognition system of digits based on an association of SOM, FKNN and MLPNN. The Beta elliptic representation is consisting of a combination between geometry and kinematics in handwriting generation movements (Bezine et al., 2003b; Kherallah et al., 2002, 2004; Viviani and Schneider, 1991). According to this method of modeling, the number of features per character can reach 63 dimensional feature vector. It depends on stroke number of trajectory. For such high dimensionality, pattern recognition techniques suffer from Digital Tablet Known digit Training data Acquisition and preprocessing MLPNN FKNN SOM Data base «digits» Beta-Elliptic model Fig. 1. A diagram bloc of the proposed system. Testing data the well-known curse of dimensionality phenomenon. This problem is resulting from the fact that the required number of labeled samples for supervised classification increases dramatically as a function of dimensionality (Kiviluoto, 1996). For reducing the problem of dimensionality, we propose a hierarchical neural network representation for the on-line recognition of handwritten digits. The training system is based on a fuzzy concept. In fact, because individual stages usually do not have all the information, we propose an association between the SOM and the FKNNA. The result obtained by the last classifiers will be used in the training process of MLPNN (see Fig. 1). This paper is organized as follows: Next section explains the details of the Beta elliptic representation. Section 3 is devoted to the recognition system architecture. Experimental results and the comparison of our method with the available methods in the literature are discussed in Section Trajectory modeling by Beta elliptical representation The basic role of the trajectory modeling is to boost the comprehension of handwriting generation and improve online recognition system. In literature, the study of hand movements was based on proposed models. Two general methodologies of handwriting modeling become apparent from the review of literature. The first methodology takes into consideration the computational models, which are based on using some features of Human handwriting movements such as velocity profiles and some relations between different aspects of the dynamic movement; such as curvilinear velocity, acceleration, length, etc. Such methodology includes oscillators models (Heutte et al., 1998; Ruiz-pinals and Lecolinet, 2000), which combine various velocity sinusoids to yield different movement shapes. The second methodology of the handwriting modeling is based on static approach such as geometric forms: curvature, direction, circular, etc. (Chan and Yeung, 1999; Connel and Jain, 2001; Flash and Hogan, 1985; Ruiz-pinals and Lecolinet, 2000; Sung and Wolfgang, 2001). The oscillation model of Hollerbach (1981) and the Hollerbach and Flash (1982) uses kinematic parameters such as velocities and amplitude of motion to represent the handwriting movement. Denier Vander Gon simulated the production of graphic patterns, representing different letters of alphabet by the timing of acceleration and deceleration of two orthogonal motor systems, the first is responsible for the excursions of writing trace parallel to the writing line (X axis), and the second is responsible for the excursions orthogonal to the first ones (Y axis) (Denier and Thuring, 1965). Optimization models refer also to such methodology (Chen et al., 1997; Flash and Hogan, 1985; Saltzman and Kelso, 1987; Sung and Wolfgang, 2001; Van Galen and Weber, 1998). Plamondon and Guerfali presented a handwriting model, which refers to the first methodology as mentioned previously (Guerfali and Plamondon, 1995; Guerfali and

3 582 M. Kherallah et al. / Pattern Recognition Letters 29 (2008) Plamondon, 1994; Plamondon, 1991; Plamondon, 1995; Plamondon et al., 1993; Plamondon and Alimi, 1997; Uno et al., 1989). Their model uses the delta-lognormal synergies. This name refers to the authors definition of the velocity of a muscle synergy as a Gaussian function of the movement parameters that vary logarithmically with time. They have been interested in the kinematic properties of handwriting generation process and omitted the relationship between kinematics and the involved handwriting trajectory. Plamondon and Guerfali (1998) suggest that timing is considered as the solely crucial factor in determining the trajectory shape. Using real handwriting data, it is obvious that some models perform better than others. Therefore, the decision of manipulating a proper model depends on the goal of the research. The trajectory/velocity modeling techniques were many years ago applied in the handwriting modeling field (Alimi and Plamondon, 1994; Alimi, 1997, 2002, 2003; Morasso et al., 1993; Plamondon, 1995; Plamondon et al., 1993; Plamondon and Alimi, 1997; Uno et al., 1989). Previously, modeling systems were mainly based on either trajectory or velocity features. Kherallah et al. (2002, 2004) present the first idea of combining the kinematics and geometry in the trajectory modeling. These studies considered neither the overlap of Beta signal nor the inflexion points which present a key role for the number determination. The number was based only on velocity signal extremum. Local extremum were not considered in (Bezine et al., 2003b and Kherallah et al., 2002, 2004); consequently, the weakness of this approach is that it cannot detect the smallest handwritten trajectory curvature or the catastrophe variability of the handwritten trajectory caused by involuntary psychological behavior of the writer. However, in this paper we proceed to locate the inflexion points of the handwritten trajectory. For each inflexion point of trajectory we attributed one Beta to the velocity signal (see Figs. 2 and 3). We also take into account the variability of the handwritten trajectory curvature. In (Kherallah et al., 2002) the modeling system was based on Beta-circular approach. The circular are superposed but there are not of overlapping shapes and cannot perfectly reconstruct the handwritten trajectory. While, in our work the modeling system is based on Beta elliptical approach in which the overlapping of Beta signal is taken into consideration. The novelty of our modeling approach is the combination of trajectory-based features (elliptic parameters) and Fig. 2. Curvilinear velocity and acceleration signal of handwritten digit.

4 M. Kherallah et al. / Pattern Recognition Letters 29 (2008) The curvilinear velocity of each individual obeys the Beta approach. So, the generation of a complex trajectory pattern is the result of an algebraic addition of velocity terms (see Eq. (2)). V ðtþ ¼ Xn i¼1 V i ðt t i Þ ð2þ Fig. 3. Beta modeling of the velocity trajectory. velocity based features (Beta function parameters). In fact, kinematic properties involved perform to joint angle trajectory which obeys an elliptic form. The parameters characterizing an elliptic trajectory are performed according to the Beta curvilinear velocity profile (Kherallah et al., 2004). According to the harmonic oscillator description of the muscle action involved in handwriting production, velocity profile of the cursive handwriting can be viewed as a sequence of overlapped Beta-functions Beta velocity modeling In our work, we consider that handwriting movement, like any other highly skilled motor process, is partially programmed in advance, and we suppose that movements are represented and planned in the velocity domain, since the most widely accepted invariant in movement generation is the Beta shape of the velocity profiles. In this context, the modeling of a complex trajectory pattern is the result of the activation of n neuromuscular subsystems characterized by an impulse response that is real, normalized, and non-negative. If n is sufficiently large, applying the central limit theorem, the global impulse response will converge to a Beta curve (Alimi and Plamondon, 1994; Alimi, 2002; Bezine et al., 2003a,b; Hafsa et al., 2004; Kherallah et al., 2002, 2004). The curvilinear velocity V(t) (see Eq. (1)) is then computed using a second-order derivative filter with finite impulse response. sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 dxðtþ V r ðtþ ¼ þ dyðtþ 2 ð1þ dt dt Handwritten scripts are, then, segmented into simple movements, as already mentioned, called, and are the result of a superimposition of time-overlapped velocity profiles. Many researchers use significant or critical points to split the pen-path into smaller entities. Commonly used significant points are local extrema in horizontal or vertical direction (Heutte et al., 1998), local extrema in velocity (Bezine et al., 2003a; Kherallah et al., 2004; Plamondon and Alimi, 1997; Uno et al., 1989), local extrema in curvature (Chen et al., 1997), and points of inflection (see Fig. 2). In our case, the inflection points trajectory with the maximum and the minimum of the velocity signals are located. These points are considered of significance. Based on the inflection points given by the acceleration of the pen movement, we attributed the overlapped form of Beta to the velocity extremum profile. Therefore, after being calculated, the velocity profile of handwriting will be modulated by the Beta signals (see Fig. 3). Consequently, the complete velocity profile of the neuromuscular system will be described by a Beta model as follows: ( p q ) t t 0 t 1 t bðt; q; p; t 0 ; t 1 Þ¼ t c t 0 t 1 t c if t 2½t 0 ; t 1 Š ð3þ 0 elsewhere where p, q are intermediate parameters, which have an influence on the symmetry and the width of Beta shape (see Fig. 4b). t 0 is the starting time of Beta function; t c is the instant when the curvilinear velocity reaches the amplitude of the inflexion point; t 1 is the ending time of Beta function; t 0 < t 1 2 IR and Fig. 4a. Beta signal of one stroke.

5 584 M. Kherallah et al. / Pattern Recognition Letters 29 (2008) Fig. 4b. Different shapes of the mono-dimensional Beta function. characterized by three parameters. These parameters are collected from the Beta function. Each elementary component called stroke is also characterized in the space domain by three statistical parameters. These parameters globally reflect the geometric properties of the set of muscles and joints used in a particular handwriting movement. The parameters a and b are respectively the half dimensions of the large and the small axes of the elliptic shape. X 0 and Y 0 are the cartesian coordinates of the elliptic center relative to the orthogonal reference (A, X and Y). As shown in Fig. 5, the angle h defines the deviation of the elliptic portion according to the orthogonal reference (A, x and y). From two points (A, B) of one stroke, we calculate the (h, a and b) parameters. These points A and B correspond respectively to the minimum and the maximum values of the velocity profile. C is rightly before the point B, we join a tangent line crossing B and C. By an orthogonal projection on B, we get the center O and the different axes a and b of the ellipsis (see Fig. 6). Consequently, a single movement, also called stroke is represented in the space and velocity domains by a curvilinear velocity starting at time t 0 at an initial point, and moving along an elliptic path. The latter obeys a variable curvature C. This curvature is not a constant one as it was proposed by few models in this direction by literature (Denier and Thuring, 1965; Flash and Hogan, 1985; Hollerbach, 1981; Plamondon et al., 1993; Plamondon and Alimi, 1997; Wada et al., 2001). Elliptic model is a static model. In the spatial state, the trajectory is represented by a sequence of elliptic arcs (Kherallah et al., 2004). The elliptic equation is written as follows: X 2 a þ Y 2 2 b ¼ 1 ð5þ 2 Each elliptic arc is drawn by the calculation of (h, a and b) parameters. Some examples are presented in Fig. 7. Fig. 5. Final Beta model representation of the velocity signal. t c ¼ p t 1 þ q t 0 p þ q One Beta signal can be represented as shown in Fig. 4a. The parameter k is the amplitude of the Beta signal (k =1 in this case). The result of Beta model reconstruction of velocity signal is shown in Fig Elliptical trajectory modeling Dynamic data contains the information about how the shapes were written. Static data conveys the result of the writing process, i.e. what has been written. In this paragraph, we focus on the description of static model. In fact, executed from an arbitrary starting position are ð4þ 2.3. Combination between Beta and elliptical models As shown previously, the Beta elliptic model considers a simple movement as the response to the neuromuscular system, which is described by an elliptic trajectory and a Beta velocity profile. In our approach of modeling, a simple stroke is approximated by a Beta profile in the dynamic domain which corresponds in turn to an elliptic arc in the static domain such that the distance AO is the half-large axe dimension a. As reported by Viviani et al. for human drawing curves, the instantaneous tangential velocity of the hand decreases as the curvature increases, and then A and B, which are characterized by minimum tangential velocity, correspond to the maximum of curvature in the static domain. Consequently, a stroke is characterized by seven parameters. The first four Beta parameters (t 0, t 1, p and k) reflect the global timing properties of the neuromuscular networks

6 M. Kherallah et al. / Pattern Recognition Letters 29 (2008) Fig. 6. Elliptical arc representation. Fig. 7. Examples of digit elliptic representation. involved in generating the movement, whereas the last three elliptic parameters (h, a and b) describe the global geometric properties of the set of muscles and joints recruited to execute the movement.

7 586 M. Kherallah et al. / Pattern Recognition Letters 29 (2008) On-line recognition of handwritten digits The recognition process is divided into pre-processing steps and subsequent classification. Facing up to the complex problems of the handwriting recognition, the use of the multiple, hybrid and an association of classifier systems proves an increasing interest during the last years (Aksela and Laaksonen, 2005; Chiang and Gader, 1997; Hafsa et al., 2004; Hebert et al., 1998; Ianakiev and Govindaraju, 2000; Kittler et al., 1998; Lam and Suen, 1999; Prevost et al., 2005; Suen and Tan, 2005; Xu et al., 1992). Based on their complementarities, the association of classifiers increases the performance of the recognition system while limiting the error bound to the use of a unique classifier. The use of the multiple classifier systems benefits from the strong points of every classifier. Among these systems, we mention the neuro-fuzzy approach. It is about a neuronal approach developed in a fuzzy concept (Alimi, 1997, 2003; Chiang and Gader, 1997; Gader et al., 1995a,b, 1997; Gomez Sanchez et al., 1998; Keller et al., 1985; Kittler et al., 1998). Digit recognition was studied 10 years ago and, conveyed that the fuzzy approach enhance the classification performance Vuori and Laaksonen (2002). In our work, one of the main classification problem is the variability of the feature vector size (35, 42,..., 63) depending of each digit number of. Our new fuzzy architecture combined the neural fuzzy approach in a hierarchical way which offers a solution to the variability of feature vector size. An interesting comparative study was done by Gader and Keller in (1995, 1997) showing the interest of the fuzzy and neural networks approaches and their complementarities (Gader et al., 1995a,b, 1997). Since the work of Wang et al. (2000), it was proved that the performance of SVM and MLPNN is better than the KNN algorithm in case of big number of classes (Wang et al., 2000). Whereas, FKNN algorithm is specialized to discriminate between classes especially in the boundary zone which presents a confusion and inference. In our approach, we use a sequential version of multiple classifiers. The complementarities between the developed classifiers are explained in the following paragraphs. Our system is based on the use of neural networks developed in a fuzzy concept. The desired outputs of MLPNN are formed using SOM and FKNNA (see Fig. 8). Therefore, our system is about neuro-fuzzy networks based on SOM and FKNNA association used in the learning process. Input vectors Testing dataset Recognition rate / squared error 9 MLPNN 5 MLPNN Desired outputs FKNNA 5 Strokes 6 FKNNA Training process 9 Strokes FKNNA 5 Strokes FKNNA SOM 6 Training dataset (Semantic classes ) 9 Strokes SOM Fig. 8. Detail steps of the proposed system.

8 M. Kherallah et al. / Pattern Recognition Letters 29 (2008) According to Fig. 8, the first step of the training process of the recognition system consists in the use of the SOM algorithm. This method is applied in order to organize the input prototype vectors. The advantages of this method are: This algorithm is a powerful tool for giving the real classes number and not the semantic one (Kiviluoto, 1996; Mezghani et al., 2002; Morasso et al., 1993). In fact, it is an unsupervised algorithm. It makes a projection of the disordered input vectors on an organized map of different clusters and it is able to reduce the high dimensionality of our system to two dimensions. The result given by this algorithm will be used in the next stage. In the second step, we adjusted the means of clusters elements, and we calculated the degree of membership of data vectors to each cluster by the Fuzzy K-Nearest Neighbor Algorithm (FKNNA), which gives a more realistic description. Finally, we attributed the membership matrix obtained by FKNNA to the target of the MLPNN in the training process as a desired output of MLPNN. Finally, we applied the MLPNN to test the handwritten digits. Note that the pre-processing system reduces the complexity existing in the principal recognition system by a hierarchical representation as explained in the next paragraph. Then, we detail the different sequences of the proposed system of on-line handwritten digit recognition Pre-processing system For the handwriting, we used a Wacom 4 electronic digitizing tablet. The information collected from this tablet was represented as the raw data x(t) andy(t) and was sampled at 200 Hz. A smoothing operation is applied to the data provided by the tablet to eliminate the hardware imperfections, the trembles in writing, etc. A filtering step is necessary to eliminate duplicated date points by forcing a minimum distance between consecutive points. For this reason, we applied a Chebyshev second-order low-pass filter with a cut-off frequency of about 12 Hz. We used this filter because it has an acceptable stability in pass-band and the band of transition is narrow. Due to the variability of the handwriting, the vector size of digits is variable. In our case, the number of handwritten digits is limited between 5 and 9 (see Table 1). In fact, our method aims to represent our system into a hierarchical architecture. The complexity of the recognition system was reduced by the five subsystems development. Every subsystem was specialized in the same number of digits. All handwritten digits are classified by their number of, which was determined automatically from the curvilinear velocity signal as it is explained in the previous section, and the same handwritten digit can have a different number of. Therefore, we have a hierarchical representation of the digits pre-classification system. Table 1 shows five subsystems. Table 1 Classification of the database digit by their number Digits 3.2. Real class detection by self-organizing map The self-organizing map (SOM), first introduced by Kohonen (1990), is a powerful clustering and data presentation method. A SOM consists of a grid shaped set of nodes. The self-organizing feature maps (SOFMs) used with pyrolysis mass spectrometry (PyMS) data consists of a two-dimensional network of neurons arranged on a square grid. Each neuron is connected to its eight nearest neighbors on the grid. A node s activation level is defined as: sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi X n ðweight i input i Þ 2 ð6þ i¼0 Five sub-system Six sub-system Seven sub-system Eight sub-system Nine sub-system This is a simply Euclidean distance between the points represented by the weight vector and the input vector in n-dimensional space. Thus, a node whose weight vector closely matches the input vector will have a small activation level, and a node whose weight vector is very different from the input vector will have a large activation level. The node in the network with the smallest activation level is deemed to be the winner for the current input vector (Kherallah et al., 2004; Keller et al., 1985; Kohonen, 1990; Mezghani et al., 2002; Morasso et al., 1993; Samervuo and Kohonen, 1999). In our system, 10 digits have not got the same number of (see Table 1), therefore we proposed a hierarchical representation of 5 SOM subsystems. Under such an environment, we proceeded to determine the number of clusters existing in every map. The output of the hierarchical system is represented by 5 maps. Each map contains the different clusters of characters having the same number of. The map size was fixed to neurons. Therefore, we obtained 3600 neurons. The optimized number of the training algorithm iterations is 500. If we take an example of a five subsystem, we have four semantic classes which are ( ), whereas Fig. 9 shows five classes which are the real classes ( ) existing in the map, one class of 0, two classes of 1, one class of 4 and one class of 6.

9 588 M. Kherallah et al. / Pattern Recognition Letters 29 (2008) Fig. 9. Example of five Map representation. As a result, the SOM algorithm which is an unsupervised algorithm, gives us the real classes number and not the semantic one. The output information given by the SOM algorithm will be used in the next stage. Note that the output of these algorithms will be injected to the target of MLPPNN. The SOM and the FKNNA are also used only in the learning process of our recognition system Membership assignment of the training data set to real classes by the Fuzzy K-Nearest Neighbor Algorithm The fuzzy logic has been conceived to adapt a special technique that consists in human thought, therefore, it is applicable in several domains. It is proved as a straightforward success in the sectors of automatic device, robotics and artificial intelligence where the information to treat is vague and imprecise, even in the engineering domain, management and making decision (Bezdek, 1981; Chiang and Gader, 1997; Gader et al., 1995a,b, 1997; Gomez Sanchez et al., 1998; Govindaraju and Ianakiev, 2000; Keller et al., 1985). Sometimes, the alphanumerical digits are ambiguous when read out of context. To minimize these difficulties, crisp classification is often replaced by fuzzy classification (Alimi, 1997, 2002; Bezdek, 1981; Chiang and Gader, 1997; Keller et al., 1985). The FKNNA was designed by Keller et al. (1985). Nearest neighbor classifier is chosen because it requires relatively low memory requirements and it is a non parametric classifier. The idea is to assign membership based on percentage of characters in each class among the neighbors of a training sample. The result of this algorithm will be used in the training process of the MLPNN outputs. The class memberships are assigned to the sample, as a function of the sample s distance from its KNN training samples. P k j¼1 u ij 1=kx x j k 2 ðm 1Þ u i ðxþ ¼ P ð7þ k j¼1 1=kx x jk 2 ðm 1Þ The parameter m is a scaling value, it takes a value limited between 1 and 2. The memberships of the training samples U ij can be defined in several ways. The crispest way is to give them complete membership in their own class and non-membership in all other classes. A more fuzzy alternative is to assign the training sample memberships based on the distance from their main class. After calculating the memberships of the training samples, we attribute this result to the target of a MLPNN as a training phase. The algorithm used in this task is presented as follows: 1. Compute distance from data point to labeled samples 2. If KNN has not been found yet, then 3. Include data point. 4. Else, if a labeled sample is closer to the data point than 5. Any other KNN, then 6. Replace the farthest with the new one. 7. Compute membership 8. Repeat for the next labeled sample.

10 M. Kherallah et al. / Pattern Recognition Letters 29 (2008) The maximum number of iteration cycles can be used as a termination criterion. We optimized m and k values in two times. First, we fixed the parameter m, and for each value of k, we calculated the recognition rate obtained by the neuro-fuzzy system. At the second time, we took the best value of k as constant and for each value of m, we calculated the recognition rate obtained by the neuro-fuzzy system (see Figs. 10a and 10b). According to these figures, we fixed respectively the m and k available values to the FKNNAs associated to different subsystems as the best values Classification by the MLPNN system Our system is composed of 33 neural networks of type OCON (one class one network). Each class of handwritten digit corresponds to one of OCON. The architecture of OCON is presented in Fig. 11. Because of the multi-variability of the handwriting, every digit has not got the same number. One or several writers can also write the same digit in different shapes. All handwritten digits of our database are composed of n. This number is variable and takes a value from 5 to 9. Each stroke is represented by seven features (t 0, t 1, p, k, h, a and b). According to Table 2, we obtained five sets. Each set is called subsystem and is composed of N clusters of digits, which have the same number of. For each subsystem, we constructed N neural networks i.e OCON, with N = real class number of all maps. Our system contains 33 neural networks. Consequently, we developed 33 OCONs. (Rec %) (Rec %) (k) Fig. 10a. Parameter m optimization of FKNN (m) Fig. 10b. Parameter k optimization of FKNN. 5 Strokes (k=5) 6 Strokes (k=4) 7 Strokes (k=2, k=5) 8 Strokes (k=6) 9 Strokes (k=5) (m=2) 5 Strokes 7 Strokes (k=2) 7 Strokes (k=5) 6 Strokes 8 Strokes 9 Strokes Stroke 1 Stroke 2 Stroke n Input layer Hidden layer Fig. 11. OCON architecture. Table 2 From semantic classes to real classes Classes Semantic classes Real classes Five subsystem Six subsystem Seven subsystem Output layer Eight subsystem Recognized digit Nine subsystem Digit 0 Digit 0 Digit 1 Digit 2 Digit 2 Digit 1 Digit 1 Digit 2 Digit 3 Digit 3 Digit 4 Digit 4 Digit 3 Digit 5 Digit 5 Digit 6 Digit 6 Digit 4 Digit 7 Digit 7 Digit 5 Digit 8 Digit 9 Digit 8 Digit 9 Digit 9 Digit 0 Digit 0 Digit 1 Digit 2 Digit 2 Digit 1 Digit 1 Digit 2 Digit 2 Digit 3 Digit 1 Digit 1 Digit 3 Digit 3 Digit 5 Digit 4 Digit 1 Digit 3 Digit 5 Digit 7 Digit 6 Digit 4 Digit 4 Digit 5 Digit 9 Digit 6 Digit 5 Digit 7 Digit 5 Digit 8 Digit 8 Digit 9 Digit 9 The targets of these OCONs are fixed by the degree of membership between the data vector X i and the cluster N obtained from the FKNNA. To evaluate the handwriting modeling design and the recognition system paradigm, we calculated the recognition rate which is a standard measure of performance for character recognizers. We also calculated the squared average (SA) error which is a standard measure used in training neural networks. Standard wisdom is that high recognition rates and low SA error values are good. In this task, we prepared the data set test, we calculated the recognition rate by introducing the test prototypes to the neural network system, and we forced the networks to decide at the character level. The calculation of the SA error will be explained later.

11 590 M. Kherallah et al. / Pattern Recognition Letters 29 (2008) Digit database formulation Database for character recognition algorithms is of fundamental interest for the training of recognition method based on neural networks. As it was explained in Section 3.1, the segmentation of digits was based on stroke detection that divides our system on some subsystems. Every subsystem was specialized on digit having the same number of as it was shown in Table 1. However, we cannot find the required repartition number of digits as it was shown in Table 2, neither in UNI- PEN nor in IRONOFF datasets. So, using the (SOM FKNN MLP) and UNIPEN database is not possible. The dimension of the representative feature vector is high, for each digit we need seven features per stroke x number of parameters (from 35 parameters to 63 parameters) per digit. Consequently, for every neural network we need 400 prototypes (300 for the learning system and 100 prototypes for the testing system). For this reason, we developed our own database which contains 30,000 digits. Twenty four participants were invited to contribute to the development of the handwritten digits data. The data for each participant are stored in one data file. When producing the data file, each participant was asked to write a set of all digits ( samples of digits). We imposed to the writer just to write 10 times the same digit, from 0 to 9 in the same page. One page contains 100 digits. He asked to prepare only one page per day. We have collected 30,000 digits in total. More than half of them are regularly written. The remaining ones are those either with noise in the data, poorly written or deliberately written in strange and unusual ways. About two thirds of the writers were male, about 90% were right handed, the youngest writer was 8 years old, and the oldest was 66. In the online domain, the forms have been sampled with a spatial resolution of 200 dpi and a sampling rate of 100 points/s (Wacom UltraPad A4) and were stored using the UNIPEN format (Guyon et al., 1994). 4. Experimental results and discussions To test the performance of our recognition system, we divided our data base into two parts, 2/3 was used for the training system and 1/3 for the testing system. We have performed experiments to evaluate the modeling system and also the on-line recognition system of the digits. Our recognition system is summarized in three levels: SOM, FKNN and MLPNN. In the first step of the training system, we developed all maps using the SOM algorithm in order to obtain the real classes of every subsystem. The configuration of the SOM algorithm used is as follows: 1. Weights initialization: we used a random initialization of the map. 2. Map lattice: in our application, we considered the rectangular lattice. 3. Neighborhood function: we chose the Gaussian function as being a function of neighborhood. 4. Map shape: we considered the cylindrical shape that generates a junction between the left and right extremities of the map. According to the variability of the handwriting existing between writers, the same digit can be written in different shapes. Thanks to SOM algorithm development, the real classes detection of the same digit was established. In our work we proceeded to separate these real classes by a right line and calculate the antecedent input vectors. Note that there are sometimes an inference (confusion and ambiguity) in the boundary zone of classes. The use of FKKNA gives the membership degree values of every digit class to the other classes existing in the same map (subsystem). This information will be used as a desired output of MLPNN. Regarding the five stroke map, we did not have a discrimination level problem of the real classes. Therefore, there were no overlaps between the different real classes (see Fig. 9). However, the distribution of digits in the other maps (six, seven, eight and nine ), showed difficulties resulting from classes overlap. In fact, samples far from the center, which tend to fall on the boundaries of classes, are error-prone. To resolve this problem, we proceeded to separate the semantic classes by a distribution of every class alone in one map. After the organization of different maps, we obtained two essential pieces of information. The first one conveys the real number of classes. The second one deals with the identification of prototype groups corresponding to different classes figured in the map. As a result, the SOM, which is an unsupervised algorithm, gives us the real number of classes and not the semantic one (see Table 2). In the second step, we developed FKNNAs in order to make a fuzzy membership matrix of the different real classes obtained by SOM algorithm. According to Figs. 10a and 10b, we optimized the best values of k and m parameters in order to maximize the performance of the FKNNA. The membership matrix calculated was used as a desired output of the MLPNN. Each neural network was trained by the standard back propagation algorithm. The performance of this algorithm is very sensitive to the proper setting of the training rate. The back propagation training parameters (The training rate: l = 0.01; the momentum factor: a = 25; and the iterative number for training: epochs = 4000) are adjusted to trade off speed and accuracy. The maximum number of iteration cycles can be used as a termination criterion. To test our system, we calculated the global recognition rate and the global squared error. We proceeded to calculate the recognition rates by an affectation to a prototype to test the class that had the best membership degree. The results of the recognition rate are presented in Table 3.

12 M. Kherallah et al. / Pattern Recognition Letters 29 (2008) Table 3 Recognition rates of the hierarchical system Subsystems Recognition rate (%) Global recognition rate (%) The global recognition rate obtained is about 95.08%. In this context, we forced our recognition system to give us a crisp result. Thanks to the membership degrees of character to every class assignment, the introduction of the fuzzy logic gives us a wealth of information at the level of the ambiguous representation. Whereas, when using the maximum membership degree and ignoring the others which means decide about only one class of character, then, the network commits errors of classification. For this reason, we considered the average squared error as criterion to give more precise results of the neuro-fuzzy system (Gader et al., 1995a). In this task, we applied the SOM algorithm to test the set of digits in order to visualize the real classes. Then, we applied the FKNNAs to test these prototypes in order to find the membership degree of every prototype test according to the real classes existing in the training map. These degrees are supposed to be the desired output of every OCON. We calculated the average squared error E i (see formula (9)) of every subsystem and the average squared error E g (see formula (10)) of the global system. This error permits to evaluate the performance of our system. The training error E r is committed by the MLPNN after the training process. The equations used for this task are: E r ¼ky d y r k 2 P OCON nber j¼1 Er j E i ¼ prototypes test nber E g ¼ P Sub sys nber i¼1 P OCON nber j¼1 Er j P Sub sys nber i¼1 prototypes test nber ð8þ ð9þ ð10þ In formula (8), y d is the desired output (membership vector) and y r is the real vector found by the MLPNN. The results of the average squared errors are presented in Table 4. To summarize, the learning process which was based on SOM and FKNNA association was done only one time and it was made in a fuzzy concept. Note that the output Table 4 Average squared error results Subsystems Five Five Six Six Seven Seven Eight Eight Nine Nine Squared error (%) Squared average error (%) of the learning process was integrated as a desired output of MLPNN. After the learning process, if we want to recognize one digit, we use only the MLPNN and the output gives the membership degree of this digit to the other digits existing in the same subsystem. When testing our system, the global average squared error obtained is about In Table 5, some recognition systems of the handwritten characters were shown. Several modeling approaches are based on geometric features only, while others are based on kinematic features. Furthermore, several recognition systems were based either on one classifier (unsupervised algorithm as SOM algorithm Mezghani et al., 2002), or a hybridization between HMM and MLPNN approach (Hafsa et al., 2004). Compared to these systems, our proposed system is based on a combination of trajectory (geometrical features) and velocity (kinematic features) modeling. In this instance, our recognition system revolves around a mixture of unsupervised and supervised algorithms in a sequential architecture of (SOM, FKNN and MLPNN algorithms). The studies of Hebert et al. (1998) consist of a fuzzy representation of feature extraction of digits (the feature was based on stroke direction: horizontal, vertical, positive or negative oblique) and recognition system based on Kohonen and MLP classifiers, it was demonstrated that the recognition result (95%) proves an interesting methodology of the modeling system. In our system, we used the fuzzy concept in the recognition system. Compared with Hafsa s recognition system (Hafsa et al., 2004) which use the same database and based on hybridization of MLP and MMC, the recognition rate is about 93%, our recognition system which based on SOM, FKNNA and MLP association performs better and gives 95.08% as a recognition rate. We have already made a new experimentation in order to compare (Beta circular/beta elliptic) modeling system keeping the same classification process (MLP) and we proved that the Beta elliptic representation gives a better performance than the Beta-circular representation. The recognition rate that obtained by using the Beta-circular approach is about 93.20%; however, using the Beta elliptic approach, this rate was increased to 94.14%. We have also made a new experimentation in order to compare the classification system (SOM FKNN MLP/ MLP) keeping the same features and database. The association of SOM FKNN MLP proves a better performance than the use of only MLP. In fact, the use of the one classifier MLP gives only 94.14% as a recognition rate. In order to validate our modeling approach, we also conducted another new experimentation consisting of the use of SVM classifier and UNIPEN dataset of digits keeping the modeling system which was based on Beta elliptic approach. The recognition rate obtained is about 94.78%. Compared to the Ratzlaff studies which consist of methods, report and survey for the comparison of diverse isolated

13 592 M. Kherallah et al. / Pattern Recognition Letters 29 (2008) Table 5 Some results from the literature on handwritten character recognition Authors Method Accuracy Notes Gader et al. (1995a) Comparison of Crisp and Fuzzy character neural networks in handwritten word recognition 86.24% on uppercase and 83% on lowercase Hebert et al. (1998) Combination of SOM and MLP 95% on 13,000 digits collected from UNIPEN database Mezghani et al. (2002) Ratzlaff (2003) Sel Organization Map, SOM, it is an unsupervised Algorithm Methods, report and survey for the comparison of diverse isolated character recognition results on the Unipen database 88.38% on 24,000 samples of Arabic letters for testing and 5000 Arabic letters samples for training 95% confidence limits are given where available Features vector reaches 100 parameters based on pixel normalization and descender ascender detection The modeling system based on fuzzy representation Features vector based on elliptic Fourier descriptors Six UNIPEN isolated character subsets. 1a digits, 1b uppercase, 1c lowercase, 1d punctuation and other symbols, 2 mixed and 3 mixed, with 10, 26, 26, 32, 94, and 94 classes, respectively Hafsa et al. (2004) Hybridizing of NN and HMM 93% on 24,000 prototypes of digits Database was collected by REGIM members Prevost et al. (2005) Hybrid generative/discriminative classifier for unconstrained character recognition 99.1% on 14,000 digits divided into three sets: 8000 digits for training, 4000 digits for testing and 2000 digits for cross valid Digits are collected from UNIPEN database. The sequence of (x, y) coordinates is resampled with 20 points per stroke Aksela and Laaksonen (2005) Elastic matching is used as dynamic time warping (DTW) 79.98% on characters (uppercase and low case letters and digits) Database is divided into three groups (9961 prototype characters, 8047 for evaluation and 80,077 for testing) The member classifier was based on stroke-bystroke distances between the given characters and prototypes character recognition results on the UNIPEN database (Ratzlaff, 2003), our modeling approach gives a similar results and that proves an acceptable performance of the Beta elliptic modeling. The execution time needed to recognize one digit is variable. It was estimated between 1 and 3 s. Whereas, the learning process presents a considerable execution time, it takes about (10 min to 1 h:13 min). It depends on subsystem number. These experiments were done on Intel CoreTM Solo processor T1350 (1.86 GHz, 533 MHz FSB, 2 MB L2 cache). Note that all algorithms were developed by MATLAB language which is an interpreted language. Eventually, we compiled all our algorithms to C++ language and the execution time was reduced to (13 35 ms). It depends on input vector size of the tested digit. 5. Conclusion In this paper, the originality of our work resides in the development of a new method for handwritten trajectory modeling based on inflection point detection, the overlapped form of Beta signals and the elliptic arcs. Our contribution also deals with a novel approach of on-line recognition of the Arabic handwritten digit. Novel architecture of recognition system was developed, with which acceptable recognition accuracy was reached. Our modeling system gave good results and proved that the Beta elliptic representation is a powerful tool for the handwriting modeling. Our system achieves a recognition rate of 95.08% and an SA error of 0.065%, which makes it an acceptable performance system. The interaction between the different classifiers (SOM, FKNN and MLPNN algorithms) contributes to increase the performance of our recognition design. The complementarities between these classifiers were demonstrated. These developed modeling and recognition methods can be extended to any size of feature vectors and can be applied not only in digits but also in the Arabic, Latin, Chinese, etc. characters. Note that the real classes taken into consideration enhance the performance of our recognition system result. As a future work, we can improve the Beta elliptic approach by the Fuzzy Beta elliptic approach. The fuzzy approach will be integrated in the angle determination of the elliptic shape. The developed recognition system can be extended to the global recognition of the cursive words. Acknowledgements The authors acknowledge the financial support of this work by grants from the General Direction of Scientific Research and Technological Renovation (DGRSRT), Tunisia, under the ARUB program 01/UR/11/02. References Alimi, M.A., Plamondon, R., Analysis of the parameter dependence handwriting generation models on movement characteristics. In:

14 M. Kherallah et al. / Pattern Recognition Letters 29 (2008) Faure, C., Keuss, P., Lorette, G., Vinter, A. (Eds.), Advances in Handwriting and Drawing, Europia, Paris, pp Alimi, M.A., The Beta system: Toward a change in our use of neurofuzzy systems. Int. J. Manage., (Invited Paper). Alimi, M.A., Beta neuro-fuzzy systems. TASK Quart. J. 7 (1), (Special issue on neural networks edited by W. Duch and D. Rutkowska). Alimi, M.A., An evolutionary neuro-fuzzy approach to recognize on-line Arabic handwriting. In: Proc. Internat. Conf. on Document Analysis and Recognition: ICDAR 97, Ulm, Germany, August, pp Alimi, M.A., Recognition of on-line handwritten characters with the Beta fuzzy neural network. Proc. IJCNN 3, Aksela, M., Laaksonen, J., On adaptive confidences for critic-driven classifier combining. ICAPR 1, Bellegarda, J.E., Bellegarda, J.R., Kim, J.H., On-line handwritten character recognition using parallel neural networks. IEEE Trans. Systems. Man and Cybernet. 2, Ben Amara, N., Belaid, A., Printed PAW recognition based on planar hidden Markov models. In: ICPR 96, Austia, August Bezdek, J., Pattern Recognition with Fuzzy Objective Function Algorithms. Plenum Press, USA. Bezine, H., Alimi, M.A., Derbel, N., An explanation for the feature of a handwriting trajectory movement controlled by a Beta elliptic model. In: Proc. 7th Internat. Conf. on Document Analysis and Recognition ICDAR 03, Edinburgh, Scotland, UK, August, pp Bezine, H., Alimi, M.A., Derbel, N., A developmental theory of the relation between geometry and kinematics in handwriting. In: Proc. IEEE Internat. Conf. on Smart Systems and Devices SSD 03, Tunisia, March, pp Chan, K.F., Yeung, D.Y., Recognizing on-line handwritten alphanumeric characters through flexible structural matching. Pattern Recognition 32 (7), Chen, H., Agazzi, O.E., Suen, C.Y., Piecewise linear modulation model of handwriting. In: Proc. Internat. Conf. on Document Analysis and Recognition ICDAR 97, Germany, August, pp Chiang, J.H., Gader, P.D., Hybrid fuzzy-neural systems in handwritten word recognition. IEEE Trans. Fuzzy Systems 5, Cho, S.J., Kim, J.H., Bayesian network modeling of and their relationships for on-line handwriting recognition. Pattern Recognition 37 (2), Connel, S.D., Jain, A.K., Template-based on-line character recognition. Pattern Recognition 34 (1), Denier, V.G.J.J., Thuring, J.P., The guiding of human writing movements. Cybernetic 4 (2), El-Sheik, T.S., El-Taweel, S.G., Real-time Arabic handwritten character recognition. Pattern Recognition 23 (12), Flash, T., Hogan, N., The coordination of arm movements an experimentally confirmed mathematical model. J. Neurosci. 5, Gader, P., Mohamed, M., Chiang, J.H., 1995a. Comparison of Crisp and Fuzzy character neural networks in handwritten word recognition. IEEE Trans. Fuzzy Systems 3, Gader, P., Keller, J.M., Cai, J., 1995b. A fuzzy logic system for the detection and recognition of handwritten street numbers. IEEE Trans. Fuzzy Systems 3, Gader, P.D., Keller, J.M., Krishnapuram, R., Chiang, Jung-Hsien, Mohamed, M.A., Neural and fuzzy methods in handwriting recognition. IEEE Comput. Soc. 30 (2), Gomez Sanchez, E., Gago Gonzalez, J.A., Dimitriadis, Y.A., Cano Izquierdo, J.M., Lopez Coronado, J., Experimental study of a novel neuro-fuzzy system for on-line handwritten UNIPEN digit recognition. Pattern Recognition Lett. 19, Govindaraju, V., Ianakiev, K., Potential improvement of classifier accuracy by using fuzzy measures. IEEE Trans. Fuzzy Systems 8 (6), Guerfali, W., Plamondon, R., The delta log-normal theory for the generation and modelling of cursive characters. In: Proc. ICDAR 95, Montréal, Québec, pp Guerfali, W., Plamondon, R., Robust parameter extraction techniques for the delta lognormal model. Vis. Interf., , Banff. Guyon, I., Schomaker, L., Plamondon, R., Liberman, M., Janet, S., UNIPEN project of on line data exchange and benchmarks. In: Proc. Internat. Conf. on Pattern Recognition, IAPR-IEEE. Hafsa, W., Kherallah, M., Ben Jemaa, M., Ben Amara, E N., A hybrid approach of neural networks/hidden markov model for on-line recognition of the Arabic digits. In: Proc. IEEE Internat. Conf. on SCS 04. Mounastir, Tunisie, pp Hebert, J.F., Parizeau, M., Ghazalli, N., Une représentation floue régionale pour la reconnaissance en ligne des chiffres manuscrits, 1er colloque Internationale Francophone sur l écrit et le document, CIFED, Quebec (Canada), pp Heutte, L., Paquet, T., Moreau, J.V., Lecourtier, Y., Olivier, C., A structural/statistical feature based vector for handwritten character recognition. Pattern Recognition Lett. 19 (7), Hollerbach, J.M., Flash, T., Dynamic interactions between limb segments during lunar arm movements. Biological Cybernet. 44, Hollerbach, J.M., An oscillation theory of handwriting. Biological Cybernet. 39, Ianakiev, K.R., Govindaraju, V., Improvement of recognition accuracy using 2-stage classification. In: Proc. Internat. Workshop on Frontiers in Handwriting Recognition, Amsterdam. International Unipen Foundation, pp Kherallah, M., Njah, S., Alimi, M.A., Derbel, N., Recognition of on-line handwritten digits by neural networks using circular and Beta approaches. In: Proc. IEEE Internat. Conf. SMC 02, Hammamet, Tunisia, p. 26. Kherallah, M., Alimi, M.A., Derbel, N., On-line recognition of handwritten digits by self organisation maps using Elliptic and Beta representations. In: Proc. IEEE Internat. Conf. SCS 04, Mounastir, Tunisia, pp Keller, J.M., Gray, R., Givens, J.A.J.R., A fuzzy K-nearest neighbor algorithm. IEEE Trans. Systems Man Cybernet. 15 (4), Kittler, J., Hatef, M., Duin, R., Matas, J., On combining classifiers. IEEE Trans. Pattern Anal. Machine Intell. 20, Kiviluoto, K., Topology preservation in self-organizing maps. In: Proc. Internat. Conf. on Neural Networks (ICNN), pp Kohonen, T., The self-organizing map. Proc. IEEE 78 (9), Lam, L., Suen, C.Y., Optimal combination of pattern classifiers. Pattern Recognition Lett. 16, Mezghani, N., Mitiche, A., Cheriet, M., On-line recognition of handwritten Arabic Characters using a Kohonen neural network. In: Proc. IWFHR 02, Niagara-On-the-Lake, Canada, pp Morasso, P., Baberis, L., Pagliano, S., Recognition experiments of cursive handwriting with self-organizing networks. Pattern Recognition 26, Plamondon, R., A kinematics theory of rapid human movements. Part I: Movement representation and generation. Biological Cybernet. 72, Plamondon, R., On the origin of asymmetric bell-velocity profiles in rapid-aimed movements. In: Stelmach, G.E., Requin, J. (Eds.), Tutorials in Motor Neuroscience. Kluwer Academic Publishers, pp Plamondon, R., Alimi, M.A., Yergeau, P., Leclerc, F., Modelling velocity profiles of rapid movements: A comparative study. Biological Cybernet. 69, Plamondon, R., Alimi, M.A., Speed/accuracy trade-offs in target directed movements. Behav. Brain Sci. 20, Prevost, L., Oudot, L., Moises, A., Michel-Sendis, C., Milgram, M., Hybrid generative/discriminative classifier for unconstrained character recognition. Pattern Recognition Lett. 26,