Hierarchical Decomposition of Handwriting Deformation Vector Field Using 2D Warping and Global/Local Affine Transformation

Transcription

1 th International Conference on Document Analysis and Recognition Hierarchical Decomposition of Handwriting Deformation Vector Field Using 2D Warping and Global/Local Affine Transformation Toru Wakahara Faculty of Computer and Information Sciences Hosei University Kajino-cho, Koganei-shi, Tokyo Japan Seiichi Uchida Faculty of Information Science and Electrical Engineering, Kyushu University 744 Motooka, Nishi-ku, Fukuoka-shi Japan Abstract This paper addresses the basic problem of how to extract, describe, and evaluate handwriting deformation from not the statistical but the deterministic viewpoint. The key ideas are threefold. The first idea is to apply 2D warping to extraction of handwriting deformation vector field (DVF) between a pair of input and target images. The second idea is to hierarchically decompose the DVF by a parametric deformation model of global/local affine transformation. As a result, the DVF is expressed by a series of deformation components each of which is characterized by a window size of local affine transformation. The third idea is interrupting of the series of deformation components to obtain natural, reasonable handwriting deformation. Experiments using the handwritten numeral database IPTP CDROM1B show that 31.1% of the handwriting DVF is expressed by global affine transformation, and the subsequent few local affine transformations successfully discriminate natural handwriting deformation from unnatural one. 1. Introduction To accomplish the aim of most accurate handwriting recognition we will adopt statistical or probabilistic pattern recognition techniques, including sophisticated discriminant functions, neural networks, support vector machines or kernel methods [5], [1]. However, we can say that the problem of what is handwriting deformation remains unsolved in the sense that the statistical description of handwriting deformation in a high-dimensional feature space cannot deepen our real understanding of handwriting deformation. From this viewpoint, 2D image elastic matching based on deterministic deformation models have been proposed [7]. The tangent distance [6] and global affine transformation (GAT) correlation method [8] tried to extract handwriting deformation by means of parametric linear transformation, but, could not deal with nonlinear distortion. On the other hand, 2D warping methods via dynamic programming (DP) [4] realized pointwise correspondence between input and target images using non-parametric, rather loose matching constraints, but, suffered from excessive, unnatural This paper proposes a new, promising technique to extract, describe, and evaluate linear/nonlinear handwriting deformation in a deterministic, parametric manner. The key ideas are threefold; generation of handwriting deformation vector field (DVF) using a 2D warping technique between a pair of input and target images, hierarchical decomposition of the DVF by a parametric deformation model of global/local affine transformation, and determination of natural, reasonable handwriting deformation by interrupting the series of deformation components. Successful experimental results using the handwritten numeral database IPTP CDROM1B open new possibilities of further improvements in handwriting recognition accuracy. 2. Generation of Handwriting DVF by 2D Warping We deal with the problem of optimal matching between a pair of handwritten input and target images in grayscale. Here, we denote input and target images by F = f f (r) g and G = f g(r) g, respectively, where r = (x; y) t ; (1» x» M; 1» y» N ), is a loci vector in a 2D image plane, and f (r) and g(r) denote grayscale values at the point of r. To generate handwriting deformation vector filed (DVF) between F and G we have to solve the problem of determining a vector mapping function fi (r) that specifies a /09 $ IEEE DOI /ICDAR

2 pointwise correspondence between f (r) and g(fi (r)). Asa result, a deformation vector d(r) for F at the point of r is defined by d(r) fi (r) r: (1) The objective function Ψ(fi j F ; G) for determining an optimalmappingfunction fi (r) is as follows. Ψ(fi j F ; G) = f jf (r) g(fi (r))j + r krf (r) rg(fi (r))k g! min for fi: (2) To solve this optimization problem we adopt DP-based piecewise linear 2D warping (PL2DW) method [4]. It is to be noted that DP guarantees global optimization of 2D warping. Figure 1 shows two typical versions of 2D warping. Fig. 1(b) corresponds to the above-mentioned PL2DW. 3. Hierarchical decomposition of DVF by global/local affine transformation In this section we propose hierarchical decomposition of the handwriting DVF by a parametric deformation model of global/local affine transformation to tell whether the extracted handwriting deformation is really natural or not. The first step is extraction of global affine transformation component from the DVF. The second step is expansion of the residual DVF containing nonlinear deformation into a series of deformation components each of which is characterized by a window size of local affine transformation. Finally, we introduce interrupting of the series of deformation components to obtain only natural, reasonable handwriting deformation Extraction of global affine transformation component from DVF (a) (b) The aim of extracting global affine transformation component from DVF is to approximate the DVF by optimal affine parameters as closely as possible. Here, we denote global affine transformation by a 2 2matrix,A,representing rotation, scale-change, and shearing, and a 2D translation vector b: A= a00 a 01 b0 ; b = : (3) a 10 a 11 b 1 Next, weintroducetheobjectivefunction Φ(A; b) to determine optimal affine parameters given by Figure 1. Typical versions of 2D warping. (a) Approximate rubber sheet. (b) PL2DW. In Fig. 1(a), the mappings of 4-adjacent pixels are mutually constrained by monotonicity and continuity that approximates ideal rubber-sheet However, the optimization of this version is an NP-hard problem. On the other hand, Fig. 1(b) has been proposed as computationally tractable version of Fig. 1(a) by piecewise linear approximation. Namely, each column of F is fitted to G as a broken line with one corner, called pivot. As a result, the pointwise correspondence between F and G can be determined by linear interpolation except for the pivot and boundary. PL2DW can compensate for fully two-dimensional deformation although it adopts piecewise linear approximation. In this sense, PL2DW is considered a very powerful tool for generating the handwriting DVF. Conversely speaking, PL2DW is likely to suffer from excessive, unnatural Φ(A; b) = = k d(r) (A r + b r) k2 r k fi (r) (A r + b) k2 r! min for A and b: (4) By setting the derivatives of Φ(A; b) with respect to each of six unknown parameters, a 00 ;a 01 ;a 10 ;a 11 ;b 0,andb 1, equal to zero, we obtain a set of simultaneous linear equations. We can solve these simultaneous linear equations by conventional techniques such as Gaussian elimination [2]. We know that the optimization problem of Eq. (4) is exactly equivalent to the least-squares criterion and its solution is easy-to-follow. Here, we denote the determined optimal affine transformation by A GAT and b GAT. Then, the decomposition of the DVF is given by d(r) = (A GAT r + b GAT r) + (fi (r) (A GAT r + b GAT )) = d GAT (r) + ~ d 0 (r); (5) 1142

3 where ~ d 0 (r) represents the residual component of the DVF after the optimal global affine transformation. Moreover, we define the resultant loci vector obtained by applying A GAT and b GAT to r by s 0 (r) =A GAT r + b GAT : (6) Namely, we consider that the pixel with the grayscale value of f (r) moves from r to s 0 (r) by global affine transformation Expansion of residual DVF by local affine transformation In this subsection, we propose expansion of the residual DVF by a parametric deformation model of local affine transformation. Here, the first key idea is that each local affine transformation is characterized by one parameter of a window size. The second key idea is that expansion of the residual DVF into the series of deformation components is obtained by using successive local affine transformations with decreasing window sizes. Here, we define local affine transformation by a 2 2 matrix A(r) and a 2D vector b(r) at the point of r. However, it is to be noted that A(r) and b(r) applies not to r but to s 0 (r) of Eq. (6). Furthermore, a single parameter of a window size is introduced by the following objective function Φ(A(r); b(r) j ) that determines optimal local affine parameters of A(r) and b(r). Φ(A(r); b(r) j ) = r 0 w 0(r 0 jr) ~ d 0 (r 0 ) (A(r)s 0 (r 0 )+b(r) s 0 (r 0 )) 2! min for A(r) and b(r);! w 0 (r 0 jr) = exp ψ k s 0(r 0 ) s 0 (r) k 2 : (7) 2 2 By introducing a Gaussian window function w 0 (r 0 jr) around the point of s 0 (r), optimal A(r) and b(r) are determined to approximate the residual DVF in the neighborhood of s 0 (r) as closely as possible. In particular, the parameter specifies the spread of the Gaussian window function around s 0 (r), and controls the stiffness of matching by local affine transformation. Namely, the smaller the value of is, the softer the matching by local affine transformation is. Because the optimization problem of Eq. (7) results to the weighted least-squares criterion, its solution is straightforward in the same way as the solution of Eq. (4) in 3.1. However, we have to calculate a set of local affine transformation fa(r); b(r)g, while global affine transformation is asinglepairoffa; bg. By denoting the determined local affine transformation by A LAT (r) and b LAT (r), we obtain the decomposition of the residual DVF by local affine transformation as follows. ~d 0 (r) = (A LAT (r) s 0 (r) +b LAT (r) s 0 (r)) + ~ d 1 (r) = d LAT (r) + ~ d 1 (r); ~d 1 (r) = fi (r) (A LAT (r) s 0 (r) +b LAT (r)): (8) Here, ~ d 1 (r) denotes the residual component of the DVF after global affine transformation and the subsequent single local affine transformation. Now, we propose to decompose the DVF into a series of deformation components using successive local affine transformations with decreasing window sizes given by 1 > 2 > > k > > K : (9) The objective function for determining the kth local affine transformation is given as follows. Φ(A k (r); b k (r) j k )= r 0 w k 1(r 0 jr) ~ d k 1(r 0 ) (A k (r)s k 1(r 0 )+b k (r) s k 1(r 0 ))! min for A k (r) and b k (r) for 1» k» K; 2 w k 1(r 0 jr) = exp ψ k s k 1(r 0 ) s k 1(r) k 2 : 2 k 2 (10) The optimization problem of Eq. (10) can also be solved according to the weighted least-squares method. Finally, we obtain the expansion of the DVF given by d(r) = d GAT (r) + K k=1 d LAT k (r) = s k (r) s k 1(r);! d LAT k (r) + d ~ K (r); s k (r) = A LAT k (r) s k 1(r) +b LAT k (r): (11) Here, ~ d K (r) denotes the residual component of the DVF after global affine transformation and a set of the subsequent K local affine transformations. This realizes a hierarchical decomposition of the DVF. Figure 2 shows a hierarchical decomposition of a deformation vector Interrupting of decomposition of DVF Now, we have a hierarchical decomposition of the DVF by a parametric deformation model of global/local affine transformation expressed by Eq. (11). The next problem is 1143

4 LAT d2lat d3 ~ s2 s3 d3 d1lat s 1 s0 dgat Last, each DVF is decomposed into the series of deformation components by hierarchical application of global/local affine transformation. Here, we set the value of at. Hence, we have a total of seven deformation components extracted from the DVF: one global affine transformation and six local affine transformations. Figure 3 shows examples of hierarchical decomposition of DVF. τ (r) d(r) r Figure 2. Hierarchical decomposition of a deformation vector. how to extract only natural, reasonable handwriting deformation from the DVF. To resolve this problem, we propose to interrupt the series of deformation components of the DVF using an appropriate selection of decreasing window sizes in the successive local affine transformations. As stated in 3.2, the smaller the value of window size is, the softer the matching by local affine transformation is. Therefore, to make a big difference to the successive local affine transformations in the ability of absorbing nonlinear deformation, we decrease window sizes as follows. (12) Actually, we consider that the window size is no less than one. Hence, by setting the value of at, we obtain a total of window sizes. Then, we interrupt the series of deformation components of the DVF at the very window size that absorbs too much nonlinear or unnatural deformation. 4. Experimental results We use the handwritten numeral database IPTP CDROM1B [3]. This database contains binary images of handwritten digits divided into two groups of 17,985 samples for training and 17,916 samples for test. First, position and size normalization by moments is applied to each binary image so that the center of gravity of black pixels is located at the center of the image and the average distance of black pixels from the center of the image is set at the predetermined value of. Then, we transform all of binary images into grayscale images by using a mean filter repeatedly ten times and set the image size at pixels. Hence, we have and in the notation used in Section 2. Second, we generate a single target image per digit by averaging each category s training samples. Third, we generate the DVF between each of 17,916 test samples and its correct target image by PL2DW. (a) (b) (c) (d) (e) (f) (g) Figure 3. Examples of hierarchical decomposition of DVF. (a) Input images. (b) Global affine transformation. (c) - (e) Successive local affine transformations with = 16, 8, and 4. (f) PL2DW. (g) Target images. From Fig. 3, it is first found that PL2DW-superimposed input images of Fig. 3(f) are likely to suffer from unnatural, excessive matching between input and target images. This is mainly because matching constraints are rather loose al1144

5 though the DP-based PL2DW itself guarantees global optimization subject to those constraints. Also, it is clear that global affine transformation cannot compensate for nonlinear handwriting deformation as shown in Fig. 3(b). Furthermore, it is important to note that the successive local affine transformations should be interrupted at a moderately large local window size, e.g. eight, in order to avoid excessive Next, we investigate changes in the average norms of residual deformation vectors through the process of hierarchical decomposition of DVF. Figure 4 shows changes in the average norms of residual deformation vectors via global/local affine transformation, where LATk (1» k» 6) corresponds to k. From these results, we can say that the proposed method provides a most effective and powerful means for extracting and decomposing handwriting deformation from a pair of input and target images and discriminate natural deformation components from unnatural ones. 5. Conclusion It is very interesting and still challenging to extract, describe, and evaluate handwriting deformation from not the statistical but the deterministic viewpoint. This paper proposed one powerful solution; the DPbased PL2DW generates the DVF between a pair of input and target images as a global optimization problem, and the successive global/local affine transformations decompose the DVF to describe handwriting deformation in a parametric manner. Experiments using the handwritten numeral database IPTP CDROM1B showed that the proposed method successfully decomposed the DVF and provided a substantial clue for discriminating natural deformation components from unnatural ones. Future work is to combine this deterministic method with statistical techniques to further improve handwriting recognition accuracy. References Figure 4. Changes in the average norms of residual deformation vectors via global/local affine transformation. From Fig. 4, it is first found that the average norms of residual deformation vectors monotonically decrease for all digit categories via global/local affine transformations. It is secondly found that a fair amount of the original norms of deformation vectors is absorbed via global affine transformation by an average of 31.1%. Last, it is clear that small norms less than 1.0 after several successive local affine transformations exhibit excessive From Fig. 4, LAT3 with the window size 3 =8is considered the best result that absorbs only natural, reasonable handwriting deformation. The residual norm after LAT3 is on average 28.7% of the original one. It is very interesting that the best window size of local affine transformation is around the size of ρ (= 12:5) used in the size normalization stated at the beginning of this section. [1] C.-L. Liu, K. Nakashima, H. Sako, and H. Fujisawa. Handwritten digit recognition: benchmarking of state-of-the-art techniques. Pattern Recognition, 36: , [2] Mathematical Society of Japan. Encyclopedic Dictionary of Mathematics. MIT Press, Cambridge, MA., [3] K. Osuga, T. Tsutsumida, S. Yamaguchi, and K. Nagata. IPTP survey on handwritten numeral recognition. IPTP Research and Survey Report, R-96-V-02, [4] M. Ronee, S. Uchida, and H. Sakoe. Handwritten character recognition using piecewise linear two-dimensional warping. In Proc. of. Sixth Int. Conf. on Document Analysis and Recognition, pages 39 43, Seattle, Sept [5] M. Shi, Y. Fujisawa, T. Wakabayashi, and F. Kimura. Handwritten numeral recognition using gradient and curvature of gray scale image. Pattern Recognition, 35: , [6] P. Simard, Y. LeCun, and J. Denker. Efficient pattern recognition using a new transformation distance. Advances in Neural Information Processing Systems, 5:50 58, [7] S. Uchida and H. Sakoe. A survey of elastic matching techniques for handwritten character recognition. IEICE Trans. Inf. & Syst., E88-D: , [8] T. Wakahara, Y. Kimura, and A.Tomono. Affine-invariant recognition of gray-scale characters using global affine transformation correlation. IEEE Trans. Pattern Anal. Machine Intell., PAMI-23: ,