Effect of Hidden Layer Neurons on the Classification of Optical Character Recognition Typed Arabic Numerals

Transcription

1 Journal of Computer Science (7): , 008 ISSN Science Publications Effect of Hidden Layer Neurons on the Classification of Optical Character Recognition Typed Arabic Numerals Nidal F. Shilbayeh and Mahmoud Z. Isandarani Faculty of Information Technology for Graduate Studies, Middle East University for Graduate Studies, P.O. Box, Post Code: 60 Faculty of Science and Information Technology, Al-Zaytoonah Private University of Jordan, P.O. Box 9597, Post Code: 9, Amman-Jordan Abstract: Problem statement: The effect of varying the number of nodes in the hidden layer and number of iterations are important factors in the recognition rate. In this paper, a novel and effective criterion based on Cross Pruning (CP) algorithm is proposed to optimize number of hidden neurons and number of iterations in Multi Layer Perceptron (MLP) neural based recognition system. Our technique uses rule-based and neural networ pattern recognition methods in an integrated system in order to perform learning and recognition of dynamically printed numerals Approach: The study investigates the effect of varying the size of the networ hidden layers (pruning) and number of iterations (epochs) on the classification and performance of the used MLP. The optimum number of hidden neurons and epochs is experimentally established through the use of our novel Cross Pruning (CP) algorithm and via designing special neural based software. The designed software implements sigmoid as its shaping function. Results: Experimental results are presented on the classification accuracy and recognition. Significant recognition rate improvement is achieved using 000 epochs and 5 hidden neurons in our MLP OCR numeral recognition system. Conclusions/Recommendations: Our approach has a significant improvement in learning and classification of any numeral, character MLP based recognition system. Key words: Neural networ, MLP, pruning, pattern recognition, classification, OCR INTRODUCTION A maor problem in applying neural networs is specifying the size of the networ. Even for moderately size networs the number of parameters may become large compared to the number of data [-]. Artificial neural networs have been successfully applied to problems in pattern classification, function approximation, optimization, pattern matching and associative memories [0]. Multilayer feed forward networs trained using the bac propagation-learning algorithm is limited to searching for a suitable set of weights. This initiates the problem of selecting appropriate topology (number of hidden layers) and weights to solve the learning problem in hand. Two small networs are unable to adequately learn the problem while excessively large networs tend to over fit the training data and consequently result in poor generalization and wea performance. Most practical learning problems are nown to be computationally complex and hard to optimize. Networ pruning offers an excellent approach for dynamically determining an appropriate networ topology. Pruning techniques involves training different networ sizes and compare the performance in terms of classification and error. The process comprises methodical and consistence elimination or addition of neurons which subsequently removes or adds weights to the hidden layers of the system. Most of available systems capture inputs as a sequence of coordinate points, taing into account character blending and merging and the problem of characters that have close similarity to each other. This is solved via pre-processing of the characters prior to recognition, hence, performing a shape recognition process. The post-processing recognition process can be achieved using zones that define directions of travel, where characters are recognized as connected zones using a looup table for matching and classification. The primary tas of OCR recognition is to tae an input and correctly assign it as one of the possible put classes. This process can be divided into two Corresponding Author:Nidal F. Shilbayeh, Faculty of Information Technology for Graduate Studies, Middle East University for Graduate Studies, P.O. Box, Post Code:

2 J. Computer Sci., (7): , 008 general stages: feature selection and classification. Feature selection is critical to the whole process since the classifier will not be able to recognize from poorly selected features []. Some requirements for character recognition system design suggest themselves such as: To create a system which can recognize nonperfect numerals To create a system which can use both static and dynamic information components To perform recognition with image data presented at field rate or as fast as possible To recognize as quicly as possible, even before the full image sampling is completed To use a recognition method which requires a small amount of computational time and memory To create an expandable system which can recognize additional types of characters High recognition rate In the event of an error, non-recognition is implemented instead of miss-recognition, especially in critical conditions Recognizes inputs quicly Require minimal training In this study networ performance is examined as a function of both networ size and number of iterations. This is carried using a feed forward multi layer perceptron neural model. This system captures and intelligently recognizes numerals. There are many proposed methods for on-line recognition, which use a wide variety of pattern recognition techniques. Neural networs have been proposed as good candidates for character recognition. Studies have also been carried for OCR recognition, comparing techniques such as dynamic programming, Hidden Marov Models and recurrent neural networs [-6]. MATERIALS AND METHODS System design: Figure shows the used OCR MLP. In designing our OCR system the following factors are considered: Initial weights: Since we used a gradient decent learning algorithm that is nown to dynamically treat all weights in similar manner and to avoid a situation of no learning, random function is included in our algorithm. Such a function is specifically designed to guarantee that the networ will not end in global minima which will adversely affect the system performance and will result in massive errors and miss classifications. Learning rate: The learning rate is bounded due to the following: Two small a learning rate will cause the system to tae a very long time to converge To large a learning rate will cause the system to oscillate indefinitely and diverge Based on the above two mentioned factors our learning rate was chosen to be equal 0.5. Transfer function: Even though some literature indicates that anti symmetric functions will cause the system to learn faster, this research used the standard sigmoid function which is nown to be a stable function due to its range from 0-. All the above is considered in our research when hidden layer neurons and number of epochs are closely examined. Mathematical modeling: Figure shows our designed and implemented MLP neural engine used in our OCR system and characterized by the following three main equations: ( 0) in The put of the input layer i Input = () i Neural networ Adust Weight Fig. Used OCR MLP system Preparation Stage Preprocessing ( ) w i 6 50 ( ) w Fig. : OCR neural classification system Target Compare + Exp?input ( ) Recognition Process Postprocessing 579

3 J. Computer Sci., (7): , 008 ( ) ( 0) ( ) f = i i The put of the hidden layer () ( ) ( ) ( ) f = The put of the put layer () Using Eq. - we can rewrite Eq. to become: ( ) ( ) ( ) ( ) ( ) = f = f f iniw i.w i () Learning in an MLP is nown to occur via modification of weights as indicated in the following equations: ( ) (5) w = η t arg p ( ) (6) w = η t arg 0 i i p Where η the learning rate and p is is the pattern on which the equation is applied. It is clear that for the put layer the hidden layer appears as an input layer and for the input layer the hidden layer appears as an put layer. Equation 5 and 6 are obtained using the sigmoid function as the shaping function, which taes the following form: f ( input ) = (7) + exp input ( ) Applying Eq. 7 to Eq., and we obtain the shaped puts for the three layers used in our MLP. ( ) ( 0) layer 0 : i = ini (8) Applying Eq. 9, 0 and will produce all layers puts Using Eq. 5 and 6, the necessary weight modification values are obtained Adding the results of 5 and 6 to the original weights according to the following equations will update the weight matrix w t + = w t + w () i i i w t + = w t + w () Each time Eq. and are applied, a single cycle is completed (epoch). However there is a need to compute the sum squared error for each two layers and for the overall networ using the following equation E( w ) ( ) i = t arg () p RESULTS Table - show the effect of varying both number of hidden neurons and number of epochs on the recognition of Arabic numerals, while Table and 5 clearly shows classification accuracy and sum squared Table Recognition results using 00 epochs Reference Hidden Neurons ( ) ( ) layer : = ( 0) ( ) + exp i i i ( ) layer ( ) : = ( ) ( ) + exp i (9) (0) Calculation of weight adustments in our used MLP follows the following steps: 580 Table Recognition results using 000 epochs Reference Hidden Neurons

4 J. Computer Sci., (7): , 008 errors. Table 6 and 7 show the extracted weight matrices for both input-hiddenand hidden-put, which is made available via the specially Designed Brain Database (DBD) that automatically adds the new trained engine weights to its contents while eeping trac of all previously used weights. Table Recognition results using 0000 epochs Reference Hidden Neurons Table Percentage classification Percentage Percentage Percentage Hidden Classification classification classification Neurons 00 Epochs 000 Epochs 0000 Epochs DISCUSSION Traditionally, pruning is carried on hidden layers and hidden layer neurons however we introduce in our wor the principle of Cross Pruning (CP) or what we can call the pruning matrix whereby we carried pruning on both hidden layer neurons and number of epochs which serves as a two dimensional controlling matrix function for the design and implementation of MLP based recognition systems [5-,7-9]. Figure shows the effect of pruning and learning process on the classification accuracy and recognition rate of our MLP based OCR system. It is clear from the graphs that best accuracy and classification is obtained when using 000 epochs and 5 hidden neurons. It observed that using 00 epochs resulted in under learning and poor recognition of tested numerals with slow oscillations due to slow recognition time while at the other extreme of 0000 epochs instability and fast oscillations which indicates shorter recall time and over learning with slightly better recognition than 00 epochs case. Table 5 Classification and sum squared errors Sum squared Sum squared Sum squared Hidden error error error Neurons (00 Epochs) (00 Epochs) (00 Epochs) Table 6: Hidden-put weight matrix W :0 W :0 W :60 W 6:80 W 8:00 W 0:0 W :0 W :60 W 6:80 W 8:

5 J. Computer Sci., (7): , 008 Table 7: Input-hidden weight matrix W :80 W 8:60 W 6:0 W :0 W :00 W 0:80 W 8:560 W 56:60 W 6:70 W 7:

6 J. Computer Sci., (7): , 008 Percentage classification Character classification 00 Epochs 000 Epochs 0000 Epochs No. of hidden neurons Fig.: Effect of cross pruning on character classification Number of Incorrect classifications Numerals miss-classifications 8 Typed 00 7 Typed Typed Reference values Fig. : Effect of cross pruning on classification errors Number of Incorrect classifications Numerals miss-classifications Reference values Typed 00 Typed 000 Typed 0000 Fig. 5: Effect of cross pruning on miss-classification Figure demonstrates application of Eq. in conunction with our cross pruning approach to obtain a much more reliable and reflective representation for the errors that have occurred as a result of miss classification. It is clear from the diagram that the greatest percentage of miss classification occurs when using a 00 epochs with larger oscillation period compared to a 000 epochs the number of tested hidden neurons with the difference of amplitude reduction in the sum squared errors at earlier stage due to the large number of cycles but still unstable due to over teaching and over fitting compared to the 00 epochs case which suffers instability due to under learning and under fitting. It is clear from the curve that the optimum number of epochs that produce a stable classification system with minimum number of errors as a function of hidden layer neurons is a 000 epochs. Figure 5 deals with an important aspect of designing our MLP OCR system and in general contributes to designing a much more accurate classification system through the plotting and analyzing of what we can call Statistical Classification Reach (SCR) whereby the figure indicates through the use of our Cross Pruning (CP) technique the effect of varying both number of epochs and number of hidden layer neurons on the absolute statistical difference of numeral miss classification which can be generalized to cover character and word miss classification. It is clear from Fig. 5 and by applying our SCR technique that our system will be most unstable when using 00 and 0000 epochs with low number of hidden layer neurons and it also shows that the numeral the system had most difficulty in learning and classifying is the numeral as it shares common properties with numerals, 8. The figure also indicates that epoch numbers and hidden neuron numbers had no effect on the classification of the numeral 7 as its composition is simple and does not share many features with other numerals; hence it is difficult to be confused or misclassified. CONCLUSION It is nown that the choice of hidden units depend on many factors such as the amount of noise in the training data, the type of the shaping or activation function, the training algorithm, the number of input and put units and number of training patterns. Overall, deciding how many hidden layer neurons to use is a complex tas which this research tried to clarify and quantify. In doing so and to have a meaningful characterization for the system performance in terms of classification and to include and analyze the dependency of all the mentioned factors the concept of cross pruning is introduced together with the Statistical Classification Reach (SCR) and weight matrix auto update. These two techniques helped in deciding the optimal number of hidden layer neurons and number of epochs and pinpointed to the numerals that the system found difficult to classify. Our approach is certainly novel in terms of reducing time and effort and simplifying approaches to system design and modification which will assist in eliminating any wea points found in numeral based system in terms of numeral, MLP based recognition system. 58

7 REFERENCES. Thivierge, J.P., F. Rivestand and T.R. Shultz, 00. A dual-phase technique for pruning constructive networs. Proceedings of the IEEE International Joint Conference on Neural Networs, July 0-, IEEE Xplore Press, USA., pp: ber=07.. Pareh, R., J. Yang and V. Honavar, 000. Constructive neural-networ learning algorithms for pattern classification. IEEE Trans. Neural Networs, : xpl/freeabs_all.sp?tp=&arnumber=890&isnum ber=800.. Engelbrecht, A., 00. A new pruning heuristic based on variance analysis of sensitivity information. IEEE Trans. Neural Networs, : DOI: 0.09/ Pareh, R., J. Yang and V. Honavar, 000. Constructive neural networ learning algorithms for multi-category pattern classification. IEEE Trans. Neural Networs, : 6-5. Digital Obect Identifier 0.09/ Yang, J., R. Pareh and V. Honavar, 000. Comparison of performance of variants of singlelayer perceptron algorithms on non-separable data. Neural, Parallel Sci. Comput., 8: onid=b960d8e989607cccca67b? doi= Chen, C.H. and V. Honavar, 999. A neural networ architecture for syntax analysis. IEEE Trans. Neural Networs, 0: Yang, J., R. Pareh and V. Honavar, 998. DistAl: An inter-pattern distance based constructive learning algorithm. Technical Report TR /0/50/index.html. 8. Yao, X., 99. A review of evolutionary artificial neural networs. Int. J. Intell. Syst., : Cantú-Paz, E., 00. Pruning neural networs with distribution estimation algorithms. Lecture Notes Comput. Sci., 7: ist.psu.edu/viewdoc/summary?doi= Amin, H., K.M. Curtis and B.R. Hayes Gill, 997. Dynamically pruning put weights in an expanding multilayer perceptron neural networ. Proceedings of the th International Conference on Digital Signal Processing, July -, Santorini, Greece, pp: DOI: 0.09/ICDSP J. Computer Sci., (7): , Chen, H.H., M.T. Manry and H. Chandrasearan, 999. A neural networ training algorithm utilizing multiple sets of linear equations. Neurocomputing, 5: viewdoc/summary?doi=0.... Ponnapalli, 999. A formal selection and pruning algorithm for feedforward artificial networ optimization. IEEE Trans. Neural Networs, 0: DOI: 0.09/ Poliar, R., L. Udpa, S. Udpa and V. Honavar, 00. Learn++: An incremental learning algorithm for supervised neural networs. IEEE Trans. Syst., Manand Cybernetics-Part C: Appl. Rev., : DOI: 0.09/ Costa, M., A. Braga and B. Menezes, 00. Constructive and pruning methods for neural networ design. Proceeding of the 7th Brazilian Symposium on Neural Networs, Nov. -, IEEE Computer Society, USA., pp: Grippo, L., 000. Convergent on-line algorithms for supervised learning in neural networs. IEEE Trans. Neural Networ, : DOI: 0.09/ Burgess, N., 99. A constructive algorithm that converges for realvalued input patterns. Int. J. Neural Syst., 5: DOI: 0.09/ Reed, R., 99. Pruning algorithms: A survey. IEEE Trans. Neural Networ, : DOI: 0.09/ Hancoc, P.J.B., 99. Pruning neural nets by genetic algorithm. Proceedings of the International Conference on Artificial Neural Networs, 99, Elsevier Science, Amsterdam, Netherlands, pp: summary?doi= Lim, T.J., W.Y. Loh and Y.S. Shih, 000. A comparison of prediction accuracy, complexity and training time of thirty-three old and new classification algorithms. Mach. Learn., 0: summary?doi= Setiono, R. and A. Gaweda, 000. Neural networ pruning for function approximation. Proceeding of the International Joint Conference on Neural Networs, July -7, IEEE Computer Society, Los Alamitos CA., pp: amodele=affichen&cpsidt=795.. de Kruif, B.J. and T.J.A. de Vries, 00. Pruning error minimization in least squares support vector machines. IEEE Trans. Neural Networs, : DOI: 0.09/TNN