An Empirical Study of Software Metrics in Artificial Neural Networks

Transcription

1 An Empirical Study of Software Metrics in Artificial Neural Networks WING KAI, LEUNG School of Computing Faculty of Computing, Information and English University of Central England Birmingham B42 2SU UNITED KINGDOM Abstract: - In the previous studies by Leung et al. [1,2,3,4], a set of software metrics, named Neural Metrics that are applicable in supervised feedforward Artificial Neural Networks (ANNs and provide measures of the network quality and training complexity, was proposed. This study extends the work by empirically evaluating the neural metrics against the standard Backpropagation Algorithm (BPA across several types of benchmark application problems. The result of the evaluation was, for each type of problem, a specification of values of all neural metrics and network parameters which can be used to successfully solve the same or similar type of problem. With such a specification, neural users can reduce the uncertainty and hence time in choosing the reliable network details for solving the same or similar type of problem. In addition, users are provided with a better understanding of the algorithmic complexity of the problem by referring to the values of the neural metrics in the specification. They may also use the specified neural metric values as reference points for similar experiments with a view to obtaining a better or sub-optimal solution for the problem. Thus, if the values of neural metrics obtained in a further experiment are less than that reported in the specification, the experiment may be considered to be an improved or more efficient approach than the standard BPA. Keywords: - Neural Networks, Backpropagation, Software Metrics, Algorithmic Complexity. 1 Introduction Due to the lack of applicable measurements, no analysis has been carried out comprehensively on the quality characteristics (e.g. efficiency and complexity of any ANN system simulated on a conventional computer. Hence, users of such systems with little or no specialist expertise in this area do not generally know how efficient the network system performs nor how complicated the training process is for solving a specific application problem. It is also difficult to acquire and compare reliable implementation details based on the published work of each researcher. Furthermore, researchers sometimes use minor variations on the published algorithms (such as the standard BPA without making it clear what the variations are and why they were necessary. The results obtained by individual researchers do not normally specify the values of all variables and parameters (e.g. number of network layers and number of hidden units that are used in the training process. Without a complete specification, users are often uncertain on the choice of reliable values in order to solve the same or similar type of problem successfully. They are also uncertain if their experiments have resulted in a better or sub-optimal solution for the problem since no reference points for such values are available. To overcome the above difficulties, Leung et al [1] have proposed a set of software metrics, named Neural Metrics, which provide indicative measures of the quality characteristics of a neural network system. This study extends the work by empirically evaluating these metrics across several types of benchmark problems. It has been reported that 82% of 113 papers published in Neural Computation and Neural Networks over the 1993/94 period do not use two or more realistic benchmarking tests [5]. This is generally still the same case in the last five years. To make the proposed neural metrics applicable to ANNs, it is essential that they are evaluated across more than two benchmark problems. The results obtained in the evaluation were the average values of neural metrics that may be used to solve a specific type of problem. They were also used in the calculation of the generalised algorithmic complexity of each type of problem. 2 Evaluation Approach In this study, the evaluation of neural metrics was done iteratively for each problem (see Figure 1. Arbitrary initial values of the neural metrics were supplied by the user to the network system. The network system was trained with a view to improving its quality characteristics. If the network

2 showed improvement (e.g. faster convergence, the values of the neural metrics were recorded; otherwise they were discarded and new values were to be used. This process continues with new values of the neural metrics replacing the old ones for each improvement. The values of the neural metrics at the end of the evaluation process should ideally provide the optimal solution achievable to the given application problem. Despite physical limitations such as machine precision and architecture, the optimal solution means the attainment of the best quality characteristics, e.g. the highest efficiency and lowest complexity of the network system. However, such an optimal solution has hardly been obtained (or even identified by any neural network researcher. What most researchers can do is to improve the results obtained in similar studies by advocating or applying one or more optimisation techniques. Neural Network Improved Neural Network Improved Neural Network Optimal Neural Network Neural Metrics Revised Neural Metrics Revised Neural Metrics Optimal Neural Metrics User Figure 1. Evaluation of Neural Metrics 3 Testing Strategy This section describes the strategy that was used in this research to test the results obtained from the evaluation of neural metrics for a specific problem. This strategy, which was applied throughout the evaluation process, includes the choice of training data, network structure, training algorithm, optimisation techniques, initial values of neural metrics and network parameters, training completion criteria, and the calculation of the complexity of the chosen problem. The training data sets were chosen such that each consisted of all input-output pairs that constituted the problem domain. Once training was completed, the same training sets were used to test the final networks. The values 0.1 and 0.9 were used to represent binary values 0 and 1 respectively in order to prevent any of the delta weights from attaining the value of 0 [6]. Such a representation also helped in speeding up the training process by avoiding any of the output values from attaining the asymptotical values of 0 or 1. Since networks with more than one hidden layers are more prone to fall into local minima [7,8], single hidden layer networks were used for all the problems. In addition, no short-cut connection weights were included in the networks as these violate the assumptions made on the standard BPA and hence the resulting complexity (and validity of neural metrics of a problem [1]. Bias weights were, however, allowed as they behave in the same way as standard connection weights and do not violate any of the assumptions. Standard BPA with periodic weight update was chosen as the training algorithm because of the nonlinear nature of most application problems [9]. The initial weights were randomly initialised between the values of -0.5 and 0.5 for periodic update and -0.1 and 0.1 for continuous update [10]. The initial value of the learning coefficient was chosen between and 1.00 [11]. The value of the steepness coefficient was fixed in the standard BPA with the value of 1.0 [12]. Further, the initial value of the momentum term was chosen between 0.1 and 1.0 [13]. There are five commonly used completion criteria [14] in BP based training. The sharp threshold criterion is widely used in binary problems where any output over 0.5 is accepted as 1, while any output below 0.5 is accepted as 0. If the small individual error criterion is used, each output value will need to be very close to the desired value. The small composite error criterion, on the other hand, requires that the sum of squared errors for all the outputs must fall below some fixed value. In the winner-take-all criterion, the value of the correct output unit must be larger than that of any other output unit. The threshold with margin criterion chooses a fixed threshold (e.g. 0.5 and treats as incorrect any value that is too close to this threshold. For problems that use this criterion (e.g. [14,15], any

3 output value over 0.6 is accepted as 1, any under 0.4 as 0, and any between 0.4 and 0.6 as indeterminate or incorrect. The sharp threshold criterion was used in this research for all the problems studied. This is because it does not require additional computations (e.g. calculating the composite errors in the small composite error criterion and comparisons (e.g. selecting the correct output in the winner-take-all criterion. There are also no indeterminate values (e.g. those in the threshold with margin criterion to be catered for. The evaluation process was repeated a number of times for each application problem until the attainment of a set of values of neural metrics and network parameters that can be used to successfully solve the problem. The average algorithmic complexity required in solving the problem was calculated using neural metric function TOT which is defined in terms of primitive metrics such as the average number of epochs taken, the number of layers, the number of units on each layer, and the number of training pairs in the training data set [1,2]. 4 Benchmark Problems Four types of benchmark application problems were chosen for the evaluation process. They were chosen as the benchmark in this research because they have been widely studied or used by a number of neural researchers. For example, they have been frequently chosen to illustrate the techniques being proposed for BP optimisation and compare the results obtained amongst the various optimisation techniques. The Encoding problem, which has been chosen in a number of studies (e.g. [14,16,17], aims to develop a network which can produce the same output as the given input. Each input or output consists of an n-bit binary number containing a single binary 1. The most common choice for n is 4, 8 or 10 and so the problem is normally referred to as 4-bit, 8-bit or 10- bit Encoding. Another benchmark that is studied or used by a number of neural scientists is the Parity problem which determines if a given binary string contains an odd number of 1 s. The 2-bit, 3-bit and 4-bit Parity problems are most commonly studied (e.g. [17,18,19]. The 2-bit Parity problem is the same as the XOR problem. The other two benchmarks are the Binary Addition and Symmetric problems. The former deals with the addition of two equal length binary numbers and the latter is to determine if a given binary string is symmetric about its centre. The 2-bit Binary Addition and 4-bit Symmetry problems are widely being studied. It must be emphasised that the evaluation results obtained in this study for each application problem are based on standard BPA whilst those obtained by other researchers may be based on non-standard BPA where specific optimisation techniques are used. Thus the results of this study may show a better or poorer efficiency than the others in terms of training complexity and convergence. They are presented in this study in order to show the average values of neural metrics involved in the successful resolution of an application problem via standard BPA. These values can be used to compute the average algorithmic complexity of the problem or be evaluated further in finding a sub-optimal solution for the problem. In other words, the results of this study may be used as a reference point for optimisation. For instance, if the number of training epochs or iterations (i.e. value of neural metric M obtained in an optimisation is less than that reported in this study, then such an optimisation may be considered an improved or more efficient algorithm than the standard BPA. 4.1 Encoding Problem The 4-bit, 8-bit and 10-bit Encoding problems were addressed. Wang et al. [16] conducted simulations on the 4-bit Encoding problem using several BP variant approaches. The average number of epochs they obtained varied between 10 and 200. Fahlman [14] solved the same problem with an average of 16 epochs. The average number of epochs (i.e. neural metric M obtained in this study (using standard BPA is 15 as shown in Table 1. The network was trained by evaluating the values of the network parameters and neural metrics over a number of simulations. A set of these values that can be used to solve the problem are specified in Table 1. As mentioned in the testing strategy, the initial values of the connection weights W were chosen at random between -0.5 and 0.5. The initial value of the learning coefficient ε was chosen between and The steepness coefficient λ was kept constant at the value The value of the momentum factor α was initially chosen between 0.1 and 1.0. Moreover, the value of m chosen in all training cycles was 3, i.e. the network consisted of 3 layers (1 input layer, 1 hidden layer and 1 output layer. The values of some of the neural metrics were determined from the training data set. These metrics are the number of input units, n [1], the number of output units, n [3], and the number of input patterns per epoch, P. They were the length of the input binary string (i.e. 4, the length of the output binary string (i.e. 4, and the number of input-output

4 pairs in the training set (i.e. 4 respectively. These values were kept unchanged in all training cycles. No scaling on the input and output values was carried out since the training data set contained only binary data. Thus the number of scaling operations, S, was 0. The number of hidden units, n [2], and the average number of epochs required for convergence, M, were determined experimentally. The initial number of hidden units was chosen to be that of the input units (i.e. 4. Encoding Problem 4-bit 8-bit 10-bit Neural Metrics Description Values Values Values n [1] Number of input units n [2] Number of hidden units n [3] Number of output units m Number of layers M Average number of epochs required P Number of input patterns per epoch N [2] Number of hidden weights N [3] Number of output weights N Number of weights S Number of scaling operations ACT Number of activation function invocations 420 7,280 19,200 ADD Number of additions and subtractions 4, , ,400 MUL Number of multiplications and divisions 7, , ,400 TOT Total number of operations 12, ,000 1,104,000 Network Parameters Values Values Values ε Learning coefficient λ Steepness coefficient α Momentum factor W Initial hidden and output weights [-0.5, 0.5] [-0.5,0.5] [-0.5, 0.5] Table 1. Encoding Problem Specification The initial average number of epochs was set to 10. As the network was fully connected with no short-cut connections, the number of hidden weights, N [2] was the product of n [1] and n [2] (i.e. 4*3 = 12. The number of output weights, N [3] was the product of n [2] and n [3] (i.e. 3*4 = 12. The total number of weights, N, was therefore 24. The values of neural metrics n [2] and M and the values of network parameters ε and α were then altered after each training cycle until the network converges. It can be seen from Table 1 that M O( N and P O( N,i.e. the average number of epochs required for convergence and the number of input patterns per epoch are both proportional to the total number of weights in the network. These are consistent to the theoretical conditions specified in [1]. The values of neural metrics ACT, ADD, MUL and TOT as shown in Table 1 were computed using formulae as defined in [1]. Since 12,180 < 24 3, the values show that TOT O( N 3, i.e. the average algorithmic complexity required to solve the 4-bit Encoding problem is proportional to the 3rd power of the number of connection weights in the network. Similarly, the results for the 8-bit and 10-bit Encoding problems as shown in Table 1 show that M O( N, P O N 3. ( and TOT O( N 4.2 Parity Problem The second type of problem being studied is that of Parity. Three cases had been conducted, namely 2- bit, 3-bit and 4-bit Parity. The 2-bit Parity problem is the same as the XOR problem and has been widely used as a benchmark problem by variant versions of standard BPA. Jacobs [19] reported that an average of 530 and 250 epochs are respectively required to solve this problem based on the standard Jacobs method and the Jacobs method with delta-bar-rule. A network was solved by Deleone et al [17] in an average of 121 epochs. The Quick-Prop algorithm (Fahlman et al. [18], on the other hand, achieves convergence in 24 epochs on average. The average number of epochs obtained in this study is 60 as shown in Table 2. The variations amongst these results are due to different network architectures and training algorithms being used by different researchers. With similar reasoning for the Encoding problems, the results obtained in this study show 2 that M O( N, P O( N and TOT O( N 4. These are the same for the 3-bit and 4-bit Parity problems. 4.3 Binary Addition & Symmetry Problems

5 The Binary Addition problem deals with the addition of two binary numbers of the same length. The 2 results of the simulations showed that M O( N, Parity Problem 2-bit 3-bit 4-bit Neural Metrics Description Values Values Values n [1] Number of input units n [2] Number of hidden units n [3] Number of output units m Number of layers M Average number of epochs required ,090 P Number of input patterns per epoch N [2] Number of hidden weights N [3] Number of output weights N Number of weights S Number of scaling operations ACT Number of activation function invocations 1,440 23, ,840 ADD Number of additions and subtractions 9, ,280 2,007,780 MUL Number of multiplications and divisions 18, ,800 4,046,080 TOT Total number of operations 29, ,120 6,245,700 Network Parameters Values Values Values ε Learning coefficient λ Steepness coefficient α Momentum factor W Initial hidden and output weights [-0.5, 0.5] [-0.5,0.5] [-0.5, 0.5] Table 2 Parity Problem Specification P O( N and TOT O( N 4, i.e. this type of problem has an average algorithmic complexity of O ( N 4. The Symmetry problem is to tell if a given binary string is symmetric about its centre. The fourbit symmetry problem has 16 input-output pairs and has been solved by Fukuoka et al. [20] in 500 to 1,200 epochs, using the network. In this study, the network converged to a solution at the 140th 2 epoch. The results in show that M O( N, P O( N and TOT O( N 5, i.e. this type of problem has an average algorithmic complexity of O ( N 5. 5 Generalised Results The simulations in this paper showed that each type of benchmark problem can be solved by standard BPA within an average algorithmic complexity that is bounded by a specific power of the total number of connection weights in the network, i.e. the computed metric TOT of each problem is such N k that TOT O( for N connection weights in the network and an integer k such that 3 k 5. This is the same as what Leung et al have claimed [1]. The results from each problem are summarised in Table 3 (where N is the total number of connection weights in the network. It shows that each type of problem Problem Simulations Number of Input Patterns Encoding 4-Bit 8-Bit O N 10-Bit Parity 2-Bit (XOR 3-Bit O N 4-Bit Binary Addition 2-Bit Symmetry 4-Bit Number of Epochs ( O( N O( N 3 ( O( N 2 O ( N 4 O ( N O( N 2 O ( N 4 O ( N O( N 2 O ( N 5 Average Algorithmic Complexity Table 3. Specification of Algorithmic Complexity for some Benchmark Problems

6 has a polynomial-bound solution and thus belongs to the class of feasible problems. It can be seen that each type of problem has the same order of algorithmic complexity. For example, the Encoding problem is O ( N 3 whether it is 4-bit, 8-bit or 10- bit. Table 3 can serve as a general reference for the average algorithmic complexity involved in solving a particular type of problem. It also provides a categorisation of the average algorithmic complexity for all types of problems addressed in this study. For instance, both the Parity and Binary Addition problems belong to the same category since they have the same order of magnitude of algorithmic complexity. 6 Conclusion By evaluating the neural metrics across several types benchmark problems, it is believed that the results of this research will provide neural users with a reliable suite of problem specifications which detail the values of neural metrics and parameters that may be used to solve the problems successfully. In addition, the results were generalised to provide the average algorithmic complexity of the problems. It is also believed that the research results can be further extended to cover other network paradigms (e.g. unsupervised ANNs and application problems. However, it must be emphasised that the problem specification proposed in this study may not give the optimal values of the neural metrics and network parameters. All it shows are the values that will lead to a reliable and successful training of the network for the given problem. Researchers who are interested in the optimal condition may use the values in the specification to further their experiments. They may need to change the values of some of the neural metrics and/or parameters, while keeping the others unchanged, until the specified optimal criteria (e.g. attaining the threshold number of epochs or hidden units is satisfied. References: [1] Leung W.K., and Simpson R., Neural Metrics - Software Metrics in Artificial Neural Networks, Proceedings of the 2000 International Conference of the Knowledge Based Engineering s, University of Brighton, Brighton, UK; Aug [2] Leung W.K., and Winfield M., A Complexity Analysis of the Backpropagation Algorithm, Proceedings of the WSES 2000 International Conference of Applied and Theoretical Mathematics, Athens, Greece; Dec [3] Leung W.K., and Winfield M., Implementing Backpropagation with Momentum, Periodic Weight Update and Gradient Descent on Steepness, Proceedings of the WSES 2000 International Conference of Applied and Theoretical Mathematics, Athens, Greece; Dec [4] Leung W.K., On the Complexity of Backpropagation with Momentum and Gradient Descent on Sigmoidal Steepness, Proceedings of the WSES 2001 International Conference of Neural Network and Applications, Tenerife, Spain, Feb [5] Prechelt L., A Quantitative Study of Experimental Evaluations of Neural Network Learning Algorithms: Current Research Practice, Neural Networks, Vol 9, 1996, pp [6] Freeman J.A., Simulating Neural Networks with Mathematica, Addison-Wesley, [7] Alder M., Lim S.G., Hadingham P., and Attikiouzel Y., Improving Three Layer Neural Convergence, The University of Western Australia, Australia, [8] De Villiers J., and Barnard E., Backpropagation Neural Nets with One and Two Hidden Layers, IEEE Trans. on Neural Networks, Vol 4, No. 1, Jan [9] Gori M., and Maggini M., Optimal Convergence of On-Line Backpropagation, IEEE Transactions on Neural Networks, Vol 7, No. 1, Jan [10] Bartlett P.L., and Downs T., Using Random Weights to Train Multilayer Networks of Hard-Limiting Units, IEEE Trans. on Neural Networks, Vol 3, no. 2, [11] Salomon R., and van Hemmen J.L., Accelerating Backpropagation through Dynamic Self Adaptation, Neural Networks, Vol 9, No. 4, 1996, pp [12] Moerland P., Thimm G., and Fiesler E., Results on the Steepness in Backpropagation Neural Networks, IDIAP, [13] Yu X.-H., and Chen G.-A., Efficient Backpropagation Learning Using Optimal Learning Rate and Momentum, Neural Networks, Vol. 10, No. 3, 1997, pp [14] Fahlman S.E., An Empirical Study of Learning Speed in Backpropagation Networks, CMU-CS , Carnegie Mellon University, [15] Lang K.J., and Witbrock M.J., Learning to Tell Two Spirals Apart, Procs. of the 1988 Connectionist Model Summer School, Carnegie Mellon University, [16] Wang G.J., and Chen C.C., A Fast Multilayer Neural- Network Training Algorithm Based on the Layer-By- Layer Optimizing Procedures, IEEE Transactions on Neural Networks, Vol 7, No. 3, May [17] Deleone R., Capparuccia R. and Merellie E., A Successive Overrelaxation Backpropagation Algorithm for Neural-Network Training, IEEE Trans. on Neural Networks, Vol. 9, No. 3, May 1998, pp [18] Fahlman S.E., and Lebiere C., The Cascade- Correlation Learning Architecture, Advances in Neural Information Processing s 2, 1990, pp [19] Jacobs R.A., Increased Rates of Convergence through Learning Rate Adaptation, Neural Networks, Vol 1, 1988, pp [20] Fukuoka Y., Matsuki H., Minamitani H., and Ishida A., A Modified Back-propagation Method to Avoid False Local Minima, Neural Networks, Vol 11, 1998, pp