Performance Evaluation of a Radial Basis Function Neural Network Learning Algorithm for Function Approximation.

Transcription

1 Performance Evaluation of a Radial Basis Function Neural Network Learning Algorithm for Function Approximation. A.A. Khurshid PCEA, Nagpur, India A.P.Gokhale VNIT, Nagpur, India Abstract: This paper presents a performance analysis of the learning algorithm used to optimize radial basis function neural network in order to approach target functions from a set of input output pairs. This algorithm combines the growth criterion which adds neurons to the net until a stopping criteria is met for efficient design with a pruning strategy based on the relative contribution of each hidden unit to the overall network output. The resulting networks leads towards a minimal topology with better approximation accuracy. The performance of this algorithm is compared with RBF networks making use of operators based on singular value decomposition & orthogonal least squares in the function approximation areas. For all these problems, the algorithm is shown to realize better approximation accuracy and fewer hidden neurons. Keywords: Artificial Neural Network, Radial Basis Functions, Function approximation I. Introduction Radial Basis function neural networks [1] consist of neurons which are locally tuned. An RBFNN can be regarded as a feed forward artificial neural network [2] with a single layer of hidden units whose responses are the output of radial basis functions as shown in figure1. Figure 1 Radial basis function neural network (RBFNN) can be described by the following equation. f =1 m F(x,Φ,ω) = W f Ф f (x,c f,r f ) where m is the number hidden units. The output of the network F depends on the input vector x = (x 1,,xd) T and on the set of RBF s Ф = (Ф 1, Ф m ) and weights W=(ω 1,..ω m ). The contribution of each RBF depends on the distance from the input vector x to the centre c f of each RBF Ф f. The functions which are of particular interest in the study of RBF net are guassian functions, multiquadrics and inverse multiquadrics. The family of RBF networks is broad enough to uniformly approximate any continuous function on a compact set [2]. Due to their simple structure as compared with multilayer perceptrons (MLP s ) [3] there has been increasing research in RBFNNs & their applicability as function approximators. The learning process undertaken by a RBF network may be visualised as follows. The linear weights associated with the output units of the network tend to evolve on a different line scale compared to the non linear activation functions of the hidden units. Thus as the hidden layers activation functions evolve slowly in accordance with some nonlinear optimization the output layer weights adjust themselves rapidly through a linear optimization strategy. The important point is that the different layers of an RBF network perform different task and so it is reasonable to separate the optimization of the hidden and output layers of the network by using different techniques. In the classical approach to RBF network implementation the basis function are usually chosen as Gaussian & the number of hidden units is fixed a priori based on some properties of the input data. The weights connecting the hidden and output units are estimated by linear least squares methods eg. Least mean square (LMS) [4,5], recursive least squares [6]. The disadvantage with the classical approach is that it results usually in too many hidden units. There are several well known methods to find optimum values for the weights such as Cholesky decomposition [7] or singular value decomposition [8]. In the literature several algorithms to identify these parameters have been published especially because one stage gradient descent algorithms have stability problems when dealing with spread parameters of RBF s [9, 10]. There is a sequential learning algorithm for radial basis function (RBF) networks referred to as generalized growing and pruning algorithm for RBF (GGAP-RBF) [14]. It introduces the concept of significance for the hidden neurons and then uses it in the learning algorithm to realize parsimonious networks. Computation of significance for the hidden neurons involves an integration of the probability density function in the sampling range for the functions other than the popular ones. Moreover it 75

2 uses variable width guassian function where the overlap factor has to be appropriately chosen to determine the overlap of the responses of the hidden neurons in the input space. During pruning the parameters of the nearest neuron is adjusted and if the nearest neuron becomes insignificant it is removed. Each time the parameter adjustment is done using EKF algorithm. Another work suggests, an algorithm [15] that initializes the centers and the radii has been proposed. The new algorithm is able to design better RBFNNs.The success of the new algorithm resides in the idea of a generic activation function that keeps a balance between the output of the target function and the coordinates of the input vectors. The output of the algorithm is used to calculate the values of the radii for each RBF performing much better than other heuristic used for this task.. This paper is organized as follows. Section II gives a brief description of the algorithm. Section III gives the comparison between the proposed algorithm and Multi layer feedforward networks(mfn s) trained using other approaches. Section IV summarizes the conclusions from this study. II. Learning Algorithm A. Growing Criterion: The learning process of this algorithm involves the allocation of new hidden neurons as well as adaptation of network parameters. The RBF network begins with no hidden neurons. As inputs are received during training, a new hidden neuron will be initiated with its centre equal to the input vector with the greatest error until it meets the specified mean squared error goal. The output layer weights are redesigned to minimize error. Approximate choice of spread of RBF is required to fit a smooth function. Larger the spread, smoother the function approximation will be, but too large spread means a lot of neurons will be required to fit a fast changing function.also too small a spread means many neurons will be required to fit a smooth function and the network may not generalize well. Hence depending upon the mean absolute deviation of the input output pairs the spread is appropriately chosen. B. Pruning Criteria: This uses the basic idea of Yingwei et al [11].The pruning strategy removes these hidden units which makes insignificant contribution to the overall network output consecutively over a number of training observations. It uses a sliding window in the pruning criteria to identify the neurons that contribute relatively little to the network output. Selection of the appropriate sizes for these windows critically depends on the distribution of the input samples. To realize compact RBF network this pruning scheme checks the pruning criteria for all hidden neurons after all training observations have been prescribed and learned. The pruning criterion is indicated as follows. For every observation, the outputs of the hidden units are first normalized with respect to the maximum output value among all hidden neurons. These normalized value are then compared with a threshold δ and if any of them falls below this threshold for a sliding window of size M, then this particular hidden neuron is removed from the network. C. The final algorithm is summarized below: For each observation (xn, yn) do. If mse > goal Compute overall network output: k f(xn) = α 0 + α k exp ( -1 11x n -μ k 11 2 ) k=1 k= no. of hidden units σ 2 k Compute the input vector with greatest error (xn max). Allocate a new hidden unit K+1 with μ k+1 = Xn max α k+1 =[w b] * [A ; ONES (Size A)] where b = sqrt (-log(0.5))/spread A = output of the hidden layer. end if end for Check the criterion for pruning hidden units: Compute hidden unit outputs Find the largest absolute hidden unit output. Compute the normalized output values (O n R norm) R = 1---k. If O n R norm < δ for sliding window of size M, then remove the R th hidden unit. Reduce the dimensionality of network parameters. end if. Adjust the network parameters using RPROP. 76

3 III. Performance of the algoriothm for function approximation In this section performance of the algorithm is given for several function approximation problems with I-D, 2-D & 3-D target functions proposed in the literature. I-D functions: In this section the algorithm is tested with I-D function used by other authors. The result obtained is compared with the solutions presented by other authors in terms of the error committed by the model and its complexity. The 1-D target function used is originally proposed by Dickerson and Kosko [12]. The function is defined as dick(x) = 3x (X-1)(X- 1.9 )( X + 0.7) (X+ 1.8), X [-2.1, 2.1].The training set used was composed up of 2100 examples equidistributed in the input interval [-2.1, 2.1] and test set contained 2200 test data also equidistributed in the same input range. Table 1 shows the approximation error reached by the algorithm. Dickerson & Kosko applied a hybrid new fuzzy system with ellipsoidal rules trained by several learning methods, while Pomares proposed a fuzzy system based on a complete table of rules using triangular membership functions. Figure 2(a) shows the result of testing the system. This algorithm is also demonstrated with the help of the function f(x)=0.5xsin 2 (x) + cos 2 (x) [16] which leads to smaller network size (m=7) and better accuracy (MSE= ) as compared to the above at the cost of training time and is as shown in figure 2(b). Again the algorithm is evaluated using the function f1(x) = sin(2πx) /e x [15] and the evolved network with m=6 has an approximation error(rmse) of , m=8 has an approximation error(rmse) of m= The number of RBF s or rules (Depending upon the model) Table Algorithm Dicker son & Kosko Pomares 2000 MOEA 2003 Unsuperv ised Supervise d Proposed Approach Figure 2(b) m mse ± ± test data real data Figure 2(c) Figure2(a):Testing results 77

4 2-D functions : In this section we have used 2-D functions f 5 and f 7 originally proposed in [13]. The two target functions are defined as follows: f 5 (x 1, x 2) = (0.1+ x 1 ( X X 1 2 X X 2 4 )). X 1, X 2 [-0.5, 0.5] f 7 (X 1,X 2 ) = 1.9(1.35+e x1 sin(13(x 1-0.6) 2 ) e -X2 sin (7X 2 )) X 1, X 2 [0, 1] The training sets for the experiments presented in this section consist of 1580 samples of the input space for f7 and 700 samples for f5. The test sets were formed by dividing the input interval with a (31 X 31) grid. Table 2 & 3 shows the approximation error reached by the algorithm. 3-D Functions: In this section the algorithm is used to approximate the following function. Y(x 1, x 2, x 3 ) = 0.1(e x1 + x 2 x 3 cos (x 1 x 2 ) + x 1 x 3 ), x 1 [0, 1], x 2, x 3 [-2, 2] This function is highly non linear, involving an exponential, three multiplications and a trigonometric function. A training set is created by generating 1000 uniformly distributed random values of X1(t) = (t), X2 (t) = 1.61(constant) X3(t) = 8t 2 0 t <0.5-8t t <1 Table 2 (Target function f 5 ) m = no. of RBF is a rules (Depending upon the model) Algorithm m Test NRMSE MLP(Cherkassky, ) Pomares,2000[17] 8X (PT) MOEA, ± ± Proposed approach MSE ± m = no. of RBF is a rules (Depending upon the model) Algorithm m NRME MSE MLP (Cherkassky ) Pomares 2000 MOEA,2003 Proposed approach Table 3(Target function f 7 ) 7X (PT ) ± ± Figure 3 shows the result of testing the system. Function Approximation Problem: Hearta- Problem The hearta dataset consists of 920 examples out of which 690 are to be used for training and the rest 230 to be used for testing. Each example is described by 13 input attributes and one output attribute. Among the training examples only 299 are complete and the rest have one or more missing values. As per PROBEN1 guidelines, missing values were replaced with the mean of the nonmissing values for that attribute. In PROBEN1, the 13 input attributes have been coded into 35 input units.the performance was evaluated by using hearta2 dataset with parameters M=15, δ= and is as shown in table 4. Table 4 Benchmark m mse(training set) mse(test Data set) hearta IV. Conclusion In this paper the performance of the algorithm is compared with other approaches for function approximation problems. Result shows that the algorithm produces a RBF neural network with smaller complexity and better accuracy. Since the algorithm uses a sliding data window in the pruning criteria, selection of the appropriate sizes 78

5 for these windows critically depends upon the distribution of the input samples. Choice of proper window sizes can only be done by trial and error based on exhaustive simulation studies. Training data needs to be stored and reused for pruning purposes. This learning algorithm adds new neurons based on their novelty to the individual instant observations until the goal is met. The implementation of this approach still depends on the data being available all at the same time and hence is strictly not a sequential one but a variation of batch algorithm only. V. References [1.]D.S.Broomhead and D. Lowe, Multivariate functional interpolation and adaptive networks, complex syst., vol 2,pp , [2.]S. Haykin, Neural Networks, a comprehensive foundation, second edition, Pearson Education. [3.] J.Hertz, A.Krough and E.G. Palmer, Introduction to the theory of Neural computation, Reedwood city, Addison-Wesley [4.] J.Moody and C.J. Darker, Fast learning in network of locally tuned processing units, Neural computation, vol 1, pp , [5.] M.Musavi, W.Ahmed, K.Faris and D.Hummels, On training of radial basis function classifiers, Neural networks, vol 5,no.4, pp , 1992 [6.[ S.Chen, S. Billings and P. Grant, Recursive hybrid algorithm for non linear system identification using radial basis function networks, Int. J. Contr., vol 55, pp ,1992. [7.] S Chen, C.F.N.Cowan and P.M. Grant, Orthogonal least squares learning algorithm for radial basis functions, IEEE Trans. Neural Network,vol 2, pp ,1991 [8.] P.P.Kanjilal and D.N. Banerjee, On the application of orthogonal transformation for the design and analysis of feed forward networks, IEEE Trans. Neural Network, vol 6, pp ,1995. [9.] N.B.Karayiannis Re-formulated radial basis neural networks trained by gradient descent, IEEE Trans. Neural Networks, vol. 10, pp , May 1999 [10.] N.B.Karayiannis and G.W.Mi, Growing radial basis neural networks: Merging supervised and unsupervised learning with network growth techniques, IEEE Trans. Neural Networks, vol 8., pp , Nov.1997 [11.] Lu. Yingwei, N. Sundarajan and P. Saratchandran, Performance Evaluation of a sequential Minimal Radial Basis function Neural Network learning algorithm, IEEE Trans. Neural Networks. vol 9,no.2, pp , March1998. [12.] J. A. Dickerson and B. Kosko Fuzzy function approximation with ellipsoidal rules, IEEE Trans syst. Man cyber. B, vol 26, pp , Aug [13.] V.Cherkassy and H. Lari- Najafi, Constrained topological mapping for nonparametric regression analysis, IEEE Trans.Neural Networks, vol.4, no. 1, pp , [14.] Guang-Bin Huang, P.Saratchandran, and Narasimhan Sundararajan, A Generalized Growing and Pruning RBF (GGAP-RBF) Neural Network for Function Approximation, IEEE Transactions On Neural Networks, Vol. 16, No. 1, Jan.05. [15] Alberto Guill en, Ignacio Rojas, Jes us Gonz alez, H ector Pomares, L.J. Herrera and A. Prieto, Supervised RBFNN Centers and Radii Initialization for Function Approximation Problems, International Joint Conference on Neural Networks, Sheraton Vancouver, Wall Centre Hotel, Vancouver, BC, Canada, July 16-21, 2006, Pg , [16] Fatemi,Mehdi Roopaei,Faridoo Shabaninia, New Enhanced Methods for Radial Basis Function Neural Networks in Function Approximation, Proceedings of the Fifth International Conference on Hybrid Intelligent Systems (HIS 05),2005 [17] H. A systematic approach to a self-generating Pomares, I. Rojas, J. Ortega, J. González, and A. Prieto, fuzzy rule-table for function approximation IEEE Trans. Syst., Man Cybern. B, vol. 30, pp , June