Heterogeneous Radial Basis Function Networks

Transcription

1 Proceedings of the International Conference on Neural Networks (ICNN ), vol. 2, pp. 23-2, June. Heterogeneous Radial Basis Function Networks D. Randall Wilson, Tony R. Martinez e-ail: Coputer Science Departent, Brigha Young University, Provo, UT 82, U.S.A. ABSTRACT Radial Basis Function (RBF) networks typically use a distance function designed for nueric attributes, such as Euclidean or city-block distance. This paper presents a heterogeneous distance function which is appropriate for applications with sybolic attributes, nueric attributes, or both. Epirical results on 3 data sets indicate that the heterogeneous distance etric yields significantly iproved generalization accuracy over Euclidean distance in ost cases involving sybolic attributes.. Introduction Much research has been directed at finding better ways of helping achines learn fro exaples. When doain knowledge in a particular area is weak, solutions can be expensive, tie consuing and even ipossible to derive using traditional prograing techniques. In such cases, neural networks can be used as tools to ake reasonable solutions possible or good solutions ore econoical. Such an autoated solution is often ore accurate than a hard-coded progra, because it learns fro actual data instead of aking assuptions about the proble. It often can adapt as the doain changes and often takes less tie to find a good solution than a prograer would. In addition, inductive learning solutions ay generalize well to unforeseen circustances. Radial Basis Function (RBF) networks [][3][5] have received uch attention recently because they provide accurate generalization on a wide range of applications, yet can often be trained orders of agnitude faster [] than other odels such as backpropagation neural networks [8] or genetic algoriths []. Radial basis function networks ake use of a distance function to find out how different two input vectors are (one being presented to the network and the other stored in a hidden node). This distance function is typically designed for nueric attributes only and is inappropriate for noinal (unordered sybolic) attributes. This paper introduces a heterogeneous distance function which allows radial basis function networks to appropriately handle applications that contain noinal attributes, nueric attributes, or both. Section 2 introduces the basic radial basis function network that will be used to deonstrate the usefulness of the heterogeneous distance function. Section 3 introduces the distance function itself. Section presents epirical results which indicate that in ost cases the heterogeneous distance function significantly iproves generalization over Euclidean distance when sybolic attributes are present and never reduces accuracy in copletely nueric doains. Section 5 presents conclusions and future research areas. 2. Radial Basis Function Network This section presents a radial basis function (RBF) network that is used as a probabilistic neural network (PNN) [] for classification. The distance function presented in this paper could be appropriately used on any different kinds of basis-function networks, so this particular network is just one exaple of its use. This network was chosen because of its siplicity, which helps to focus on the new distance function instead of on other factors. The network learns fro a training set T, which is a collection of exaples called instances. Each instance i has an input vector y i, and an output class, denoted as class i. During execution, the network receives additional input vectors, denoted as x, and outputs the class that x sees ost likely to belong to. The probabilistic neural network is shown in Figure. The first (leftost) layer contains input, each of which receives an input value fro the x: Input x x2 x3 Hidden h h2 h3 h Class c c2 Decision node Figure. Radial Basis Function Network. z

2 corresponding eleent in the input vector x. Thus, for an application with input attributes, there are input. All connections in the network have a weight of. In essence, this eans that the input vector is passed directly to each hidden node. There is one hidden node for each training instance i in the training set. Each hidden node h i has a center point y i associated with it, which is the input vector of instance i. A hidden node also has a value σ i which deterines the size of its receptive field. This value is set to the distance of the nearest neighbor of i in the training set, using the sae distance function as that used during execution. A hidden node receives an input vector x and outputs an activation given by the function: g(x, y i,σ i ) = exp[-d 2 (x, y i )/2σ i 2 ] () where D is a distance function such as Euclidean distance or the heterogeneous distance function that will be discussed in Section 3. This function g is a Gaussian function which returns a value of if x and y i are equal, and drops to an insignificant value as the distance grows. Each hidden node h i is connected to a single class node. If the output class of instance i is j, then h i is connected to class node c j. Each class node c j coputes the su of the activations of the hidden that are connected to it (i.e., all the hidden for a particular class) and passes this su to a decision node. The decision node outputs the class with the highest sued activation. One of the greatest advantages of this network is that it does not require any iterative training. One disadvantage of this network is that it has one hidden node for each training instance and thus requires ore coputational resources during execution than any other odels. In addition, it does not iteratively train weights on any of the connections, which can ake its generalization less flexible. 3. Heterogeneous Distance Function In Section 2, a probabilistic neural network was presented using radial basis functions and a siple weighting schee that avoided iterative training. In this section, several alternatives for the distance function D are defined, including a new heterogeneous distance function H. Radial basis functions typically use the Euclidean distance function: E(x, y) = (x i y i ) 2 (2) i= where is the nuber of input variables (attributes) in the application. An alternative function, the cityblock or Manhattan distance function, uses less coputation and often does not significantly change the results []. It is defined as: M(x, y) = x i y i (3) i= One proble with both of these distance functions is that they assue that the input variables are linear. However, there are any applications that have noinal attributes. A noinal attribute is one with a discrete set of attribute values that are unordered. For exaple, a variable representing syptos ight have possible values of headache, sore throat, chest pains, stoach pains, ear ache, and blurry vision. Using a linear distance easureent on such values akes little sense in this case, because nubers assigned to the values are in an arbitrary order. In such cases a distance function is needed that handles noinal inputs appropriately. Stanfill & Waltz [] introduced the value difference etric (VDM) which has been used as the basis of several distance functions in the area of achine learning [2][3][]. Using VDM, the distance between two values x and y of a single attribute a is given as: C N vd a (x, y) = a,x,c c= N a,x N a,y,c N a,y 2 () where N a,x is the nuber of ties attribute a had value x; N a,x,c is the nuber of ties attribute a had value x and the output class was c; and C is the nuber of output classes. Using this distance easure, two values are considered to be closer if they have ore siilar classifications, regardless of the order of the values.

3 Soe odels that have used the VDM or extensions of it (notably PEBLS []) have discretized continuous attributes into a soewhat arbitrary nuber of discrete ranges and then treated these values as noinal values. Discretization throws away uch of the inforation available to the learning odel and often reduces generalization accuracy [2]. The Heterogeneous Radial Basis Function (HRBF) odel presented in this paper akes use of a new distance function that uses the above part of the VDM as a building block. In the HRBF odel, the heterogeneous distance H between two vectors x and y is given as: H(x, y) = d 2 a (x a, y a ) (5) a=, if x or y is unknown; otherwise... d a (x, y) = noralized_ vd a (x, y), if a is noinal noralized_ diff a (x, y), if a is nueric () where is the nuber of attributes. The function d a (x,y) returns a distance between the two attribute values x and y using one of two functions (defined below), depending on whether the attribute is noinal or nueric. Many data sets contain unknown input values which ust be handled appropriately in a practical syste. The function d a (x,y) therefore returns a distance of if either x or y is unknown. Other ore coplicated ethods have been tried, as in [], but with little effect on accuracy. The function H is siilar to that used in [], except that it uses VDM instead of an overlap etric for noinal values and noralizes differently. One weakness of the basic Euclidean and Manhattan distance functions is that if one of the input variables has a relatively large range, then it can overpower the other input variables. For exaple, suppose an application has just two input attributes, f and g. If f can have values fro to and g has values only fro to, then g s influence on the distance function will usually be overpowered by f s influence. Therefore, distances are often noralized by dividing the distance for each variable by the range of that attribute, so that the distance for each input variable is in the range... However, this allows outliers (extree values) to have a profound effect on the contribution of an attribute. For exaple, if a variable has values which are in the range.. in alost every case but with one (possibly erroneous) value of 5, then dividing by the range would alost always result in a value less than.2. A ore robust alternative is to divide the values by the standard deviation in order to reduce the effect of extree values on the typical cases. In the heterogeneous distance etric, the situation is ore coplicated because the noinal and nueric distance values coe fro different types of easureents. It is therefore necessary to find a way to scale these two different easureents into approxiately the sae range in order to give each variable a siilar influence on the overall distance easureent. Since 5% of the values in a noral distribution fall within two standard deviations of the ean, the difference between nueric values is divided by standard deviations in order to scale each value into a range that is usually of width.. Using VDM, the average value for N a,x,c /N a,x (as well as for N a,y,c /N a,y ) is /C. Since the difference is squared and then added C ties, the su is usually in the neighborhood of C(/C 2 )=/C. This su is therefore ultiplied by C to get it in the range.., aking it roughly equal in influence to noralized nueric values. The functions noralized_vd and noralized_diff are thus defined as: N noralized_ vd a (x, y) = C * a,x,c N 2 C a,y,c () c= N a,x N a,y noralized_ diff a (x, y) = x y σ a (8) where C is the nuber of classes, N is defined as in (), and σ a is the standard deviation of the nueric values of attribute a. Note that in practice the square root in () is not perfored since the squared attribute distances are needed in (5) to copute H. Siilarly, the square root in (5) is not typically perfored in coputing H, since the squared distance (H 2 instead of D 2 in this case) is used in () to copute g, the activation of a hidden node.

4 . Epirical Results The Heterogeneous Radial Basis Function (HRBF) algorith was ipleented and tested on several databases fro the Machine Learning Database Repository at the University of California Irvine [5]. Each test consisted of ten trials, each using one of ten partitions of the data randoly selected fro the data sets, i.e., -fold cross-validation. Each trial consisted of building a network using % of the training instances in hidden and then using this network to classify the reaining % of the instances to see how any were classified correctly. In order to see what effect the new heterogeneous distance function has on accuracy, a hoogeneous version of the algorith was ipleented as well, which is exactly the sae as HRBF, except that it uses a noralized Euclidean distance function. This is accoplished by using noralized_diff a (x,y) instead of noralized_vd a (x,y) in () for noinal as well as for nueric attributes. The hoogeneous algorith will be referred to as the default algorith, or siply RBF. Both algoriths used the sae training sets and test sets for each trial. The average accuracy for each database over all trials is shown in Figure 2. A bold value indicates which value was highest for each database. One asterisk (*) indicates that the higher value is statistically significantly higher at a % confidence level, using a one-tailed paired t-test. Two asterisks (**) are used to ark differences that are significant at a 5% or higher confidence interval. Figure 2 also lists the nuber of continuous and noinal input attributes for each database. Note that the accuracy for every application that has only nueric attributes is exactly the sae for both RBF and HRBF. This is no surprise, since the distance functions are equivalent on nueric attributes. However, on the databases that have soe or all noinal attributes, HRBF obtained higher generalization accuracy than RBF in 2 out of 23 cases, of which were significant at the 5% level or above. RBF had a higher accuracy in only four cases, and only one of those (the Zoo data set) had a difference that was statistically significant. It is interesting to note that in the Zoo data set, 5 out of of the attributes are boolean, and the reaining attribute, while not linear, is actually an ordered attribute. These attributes are tagged as noinal, but the Euclidean distance function is appropriate for the as well. In all, HRBF perfored as well or better than the default algorith in 2 out of 3 cases. The above results indicate that the heterogeneous distance function is typically ore appropriate than the Euclidean distance function for applications with one or ore noinal attributes, and is equivalent to it for doains without noinal attributes. RBF Database (Euclidean) Annealing. Audiology 3. Australian-Credit 8. Bridges 52.3 Credit-Screening 5.3 DNA Prooters 5.2 Echocardiogra 8. Flags 5. Hayes-Roth 52. Heart 8. Heart-Disease (Hungarian). Heart-Disease (More) 5.5 Heart-Disease (Swiss) Hepatitis.33 Horse-Colic. House-Votes-8.22 Iage Segentation 8.8 Solar-Flare 8. Solar-Flare 2.53 Soybean-Large 3. Thyroid-Disease (Euthyroid). Tic-Tac-Toe 5.8 Zoo 8.8** Average: 5. Nueric Databases Breast-Cancer-Wisconsin Liver-Disorders Iris Pia-Indians-Diabetes Sat.Test Vowel Wine HRBF. 5.** 83.** 55.2* 83.8**.**. 5.** 5.** 8..2** ** **..** ** Nueric Attributes Noinal Attributes Figure 2. Coparative experiental results of RBF and HRBF.

5 5. Conclusions & Future Research The Heterogeneous Radial Basis Function (HRBF) network uses a noralized heterogeneous distance function which is typically ore appropriate than the Euclidean distance function for applications that have at least soe noinal or sybolic attributes. By using a ore appropriate distance function, higher generalization accuracy can be obtained on ost typical heterogeneous or copletely sybolic doains. Furtherore, the heterogeneous distance function is equivalent to a noralized Euclidean distance function in copletely nueric doains so generalization accuracy will be identical in those doains as well. In this paper the heterogeneous distance function was used with a probabilistic neural network for classification, which allowed very fast training at the cost of a large, static network. However, this function is appropriate for a wide range of basis function networks that use distance functions. Current research is seeking to test the heterogeneous distance function on a variety of other odels, including various Radial Basis Function networks and instance-based achine learning systes. The noralization factors are also being exained to see if they provide the best possible noralization. In addition, it appears that soe data which is tagged as noinal is often soewhat ordered. It is hypothesized that if the values of noinal attributes are randoly rearranged then the HRBF would perfor about the sae (since it does not depend on the ordering of noinal values), but that the hoogeneous RBF would suffer a loss in accuracy. The accuracy of this hypothesis and the severity of the loss in accuracy are currently being explored. The results of this research are encouraging, and show that heterogeneous distance functions can be used to apply basis function networks to a wider variety of applications and achieve higher generalization accuracy than the hoogeneous distance functions used in the past. References [] Chen, S., C. R. N. Cowen, and P. M. Grant, Orthogonal Least Squares Learning Algorith for Radial Basis Function Networks, IEEE Transactions on Neural Networks, vol. 2, no. 2, pp. 32-3,. [2] Cost, Scott, and Steven Salzberg, A Weighted Nearest Neighbor Algorith for Learning with Sybolic Features, Machine Learning, vol., pp. 5-8, 3. [3] Doingos, Pedro, Rule Induction and Instance-Based Learning: A Unified Approach, to appear in The 5 International Joint Conference on Artificial Intelligence (IJCAI-5), 5. [] Giraud-Carrier, Christophe, and Tony Martinez, An Efficient Metric for Heterogeneous Inductive Learning Applications in the Attribute-Value Language, Intelligent Systes, pp. 3-35, 5. [5] Murphy, P. M., and D. W. Aha, UCI Repository of Machine Learning Databases. Irvine, CA: University of California Irvine, Departent of Inforation and Coputer Science. Internet: ftp://ftp.ics.uci.edu/pub/achine-learning-databases, 3. [] Rachlin, John, Sion Kasif, Steven Salzberg, David W. Aha, Towards a Better Understanding of Meory-Based and Bayesian Classifiers, in Proceedings of the Eleventh International Machine Learning Conference, New Brunswick, NJ: Morgan Kaufann, pp ,. [] Renals, Steve, and Richard Rohwer, Phonee Classification Experients Using Radial Basis Functions, Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN 8), vol., pp. -,. [8] Ruelhart, D. E., and J. L. McClelland, Parallel Distributed Processing, MIT Press,. [] Spears, Willia M., Kenneth A. De Jong, Thoas Bäck, David B. Fogel, and Hugo de Garis, An Overview of Evolutionary Coputation, Proceedings of the European Conference on Machine Learning, vol., pp. 2-5, 3. [] Specht, Donald F., (2). Enhanceents to Probabilistic Neural Networks, in Proceedings International Joint Conference on Neural Networks (IJCNN 2), vol., pp. -8. [] Stanfill, C., and D. Waltz, Toward eory-based reasoning, Counications of the ACM, vol. 2, Deceber, pp ,. [2] Ventura, Dan, and Tony Martinez, An Epirical Coparison of Discretization Models, Proceedings of the th International Syposiu on Coputer and Inforation Sciences, pp. 3-5, 5. [3] Wasseran, Philip D., Advanced Methods in Neural Coputing, New York, NY: Van Nostrand Reinhold, pp. -, 3. [] Wilson, D. Randall, Prototype Styles of Generalization, Master s Thesis, Brigha Young University,. [5] Wong, Yiu-fai, How Gaussian Radial Basis Functions Work, Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN ), vol. 2, pp ,.