of a certain criterion δ(d) that determines the classifier imperviousness of the curse of dimensionality and low

Transcription

1 Neuro-fuzzy classifier based on the gaussian membership function U. V. Kulkarni S. V. Shinde Dept. of Computer Science & Engineering. Dept. of Information Technology SGGS Institute of Engineering & Technology Pimpri Chinchwad College of Engineering Nanded, India Pune, India Abstract This paper proposes the neuro-fuzzy classification model to perform the supervised classification of the data. In the proposed classification model, fuzzy membership matrix is formed by using Gaussian membership function. Membership matrix contains the membership of each feature value to the given classes. This membership matrix is given as an input to the artificial neural network and membership of each pattern to the given classes is obtained. Using the MAX defuzzification, target class for each pattern is predicted. The proposed model is applied to four datasets: Iris, Pima, Bupa and Phoneme. The datasets were obtained from the University of California at Irvine (UCI) machine learning repository & ELENA database. Accuracy of the results for medical databases is measured by using the performance measures- Accuracy, Sensitivity & Specificity and that for non medical databases- Percentage of overall class accuracy and Kappa index of agreement. The performance of the proposed classifier is compared with the well known classifiers: Artificial neural network and C4.5 algorithm. The experimental results show that the proposed classifier gives the higher accuracy with good KIA values than these classifiers. Index Terms Artificial neural network, fuzzy system, classification, membership function. INTRODUCTION A lot of data is generated in everyday life. The growing amounts of data has made manual analysis by data analysts a tedious task and sometimes impossible. Many hidden and potentially useful relationships may not be recognized by the analyst. The explosive growth of data requires an automated way to extract useful knowledge [1]. Data mining technology has become increasingly important in the field of large databases and data warehouses. Through data mining, interesting knowledge and regularities can be extracted and the discovered knowledge can be applied in the corresponding field to increase the working efficiency and to improve the quality of decision-making [1]. Data mining classification is one of the most frequent decision making tasks performed by humans. A classification problem occurs when an object needs to be assigned to a predefined group or class based on the number of observed attributes related to that object [2]. Classification problems involve assigning a class C i from a predefined class set C = { C 1, C 2..., C M } to an object, described as a point in a certain feature space x S is to find an optimal mapping N. The problem of designing a classifier D : SN C optimal in the sense of a certain criterion δ(d) that determines the classifier performance [3]. Usually, the final goal is to design a classifier that assigns class labels with the smallest possible error across the whole feature space. Artificial neural network (ANN) is one of the most commonly used classifier technique [4]. The reason for being commonly used is to present some properties such as learning from examples and exhibiting some capability for generalization beyond the training data [5]. Also they have universal approximation property [6] [7]. In data mining, the large number of dimensionality and the huge volume of data make neural networks competitive in classification due to their imperviousness of the curse of dimensionality and low computational cost [8]. Fuzzy set and logic theory [9] is the most prominent tool to handle uncertainty in decision-making. The major advantages of fuzzy system model are their robustness and transparency. Fuzzy system modeling achieves robustness by using fuzzy sets which incorporates imprecision to system models. In addition, unlike some system models, such as neural networks, the fuzzy system models are highly descriptive. Fuzzy systems allow a pattern to belong to multiple classes with different degrees. A summary of different fuzzy classifiers and their applications are described by Kuncheva [10]. When the classifier is a set of fuzzy rules, the resulting system is called a Fuzzy Rule-Based Classification System (FRBCS). FRBCS include a collection of if-then rules, stated using natural language. This is a form of knowledge representation that humans can easily understand, verify and refine. Indeed, the true power of fuzzy logic lies in its ability to handle and manipulate linguistic information [11] [12] [13]. If NN techniques are combined with fuzzy system then this hybrid system is called neuro-fuzzy system which is very effective for a variety of real world problems. Without any precise mathematical model, both ANN and fuzzy systems are adaptive in the estimation of the inputoutput function. ANN handles numeric and quantitative information while fuzzy systems can handle symbolic and qualitative data. In addition to this, in a fuzzy classifier patterns are assigned with a degree of belonging to different classes. Thus the partitions in fuzzy classifiers are soft and gradual rather than hard and crisp.

2 Fuzzification using GMF Fuzzy Neural Network Defuzzification using MAX Feature Vector Fuzzy Input Fuzzy Output Crisp Output Fig.1. classification model Therefore, an integration of neural and fuzzy systems should have the merits of both and it should enable one to build more intelligent decision making systems. Fuzzy set theory based hybrid classification systems are found to be more suitable and appropriate to handle these situations reasonably [14]. Also the two important aspects, namely learning and generalization capability play an important role in any pattern classification problem [15]. But the fuzzy systems lack capabilities of learning and have no memory [16]. This is why hybrid systems, particularly neuro-fuzzy systems, are becoming more and more popular for applications in many areas including process control, engineering design, financial trading, credit evaluation, medical diagnosis, and cognitive simulation. The proposed classification model presented in this paper is the hybrid neuro-fuzzy system designed to classify the patterns based on the output given by the fuzzy neural network (FNN) which trained on feature-wise membership values computed by Gaussian membership function. This is useful mainly when the classes are overlapping or ill-defined because the decision given by the FNN is soft which gives the membership value of the patter to each class.. The organization of the paper is as follows. Section II describes the proposed classification model. Different performance measurement parameters are discussed in Section III. Experimental results and discussions are given in Section IV. Finally, the concluding remarks are given in Section V. II. PROPOSED CLASSIFICATION MODEL The proposed model consists of three steps as illustrated in Fig.1: Fuzzification, FNN training and Defuzzification. The following subsections describe these steps in detail.the first step is the fuzzification of the input dataset by using Gaussian membership function. The output of this step is the membership matrix of size S (D C) where S is the number of rows which is equal to the number of samples and (D C) is the number of columns which is equal to the number of classes multiplied by number of features. The columns give the membership value of each feature to all the classes. Each row constitutes individual pattern of the dataset. This membership matrix is given as an input to FNN. This FNN is trained by using standard backpropagation algorithm which gives the output as the membership value of each pattern to the given the classes. Defuzzification step assigns the class to each pattern based on the membership value given by FNN. A. Fuzzification using the Gaussian Membership Function Fuzzification is the process of making a crisp quantity fuzzy [17]. It determines the degree of membership to which these crisp inputs belong to each of the appropriate fuzzy sets. Membership Value Feature Value Fig.2. Gaussian membership function Let X be a sample of size D comprised by a: n { x1, x2..., xd; xi, i 1,2... D}. A more convenient and concise way to define to define a Gaussian MF is to express it as a mathematical formula [18]. 2 ( xc) 2 2 G( x; c, ) e where x is the feature vale, c represents the membership function (MF) centre and σ determines MFs width. σ is also called standard deviation. Standard deviation shows how much variation or dispersion exists from the average (mean), or expected value. A low standard deviation indicates that the data points tend to be very close to the mean; high standard deviation indicates that the data points are spread out over a large range of values.

3 ... C 2... C D D... C D g1,1 x1 g1,2 x1 g1, x1 g1,1 x g1,2 x2 g1, x2 g1,1 x g1,2 x g1, x g x g x g x g x g x g x g x g x g x G X g x g x g x g x g x g x g x g x g x 2,1 1 2,2 1 2, C 1 2,1 2 2,2 2 2, C 2 2,1 D 2,2 D 2, C D... S,1 1 S,2 1 S, C 1 S,1 2 S,2 2 S, C 2 S,1 D S, 2 D S, C D ( c is given by: c d 1 S S j 1 x jd Hidden layer Output layer where d=1, 2..D, j=1, 2.S and x jd the d th feature of sample j. σ is given by:. Fuzzy... Input.. C 1 1 x c x c... x c S 1 d 1 d d 2 d d Sd d. C n where x 1d, x 2d. x Sd are the S feature values of d th feature vector and c d denotes the mean value of d as given in Eq. (4). The MF given in Eq. (2) generates the membership matrix G(X) of size S (D C) for the all the samples S as given in Eq. (3). In this matrix each value g s,c (d) represents the belongingness of d th feature of s th pattern to the c th class, where d=1,2,...d, s=1,2,...s and c=1,2,...c. B. Fuzzy Neural Network The ANN used here is referred as FNN because it takes fuzzy data as an input and gives the fuzzy output. As given in [19] the difference between the fuzzy neural network (FNN) and the crisp neural network is to have either at least one of the input data and values of weights to be fuzzy or both of them to be fuzzy. There are three types of FNN: FNN1, FNN2 and FNN3. In FNN1 the inputs are crisp values while the weights are fuzzy values. In FNN2 inputs are fuzzy numbers and the weights are crisp values. In FNN3 both inputs and weights are fuzzy values [20]. The traditional backpropagation algorithm used for crisp ANN can be used to train a fuzzy neural network. Considering the typical backpropagation, there are three layers used namely input, output and one hidden layer as shown in Fig. (3). One hidden layer is sufficient to model very high level of complexity. Each row of the membership matrix G(X) constitutes one pattern with D C values in it and total S rows are S patterns. Therefore the numbers of nodes used in the input layer are D C. The numbers of nodes used in the output layer are equal to the number of classes. Fig.3. Artificial neural network The proposed algorithm used to train the FNN is given as: 1. Calculate the center and standard deviation for each feature vector using Eq. (4) and Eq. (5). 2. Fuzzify the input vectors using Gaussian membership function using Eq. (2). 3. Initialize weight and biases. 4. Feed the training sample. 5. Propagate the inputs forward; we compute the net input and output of each unit in the hidden and output layers. 6. Back propagate the error. 7. Update weight and biases to reflect the propagated errors. 8. Until the required error repeat the steps from 4 to 7 for all samples. 9. Using Max operation, defuzzify the output of FNN and assign the class label. The output of j unit is calculated by using sigmoid activation function as follows [21]: 1 e 1 o j wo i ij i j where w ij is the weight of the connection from unit i in the previous layer to unit j; O i is the output of unit i from the previous layer; and θ j is the threshold of the unit. Weights are updated by the following equation:

4 w w l * E * O ij ij j i where l is the learning rate. For a unit j in the hidden layer, the error is computed as follows: E O (1 O ) E w j j j k jk k For a unit j in the output layer, the error is computed as follows: E O (1 O )( T O ) j j j j j The output given by the ANN is in the following form: O O O O O O A O O O 1,1 1,2 1, C 2,1 2,2 2, C S,1 S,2 S, C where O s,c is the output of the s th pattern to the c th class. C. Defuzzification The last step of the proposed classifier is defuzzification and is obtained by using a MAX operation to defuzzify the output of the FNN. The defuzzification means converting a fuzzy set to a crisp single-valued quantity. In classification and pattern recognition we may want to transform a fuzzy partition or pattern into a crisp partition or pattern. Mathematically, the defuzzification of a fuzzy set is the process of rounding it off from its location in the unit hypercube to the nearest vertex. As the matrix A given in Eq. (10) represents the membership values of each pattern to the given classes, defuzzification operation is applied to this matrix to predict the target class for each pattern. Mathematically the expression for this operation is given as: i 1,2, S j1,2, C and j c i, c i, j o o where o i,j is the membership value of i th pattern to the j th class and the pattern is classified to class c with the highest class membership value obtained through the FNN. III. PERFORMANCE MEASURES The measures used to analyze the performance of the proposed model for medical databases are accuracy, sensitivity and specificity. Accuracy specifies the overall percentage of correctly classified samples, sensitivity specifies the ability of classifier to classify positive samples and specificity specifies the ability to classify negative samples. Positive sample means disease positive and negative sample means disease negative sample. These are given by [24]: Sensitivity = Specificity = Total number of positive cases correctly diagnosed Total number of positive cases Total number of negative cases correctly diagnosed Total number of negative cases The performance measures used for non medical databases are Percentage of overall class Accuracy (PA) and Kappa Index of Agreement (KIA) [22] [25]. The PA value shows the total percentage of correctly classified patterns. The MC and PA parameters are calculated with respect to the total number of patterns with the given class labels. They do not provide the overall agreement of accuracy based on class-wise accuracy. Thus, to get an overall class-wise agreement based on the individual class accuracy KIA is used. A good KIA value signifies better agreement of the estimated data with the true one. The KIA value is estimated from a confusion or error matrix (CM) [22] [25]. A CM is a square matrix that represents the number of patterns assigned to a particular class relative to the true class. Any classifier classifies the given instance among four possible categories. If the positive instance is classified in positive class then it is true positive and if it is classified in negative category then it is false negative. If the negative instance is classified as positive then it is false positive and if it is classified as negative then it is true negative [10]. These counts form the confusion matrix for that classifier as given in Fig. 3: Hypothesized Class P N True Class P N Fig. 4. Confusion matrix This matrix produces many statistical measures of class accuracy including overall classification accuracy (the sum of the diagonal elements divided by the total number of samples) and KIA. KIA is defined as: r True Positives False Positives N X X X (. ) i1 ii i1 i i KIA, 2 r N ( X. ) i 1 i X i where the confusion matrix(cm) has r = number of rows X ii - number of observations in row i and column i X i+ - total number of observations in row i X +i - total number of observations in column i N - total number of observations in the CM r False Negatives True Negatives Accuracy = Total number of correctly diagnosed cases Total number of cases

5 Dataset IV. TABLE I. DATASET DESCRIPTION No. of Patterns No. of Features No. of Classes Iris Pima Bupa Phoneme EXPERIMENTAL RESULTS AND ANALYSIS In this section, we evaluate the performance of the proposed classifier model using well-known benchmark data sets used for classification as given in Table 1. The proposed model is applied on medical databases (Pima and Bupa) as well as on non-medical databases (Iris and Phoneme). The performance measures used for medical databases are accuracy, sensitivity and specificity and that for non-medical databases are PA and KIA. The computational complexity of the proposed model is more due to the reason that the input data needs to be fuzzified. Therefore to reduce the computations, all the experiments are conducted with 20% training and 80% testing data. The performance given by the proposed model is compared with well known classifier techniques like C4.5, ANN, Neuro-fuzzy given in [15]. A. BUPA Dataset Classification Bupa liver disorders dataset is created by Richard S. Forsyth. The dataset contains 2 classes of 145 and 200 samples each, where each class refers to the sensitivity of the disease either positive or negative. There are in total 6 numerical attributes and no missing value in the dataset [25]. Details of the dataset are given in Table I. After applying all the four classifier techniques on BUPA dataset following results as given in Table II are obtained. TABLE II. BUPA CLASSIFICATION RESULTS Class-wise Samples C1 50 C2 50 C3 50 C1 500 C2 268 C1 200 C2 145 C C Method Accuracy Sensitivity Specificity ANN C As seen from the Table II, the proposed has given the accuracy of 67.27% which is higher than all other referenced s. There is almost 16% difference between the accuracy of proposed and C4.5 while there is the difference of 11% between ANN and proposed. The accuracy of the Neuro-fuzzy is 8% lower than the proposed. Also the well balanced values of sensitivity and specificity show that the proposed has good capability to handle with disease positive and negative samples. For all other s the difference between the values of sensitivity and specificity is larger except the neurofuzzy. This analysis justifies the suitability of the proposed for medical databases. B. PIMA Dataset Classification Pima Indian Diabetes dataset is created by Vincent Sigillito. The dataset contains 2 classes of 500 and 268 instances each, where each class refers to the sensitivity of the disease either positive or negative. There are in total 8 numerical attributes and with some missing values in the dataset [25]. Details of the dataset are given in Table I. TABLE III. PIMA CLASSSIFICATION RESULTS Method Accuracy Sensitivity Specificity ANN C Neurofuzzy As given in Table III, the accuracy of proposed is 77.85%. This is higher than all other s. Also the difference between sensitivity and specificity values is lesser than all other s which justify its suitability for medical applications. C. IRIS Dataset Classification The IRIS dataset is created by R.A. Fisher and perhaps it is the best-known database found in the pattern recognition literature. The dataset contains 3 classes of 50 instances each, where each class refers to a type of IRIS plants. There are in total 4 numerical attributes and no missing value in the dataset [25]. Details of the dataset are given in Table I; Table IV gives the experimentation results applied on IRIS dataset. Neuro-fuzzy

6 TABLE IV. IRIS CLASSIFICATION RESULTS Method PA KIA ANN C Neuro-fuzzy As given in Table IV, the PA value given by proposed is which is higher than all other s. Although the proposed gives very small rise in the accuracy and KIA value, the main advantage of the proposed is that it can be applied with very small training data. D. PHONEME Dataset Classification The aim of this data set is to distinguish between nasal and oral vowels (two classes) [26]. It contains vowels coming from 1809 isolated syllables (for example: pa, ta, pan,...). Five different attributes were chosen to characterize each vowel. They are the amplitudes of the five first harmonics. TABLE V. PHONEME CLASSIFICATION RESULTS Method Accuracy KIA ANN C Neuro-fuzzy As given in Table V, the accuracy of the proposed for phoneme dataset is 85.93%. This accuracy is greater than all other mentioned classification s. Also the proposed has good value of KIA as compared to all s except ANN whose KIA value is same as proposed. has given the good results specifically when the database is unbalanced in proportion of samples for each class. The value of KIA for Neuro-fuzzy is 0 because the accuracy of 69.85% is obtained for class-1 and all the samples of class-2 are misclassified. V. CONCLUSION In this paper, the hybrid fuzzy classifier model is presented which is capable to handle the supervised classification problem. From the experimental results it is proved that the proposed classification model gives the higher accuracy as compared to well known classifiers like ANN, C4.5 and Neuro-fuzzy model. The use of the Gaussian membership function captures the variation/dispersion in the data and gives the membership value of each feature value to the fuzzy classes. These membership values give the freedom to handle each feature independently. In addition to this, this information is useful for higher level decision making. The proposed classification model has exploited the learning capability of ANN and robustness of fuzzy system which has incorporated the imprecision in to the system model REFERENCES [1] K. C. Tan, Q. Yu, C. M. Heng, and T. H. Lee, Evolutionary computing for knowledge discovery in medical diagnosis, Artificial Intelligence in Medicine, vol. 27, pp , [2] C. Lin, I. Chung, and C. Chen, An entropy-based quantum neuro-fuzzy inference system for classification applications, Neurocomputing, vol. 70, pp , [3] O. Cordon, M. Jesus, and F. Herrera, A proposal on reasoning s in fuzzy rule-based classification systems, International Journal of Approximate Reasoning, vol. 20, pp , [4] A. Zak, Neural model of underwater vehicle dynamics, International Journal of Mathematics and Computers in Simulation, vol. 1(2), pp , [5] S. Mukhopadhyay, C. Tang, J. Huang, M. Yu, and M. Palakal, A comparative study of genetic sequence classification algorithms, Proceedings of the th IEEE Workshop on Neural Networks for Signal Processing, pp , [6] R. Setiono, B. Baesens, and C. Mues, Recursive neural network rule extraction for data with mixed attributes, IEEE Trans. Neural Networks, vol. 19(2), pp , [7] W. Duch, R. Adamczak, and K. Grabczwski, A new ology of extraction, optimization and application of crisp and fuzzy logical rules, IEEE Trans. Neural Networks, vol. 12(2), pp , [8] S. Wang, Classification with incomplete survey data: A hopfield neural network approach, Computers & Operations Research, vol. 32, pp , [9] L. A. Zadeh, Fuzzy sets, Inform. Control, 1965, pp [10] L. I. Kuncheva, Fuzzy Classifier Design, Springer, Berlin, [11] A. Bdrdossy, L. Duckstein, Fuzzy rule-based modeling with applications to geophysical, biological and engineering systems, CRC Press, Boca Raton, [12] J. C. Bezdek, S. K. Pal, Fuzzy models for pattern recognition, s that search for structures in data, IEEE Press, New York, [13] Z. Chi, H. Yan, and T. Pham, Fuzzy algorithms with applications to image processing and pattern recognition, World Scientific, Singapore, [14] L. I. Kuncheva, Fuzzy classifier design, Springer-Verlag, [15] A. Ghosh, B. U. Shankar, and S.K. Meher, A novel approach to neoro-fuzzy classification, Neural Networks, vol. 22, pp , [16] T. Munkata, and Y. Jani, Fuzzy systems: An overview, Communications of ACM, vol. 37, pp , [17] J. R. Timothy, Fuzzy logic with engineering applications, 2nd ed., John Wiley & Sons Ltd., [18] G. J. Klir, B. Yuan, Fuzzy Sets and Fuzzy Logic: Theory and Applications, Prentice-Hall, India, 1997.

7 [19] L. H. Tsoukalas, and R. E. Uhrig, Fuzzy and neural approaches in engineering, A Wiley-Interscience Publication, [20] R. A. Aliev, and R. R. Aliev, Soft computing and its applications, World Scientific Publishing Co. Pvt. Ltd, [21] J. M. Zurada, Introduction to artificial neural systems. PWS Publishing Company, [22] D. H. Card, Using known map category marginal frequencies to improve estimates of thematic map accuracy, Photogrammetric Engineering and Remote Sensing, vol. 48(3), pp , [23] R. G. Congalton, A review of assessing the accuracy of classifications of remotely sensed data, Remote Sensing of Environment, vol. 37(1), pp , [24] C. K. Loo, Accurate and reliable diagnosis and classification using probabilistic ensemble simplified fuzzy ARTMAP, IEEE Transaction on Knowledge and Data Engineering, vol. 17(11), [25] UCI repository of machine learning databases, Department of Information and Computer Sciences, University of California, Irvine, [26] ELENA Databasem, Deliverable R3-B1-P Task B1: Databases (ESPRIT Basic Research Project Number 6891), MLG, UCL,Belgium. A/REAL/, 1995.