A New RBFNDDA-KNN Network and Its Application to Medical Pattern Classification

Transcription

1 A New RBFNDDA-KNN Network and Its Appliation to Medial Pattern Classifiation Shing Chiang Tan 1*, Chee Peng Lim 2, Robert F. Harrison 3, R. Lee Kennedy 4 1 Faulty of Information Siene and Tehnology, Multimedia University, Malaysia * (orresponding author): stan@mmu.edu.my 2 Centre for Intelligent Systems Researh, Deakin University, Australia 3 Department of Automati Control and Systems Engineering, University of Sheffield, UK 4 Shool of Mediine, Deakin University, Australia Abstrat. In this paper, a new variant of the Radial Basis Funtion Network with Dynami Deay Adjust (RBFNDDA) is introdued for undertaking pattern lassifiation problems with noisy data. The RBFNDDA network is integrated with the k-nearest neighbours algorithm to form the proposed RBFNDDA-KNN model. Given a set of labelled data samples, the RBFNDDA network undergoes a onstrutive learning algorithm that exhibits a greedy insertion behaviour. As a result, many prototypes (hidden neurons) that represent small (with respet to a threshold) lusters of labelled data are introdued in the hidden layer. This results in a large network size. Suh small prototypes an be aused by noisy data, or they an be valid representatives of small lusters of labelled data. The KNN algorithm is used to identify small prototypes that exist in the viinity (with respet to a distane metri) of the majority of large prototypes from different lasses. These small prototypes are treated as noise, and are, therefore, pruned from the network. To evaluate the effetiveness of RBFNDDA-KNN, a series of experiments using pattern lassifiation problems in the medial domain is onduted. Benhmark and real medial data sets are experimented, and the results are ompared, analysed, and disussed. The outomes show that RBFNDDA-KNN is able to learn information with a ompat network struture and to produe fast and aurate lassifiation results. Keywords: Radial Basis Funtion Neural Network, Nearest Neighbour, Pattern Classifiation 1 Introdution During the past few deades, many different soft-omputing tehniques suh as artifiial neural networks [1-3], deision trees [4-6], and fuzzy system [7-9] have been suessfully applied as intelligent data proessing tools to various domains, e.g., biomediine [1], [4], [7], manufaturing proesses [2], [5], [8], and power systems [3], [6], [9]. This line of researh ontinues to attrat the attention of the soft-omputing ommunity to further develop more effiient models for learning information from databases and for dealing with various omplex problems enountered in different adfa, p. 1, Springer-Verlag Berlin Heidelberg 2011

2 domains. Learning from databases is a hallenging task. Normally, real-world data samples are ompliated to analyse, as they usually ontain noise. Indeed, noisy data denote a group of unrepresentative samples in the database [10]. On the other hand, the database an also ontain some small (with respet to a threshold) lusters of labelled data that ontain important information for supervised learning. As an example, in medial diagnosis, some patients may develop some distintive reations or symptoms pertaining to a treatment, e.g. a severe hest pain [11] an be aused by a novel ourrene of drug allergy, and it should be aptured as an important ase for further study. Hene, it is useful to have a learning model that is able to first identify small prototypes (i.e. representatives of small lusters of labelled data) in the viinity (with respet to a distane metri) of large ones. Then, the learning model needs to estimate whether these small prototypes are noise, or they are valid representatives of small lusters of labelled data (whih arry important information) whereby these small representatives belong to the same lass as the majority of the large prototypes in the viinity (with respet to a distane metri). The former needs to be pruned, while the latter needs to be preserved in the network. This is the main motivation of the urrent study. The main purpose of this study is to devise a new variant of the Radial Basis Funtion Network with Dynami Deay Adjust (RBFNDDA) [12] model for pattern lassifiation with noisy data. Speifially, the k-nearest neighbour algorithm [13-14] is integrated with RBFNDDA to form the proposed RBFNDDA-KNN model to ahieve this purpose. The original RBFNDDA network has a fast onstrutive supervised learning proess, and is able to produe good lassifiation rates. One of the salient features of RBFNDDA is that its struture grows inrementally during the training proess, whereby new hidden neurons are reated to inlude new training instanes in form of prototypes. However, RBFNDDA is sensitive to noise. Its onstrutive learning algorithm manifests a greedy insertion behaviour, whih an lead to an oversized network that ontains a lot of small prototypes (hidden neurons). The proposed RBFNDDA-KNN network is designed with the aim to prune small noisy prototypes while preserving other small useful prototypes into its network struture. To demonstrate the effetiveness of the proposed RBFNDDA-KNN network, ase studies in the medial pattern lassifiation domain are onduted. Both benhmark and real medial data sets are used in the experiments. The results are analysed, ompared, and disussed. The organization of this paper is as follows. In Setion 2, the bakground of RBFNDDA-based networks is first presented. Then, the proposed RBFNDDA-KNN network is explained in detail. In Setion 3, an empirial study using a number of benhmark data sets from the UCI mahine-learning repository [15] and a real medial data set from a UK hospital is desribed. The results are ompared and analysed. In Setion 4, a summary pertaining to the urrent study and suggestions for further work is inluded.

3 2 Classifiation Methods In this setion, the dynamis of RBFNDDA and KNN are explained. This is followed by a desription of the proposed RBFNDDA-KNN network for seletively pruning small prototypes for undertaking pattern lassifiation problems. 2.1 Radial Basis Funtion Neural Network with Dynami Deay Adjustment (RBFNDDA) RBFNDDA inherits the salient features of the Probabilisti Neural Network (PNN) [16] and the Restrited Coulomb Energy Network (RCEN) [17-18]. It extrats the probabilisti harateristis of a stati PNN and the ability of adjusting the prototype (hidden neuron) width of the RCEN, and embeds them into a network struture that an grow inrementally during training. The RBFNDDA network is a three-layered arhiteture, as shown in Fig. 1. k outputs y y1, y2,, y k.. Summation Layer lass 1 prototypes (hidden neurons) weights w j lass k prototypes (hidden neurons) Hidden Layer Input Layer input pattern (x, ) x = x 1,, x n ; = lass label Fig. 1. The RBFNDDA arhiteture RBFNDDA requires only two user-defined thresholds (i.e., and ) for adjusting the prototype width during the network training phase. These two userdefined thresholds are shown in Fig. 2. The thresholds separate a prototype from its neighbours (prototypes) of different lasses. For eah training pattern, threshold sets the minimum orret-lassifiation probability for the orret lass, whereas threshold sets the highest probability allowable for an inorret lass. This orresponds to an area of onflit whereby neither mathing nor onfliting training patterns an reside, as shown in Fig. 2. In other words, thresholds and ontrol

4 the size of the overlapping region of eah prototype in RBFNDDA. Aording to [10], [12], and are suitable to be used as the default settings, and they are good enough to give satisfatory lassifiation results. p(x) 1 Thresholds Areas of onflit Fig. 2. The two user-defined thresholds of a radial basis hidden node RBFNDDA employs the DDA algorithm to onstrut its network struture. During the training phase, additional prototypes are inserted into the network in order to learn new information from inoming data. All ommitted prototypes are aumulated along the ourse of its training session. Fig. 3 illustrates the learning proedure of RBFNDDA in a single epoh. The RBFNDDA training proedure an be summarised as follows. First, all the weights of the prototypes are initialized to zero (in order to prevent aumulation of dupliated information pertaining to the training patterns). On presentation of a new training sample, if the sample is orretly lassified by a few existing prototypes, the weight of the largest prototype is inreased. However, if the training sample is inorretly lassified, a new prototype is introdued to inlude the sample in the network. In this ase, the new prototype has the sample as its entre, with its weight set to one. Its initial width is then set in suh a way to avoid overlapping with the neighbouring prototypes of different lasses. The next step is to perform width shrinking for all prototypes of different lasses, if their ativations aused by the training sample are above. Details of the RBFNDDA an be found in [10], [12]. 2.2 The k-nearest Neighbour (KNN) Algorithm The KNN algorithm [13-14] is a simple instane-based learning method. It lassifies a new instane (training sample) based on the k losest instanes in the data spae. Normally, a distane metri, e.g., the Eulidean distane, is used to measure the loseness between the new sample and the existing ones. Using a single neighbour lassifiation sheme, the new sample is lassified as belonging to the lass of the losest sample. If more than one (i.e., k-) nearest neighbours from the existing samples is onsidered for lassifiation, then the new sample is assigned to the majority lass of its losest k neighbours.

5 // reset weights 1 FORALL prototypes 2 w j 0 3 ENDFOR // train one omplete epoh p j DO 4 FORALL training patterns, k j 5 IF k j x DO: p : p x THEN 6 w j w j 1 7 ELSE // introdue a new prototype (hidden neuron) 8 m m 1 9 w 1 m 10 zm x // adapt radii 11 2 s s z j zm rm min 1 j m s ln 12 END // shrink radii of onfliting prototypes (hidden neurons) 13 FORALL s, 1 i ms DO END 16 END s s r i minri, s j m z z ln 2 Fig. 3. The learning algorithm of RBFNDDA in one epoh [10] 2.3 The RBFNDDA-KNN Network RBFNDDA-KNN is an extension of RBFNDDA for whih its learning proess enompasses an instane-based KNN algorithm to deal with noisy information. A olletion of prototypes is formed in the network after a training epoh of RBFNDDA- KNN. This olletion of prototypes aptures information from all training samples. Some of the prototypes are ommitted to over only a small number of training samples. These small prototypes are aused either by noisy data or valid representatives of small lusters of labelled data. The KNN algorithm helps avoid establishing a large number of small prototypes in the network, espeially in the presene of noisy samples. In eah training epoh, suh small prototypes, j p, 1,2,,C, j 1,2,,m, are identified if their weights are smaller than a threshold value, v.

6 On the other hand, those prototypes with weights larger than v, i.e.,, 1,2, l,m l p 1,2,,C, are deemed useful. When applying the KNN algorithm, let N be a set of k entres of p l prototypes that are the losest to eah entre of k p l p j meas- ured using the standard Eulidean distane metri. The majority lass of determined aording to If s, prototype max Nk p 1,2,, C l N is k p l s arg (1) p j is removed from the network beause it is onsidered as a noisy prototype loating lose to a group of large prototypes from other lasses in the data spae. However, if s, it is retained in the network beause both prototypes belong to the same lass, exept one is a large prototype while another is a small prototype (whih may ontain some speifi and important information) that needs to be preserved. Fig. 4 shows a shemati diagram (a two-dimensional example) that uses the KNN algorithm to identify and remove a small noisy prototype from the RBFNDDA-KNN network. X X Fig. 4. The use of the KNN algorithm to identify and remove noisy prototypes (hidden neurons). Note that filled elements represent large prototypes (larger than v) and hollow elements represent small prototypes (smaller than v) from two lasses (C1=irle and C2=triangles). With k=3, the small C2 prototype (hollow triangle) is preserved while the small C1 prototype (hollow irle) is removed (marked by X) from the RBFNDDA-KNN network. 3 Experimental Study The proposed RBFNDDA-KNN network was evaluated with three benhmark and one real medial data sets in this study. The benhmark data sets used were taken from the UCI Mahine Learning Repository [15]. The real medial data set omprised Myoardial Infartion (MI) patient reords olleted from a hospital in the UK. The purpose of the experiment was two-fold: (i) to evaluate the effetiveness of the proposed RBFNDDA-KNN network by omparing its lassifiation performane with RBFNDDA (as reported in [10]) using the benhmark data sets; (ii) to demonstrate the appliability of the proposed RBFNDDA-KNN network using real MI patient reords. A number of experimental runs were onduted, and the average results and

7 their 95% onfidene intervals were estimated using the bootstrap method [19]. The bootstrapped results were reported mainly beause of the ability of the bootstrap method to provide aurate estimates of statistial parameters (e.g. the average results and their onfidene intervals) based on a small sample size. These results were then analysed, ompared, and disussed. 3.1 Benhmark Medial Problems Three medial data sets from the UCI mahine learning repository were used in this experimental study. They were the diabetes, the aner and the heart data sets. Their respetive number of data samples, number of attributes, number of lasses were 768, 8, 2, 699, 9, 2, and 270, 13, 2. The study aimed to ompare the lassifiation performanes of RBFNDDA-KNN (from this study), RBFNDDA-T (from [10]), and original RBFNDDA (from [10]) based on the same experimental setup as in [10]. Eah data set was divided equally into a training set and a test set. For larity, RBFNDDA-T is a modified version of RBFNDDA that has been enhaned with an online pruning algorithm for removing noise from the network after every training epoh. If a prototype does not ontain a suffiient number of training samples (by omparing the number of training samples against a threshold), then it is treated as a noise and is kept in a blaklist. In this regard, the prototype is removed for subsequent training epohs. In [10], both RBFNDADDA and RBFNDDA-T were trained in multiple epohs with 0. 40, In addition, RBFNDDA-T used a threshold setting of v=2 [10]. In this study, the RBFNDDA-KNN network was trained repeatedly in multiple epohs with the same setting of 0. 40, 0. 20, and with k=10. The experiment was onduted using a personal omputer with a onfiguration of Genuine Intel (R) CPU 2160 at 1.80GHz 1.80GHz, 4GB of memory. The trained network was then evaluated using unseen samples from the test set. The experiment was onduted for eight runs, and the average results in terms of the auray rates and number of prototypes were omputed. Table 1 shows the lassifiation performanes from the RBFNDDA-KNN, RBFNDDA and RBFNDDA-T networks. Note that the lassifiation results in terms of the auray rates and number of prototypes of RBNDDDA and RBFNDDA-T are those reported in [10]. Table 1. The lassifiation results of RBFNDDA-KNN, RBFNDDA-T and RBFNDDA from the Diabetes, Caner and Heart ase studies (A.-average test auray rate; #Prototypes average number of prototypes (hidden neurons); standard deviations are typed in round brakets). Data Set RBFNDDA (from [10]) RBFNDDA-T (from [10]) RBFNDDA-KNN (from this study) A.(%) #Prototypes A.(%) #Prototypes A.(%) #Prototypes Diabetes 74.35(0.97) 288.5(6.1) 73.50(2.38) 65.6(4.2) 71.14(1.92) 161.9(8.5) Caner 96.86(0.99) 70.6(13.4) 96.90(0.39) 38.3(4.6) 96.85(0.47) 38.7(5.3) Heart 79.26(2.83) 83.6(3.3) 79.82(3.54) 32.6(2.1) 79.90(2.45) 39.5(9.5)

8 The auray rates of RBFNDDA-KNN, RBFNDDA and RBFNDDA-T are almost similar, as shown in Table 1. However, RBFNDDA-KNN used approximately 50% fewer number of prototypes than RBFNDDA to enode information from the data sets. By analysing the RBFNDDA-KNN struture, it was notied that those small prototypes, whih were loated among other prototypes representing the general onept of other lass(es) in the data spae, ould be identified as noise and, therefore, removed from the network. The network size of RBFNDDA-KNN was larger than that of RBFNDDA-T. This was beause the former did not simply remove all small prototypes during the training proess, while the latter would remove all prototypes smaller than the speified threshold and put them in a blaklist. However, RBFNDDA-KNN had an advantage from the omputational viewpoint. While RBFNDDA-T ould identify and aumulate a list of unrepresentative samples to prevent the network from absorbing noise, RBFNDDA-KNN, on the other hand, did not need suh a blaklist to prune noisy prototypes from its network. This omputational advantage of RBFNDDA-KNN was learly exhibited in the next ase study. 3.2 A Real Medial Problem In this experiment, a data set omprising 500 patient reords olleted from the Northern General Hospital, Sheffield, UK, was used to evaluate the appliability of the RBFNDDA-KNN network to real-world medial lassifiation problems. The task was to ategorise the data samples into to two ategories: suspeted Myoardial Infartion (MI) and non-mi ases. There were about 31% and 69% MI and non-mi ases, respetively, in the data set. Eah data sample omprised a total of 29 features that were found to be signifiant to MI diagnosis, whih inluded linial measurements (e.g. ECG readings), physial symptoms (e.g. sweating, vomiting, desription of hest pain), and other relevant information (e.g. age, smoker/ex- smoker). The features were oded as binary numbers, with a 1/0 indiating presene/absene of symptoms. In addition, there were two real-valued features for age and duration of pain (normalised between 0 and 1), respetively. The experimental setup and the network parameter settings were the same as those desribed in Setion 3.1. To better evaluate the stability of the performanes, the experiment was repeated for 100 times with different sequenes of the training samples. In addition, the Data Retention Rate (DRR) [20], i.e. a useful measure of network ompatness, was also omputed, as follows. number of hidden neurons (prototypes) data retention rate (2) number of training samples Table 2 summarises the bootstrapped results of RBFNDDA, RBFNDDA-T, and RBFNDDA-KNN. The average auray rate of RBFNDDA-KNN is slightly higher than those of RBFNDDA-T and RBFNDDA. The number of prototypes of RBFNDDA-KNN and RFNDDA-T are smaller than that of RBFNDDA. The empirial results indiated that RBFNDDA-KNN was able to mitigate the influene of unrepresentative data samples and to produe high lassifiation auray. Between

9 RBFNDDA-KNN and RBFNDDA-T, the former reated more number of prototypes than the latter, as shown by the DRR results. However, the omputational time of RBFNDDA-KNN was faster than that of RBFNDDA-T, although the former had more than twie the number of prototypes than the latter. As explained in Setion 3.1, RBFNDDA-T stored a list of unrepresentative samples to hek against noise from the training samples in eah training epoh. Hene, additional omputational time was required. This problem ould beome severe when large and noisy training data sets were used. In short, RBFNDDA-KNN ould form a ompat network struture (as ompared with RBFNDDA) and produe fast and aurate lassifiation results (as ompared with both RBFNDDA and RBFNDDA-T). Table 2. The average boostrapped results of RBFNDDA-KNN, RBFNDDA-T and RBFNDDA from the MI ase study (the ranges in the square brakets indiate the 95% onfidene intervals of the estimated average boostrapped results). Classifier A. (%) #Prototypes DDR Time(s) RBFNDDA [ ] RBFNDDA-T [ ] RBFNDDA-KNN [ ] [ ] 30.9 [ ] 69.8 [ ] [ ] [ ] [ ] 4 Summary In this paper, a new variant of the RBFNDDA network has been proposed for undertaking pattern lassifiation problems with noisy data. The proposed network, known as RBFNDDA-KNN, is based on integration between RBFNDDA and the KNN algorithm. The proposed RBFNDDA-KNN network is able to identify small noisy prototypes and preserve small useful prototypes that represent small lusters of important labelled data. A series of experiments using both benhmark and real medial data sets has been onduted to evaluate the effetiveness of RBFNDDA-KNN. The results have shown that RBFNDDA-KNN is able to learn and enode information from the database using a smaller number of prototypes than RBFNDDA, and with improved lassifiation results. As ompared with RBFNDDA-T, RBFNDDA-KNN has exhibited a larger network struture. However, RBFNDDA-T does not need to keep a list of unrepresentative samples (as what RBFNDDA-T does to identify noisy data) during the training proess. As suh, RBFNDDA-KNN is able to produe fast and aurate results in the presene of noisy samples in the database. For future work, additional experiments to asertain the effetiveness of the proposed RBFNDDA-KNN network in various real-world lassifiation domains will be onduted. In addition, rule extration from RBFNDDA-KNN an be arried out so that it an provide explanation for its predited outomes.

10 Referenes 1. Lee, J., Steele, C.M., Chau, T.: Classifiation of Healthy and Abnormal Swallows Based on Aelerometry and Nasal Airflow Signals. Artifiial Intelligene in Mediine 52, (2011) 2. Yu, J.-b., Xi, L.-f.: A Neural Network Ensemble-Based Model for On-Line Monitoring and Diagnosis of Out-of-Control Signals in Multivariate Manufaturing Proesses. Expert Systems with Appliations 36, (2009) 3. Barakat, M., Druaux, F., Lefebvre, D., Khalil, M., Mustapha, O.: Self Adaptive Growing Neural Network Classifier for Faults Detetion and Diagnosis. Neuroomputing 74, (2011) 4. Son, C.-S., Kim, Y.-N., Kim, H.-S., Park, H.-S., Kim, M.-S.: Deision-Making Model for Early Diagnosis of Congestive Heart Failure Using Rough Set and Deision Tree Approahes. Journal of Biomedial Informatis 45, (2012) 5. Ciflikli, C., Kahya-Özyirmidokuz, E.: Implementing a Data Mining Solution for Enhaning Carpet Manufaturing Produtivity. Knowledge-Based Systems 23, (2010) 6. Upendar, J., Gupta, C.P., Singh, G.K.: Statistial Deision-Tree Based Fault Classifiation Sheme for Protetion of Power Transmission Lines. Eletrial Power and Energy Systems 36, 1-12 (2012) 7. Alayón, S., Robertson, R., Warfield, S.K., Ruiz-Alzola, J.: A Fuzzy System for Helping Medial Diagnosis of Malformations of Cortial Development. Journal of Biomedial Informatis 40, (2007) 8. Piltan, M., Mehmanhi, E., Ghaderi, S.F.: Proposing a Deision-Making Model Using Analytial Hierarhy Proess and Fuzzy Expert System for Prioritizing Industries in Installation of Combined Heat and Power Systems. Expert Systems with Appliations 39, (2012) 9. Kazemi, M.V., Moradi, M., Kazemi, R.V.: Minimization of Powers Ripple of Diret Power Controlled DFIG by Fuzzy Controller and Improved Disrete Spae Vetor Modulation. Eletri Power Systems Researh 89, (2012) 10. Paetz, J.: Reduing the Number of Neurons in Radial Basis Funtion Networks with Dynami Deay Adjustment. Neuroomputing 62, (2004) 11. Davey, P., Lalloo, D.G.: Drug Indued Chest Pain Rare but Important. Postgrad Med J. 76, (2000) 12. Berthold, M.R.; Diamond, J.: Construtive Training of Probabilisti Neural Networks. Neuroomputing 19, (1998) 13. Cover, T.M., Hart, P.E.: Nearest Neighbor Pattern Classifiation. IEEE Transations on Information Theory 13, 21 7 (1967) 14. Dasarathy, B.V.: Nearest Neighbor (NN) Norms: NN Pattern Classifiation Tehniques. IEEE Computer Soiety Press, Los Alamitos, CA (1991) 15. Asunion, A., Newman, D.J.: UCI Mahine Learning Repository [ Irvine, CA: University of California, Shool of Information and Computer Siene (2007) 16. Speht, D.F.: Probabilisti Neural Networks. Neural Networks 3, (1990) 17. Hudak, M.H.: RCE Classifiers: Theory and Pratie. Cybernetis and Systems 23, (1992) 18. Reilly, D.L., Cooper, L.N., Elbaum, C.: A Neural Model for Category Learning. Biologial Cybernetis 45, (1982)