Feature-weighted k-nearest Neighbor Classifier
|
|
- Brett Taylor
- 1 years ago
- Views:
Transcription
1 Proceedings of the 27 IEEE Symposium on Foundations of Computational Intelligence (FOCI 27) Feature-weighted k-nearest Neighbor Classifier Diego P. Vivencio scar.br Estevam R. Hruschka Jr. M. do Carmo Nicoletti Edimilson B. dos Santos Sebastian D. C. O. Galvão dc.ufscar.br Abstract This paper proposes a feature weighting method based on 2 statistical test, to be used in conjunction with a k-nn classifier. Results of empirical experiments conducted using data from several knowledge domains are presented and discussed. Forty four out of forty five conducted experiments favoured the feature weighted approach and are empirical evidence that the proposed weighting process based on 2 is a good weighting strategy. Keywords: Feature Selection, Instance-Based Learning, Feature Ranking. 1 Introduction The nearest neighbor (NN) [Cover and Hart, 1967] is a learning process that simply stores the training examples as the representation of the concept. Each time a new instance needs to be classified, its similarity to the stored instances is measured and the new instance inherits the class of its closest instance. As commented in [Mitchell, 1997], NN learning approaches are especially sensitive to the dimensionality curse, thus, the identification of the relevant features in a training set can improve results of a NN learning process. Feature ranking algorithms are used to identify the relevance of features in a dataset and can be used with many different distance measures. In this work a feature ranking algorithm based on the Chi-squared distance measure is applied to rank the features of a dataset, and the ranked list of features is then used to define a feature weighting vector to be embedded in a k-nn classifier. The paper is organized along the following lines. Section 2 describes in a nutshell the class of feature ranking algorithms. In Section 3, an overview of the NN Classifier implemented in this work is given. Section 4 describes the proposed chi-squared ( 2 ) feature weighting ( 2 FW) algorithm. A description of the knowledge domains, the experiments, as well as the obtained results are presented in Section 5. Finally, Section 6 highlights the conclusions and points out future works. 2 Feature Ranking Feature Ranking can be defined as a category of feature selection methods. Feature selection has become the focus of research work in areas where datasets with tens or hundreds of thousands of variables are available [Guyon and Elissef, 23]. While, in a theoretical sense, having more features should only give us more discriminating power, the realworld provides us with many reasons why this is not generally the case [Koller and Sahami, 1996]. Reunanen [Reunanen, 23] observes that there can be many reasons for selecting only a subset of features: (i) it is cheaper to measure only a subset of features; (ii) prediction accuracy might be improved through exclusion of irrelevant features; (iii) the predictor to be built is usually simpler and potentially faster when less input features are used; (iv) knowing which features are relevant can give insight into the nature of the prediction problem at hand. Therefore, the problem of focusing on the most relevant information has become increasingly important for machine learning and data mining procedures [Blum and Langley, 1997]. When considering the method s output, feature selection methods can be grouped in two categories: feature ranking and minimum subset selection algorithms [Liu and Motoda, 1998]. A feature ranking algorithm defines a score to express the relevance of a feature; a subset selection algorithm tries to identify a subset of relevant features. In this work we are more interested in feature ranking algorithms. These methods require the evaluation of each feature using a specific distance metric, for identifying its degree of relevance (DR). The DR is then used to sort the features into a list called ranked list of features. Figure 1 shows the algorithm in a nutshell. The distance metric used in this work is chi-squared ( 2) statistical score [Liu and Motoda, 1998]. The motivation for using the 2 as a mutual information measure in a feature /7/$2. 27 IEEE 481
2 Proceedings of the 27 IEEE Symposium on Foundations of Computational Intelligence (FOCI 27) weighting task comes from the ability of this measure in ranking features [Witten and Frank, 2]. Considering a discrete variable i which can assume l possible values; a discrete variable j which can assume c possible values; n ij the observed frequency and e ij the expected frequency. Then, the 2 test can be used to measure the mutual information between variables i and j following equation 1. 2 ( nij eij ) e i 1 j 1 Figure 1. Feature ranking algorithm. 3 Nearest Neighbor Classifier l c Feature Ranking Algorithm Input: dataset, dist_measure; Output: ranked list of features List: ordered features list; DR:degree of relevance function; D_R: vector with the degree of relevance of each feature begin List emptylist; for each feature f do begin D_R[f] DR(f,dist_measure,dataset); /*degree of relevance */ List (f,dr[f]); endfor; order(list,dr); end. Basically NN techniques assume the class of the nearest instance from x, as the class of an instance x. In order to determine the nearest instance, NN techniques adopt a distance metric that measures the proximity of instance x to all stored instances. Various distance metrics can be used, including the Euclidean. Figure 2 presents the formal definition of NN technique found in [Cover and Hart, 1967]. The rule described in Figure 2 is more properly called the 1NN rule since it uses only one nearest neighbor. One of the variants of the 1NN rule is the k-nn rule, which considers the k nearest instances {i 1, i 2,.., i k } and decides upon the most frequent class in the set { i1, i2, ik }. Provided that the number of training instances is large enough, generally k-nn exhibits a good performance. As mentioned before, a disadvantage of k-nn methods is that all of the training instances must be retained. ij 2 (1) It is worth mentioning that when the set of training instances is large, k-nn based methods invest a high computational effort to perform a classification. This happens because for each new query (q) the whole training set T NN needs to be visited. The idea of weighting features when using a k-nn algorithm is used to give more importance, in the classification process, to relevant features. Consider a dataset containing m features (f1,f2, fm), where only one (fj) is relevant for classifying instances. Two instances having the same fj value, however, may be distant from each other in the m-dimensional space; this shows that non relevant features can play an important role in the classification process, dominating the distance measure. Trying to minimize this problem, many authors suggest [Wettschereck et al., 1997] the use of weights associated with features in order to guide the computation of the distance between them. As the main goal of this work is to empirically verify the performance of a k-nn algorithm using a feature weighting method based on 2 statistical score, a k-nn algorithm was implemented for classifying instances described by discrete (or ordinal) features and a nominal class. All the implemented functions followed the description given in [Mitchell, 1997]. Assume: n-dimensional feature space. M classes, numbered 1,2,,M. p training instances, each one expressed as a pair (x i, i ), for 1 i p where a) x i : training instance, expressed by a vector of pairs attributevalue xi (xi, x,..., x ) 1 i2 i1n b) i {1,2,,M} represents the correct class of the instance x i Let T NN = {(x 1, 1 ), (x 2, 2 ),, (x p, p )} be the nearest neighbor training set. Given an unknown instance x, the decision rule is to decide x is in class j if d(x,x j ) d(x,x i ), for 1 i p where d is some n-dimensional distance metric. Figure 2. 1-NN formal definition 4 The 2 Feature ( 2FW) The proposed chi-squared ( 2 ) feature weighting ( 2 FW) method can be classified as a mutual information approach for assigning features weights [Wettschereck et al., 1997]. In this sense, the mutual information (the Chi-Squared 482
3 Proceedings of the 27 IEEE Symposium on Foundations of Computational Intelligence (FOCI 27) statistical score) between the values of a feature and the class of the training instances are used to assign feature weights. 2 FW follows the idea presented in [Daelemans and Bosch, 1992] but instead of using the information gain [Quinlan, 1986] measure, it uses the 2 statistical score. As mentioned in Section 2, the motivation for using 2 as a mutual information measure in a feature weighting task is due to the ability of this measure in ranking features [Witten and Frank, 2]. The method can be defined in three steps. Initially the 2 score between each feature and the class must be defined using the whole dataset; subsequently, based on a weighting criterion, a vector containing the weights of each feature is created and finally the weights of each feature are used in the k-nn classification task. Figure 3 describes an algorithm for the first and the second steps of this process. The third step is the use of k-nn algorithm based on weights defined in the previous steps. Many criteria for defining a weight vector may be used. In this work, two criteria were used and the obtained results were compared. The first criterion, called (SW), defines a weight vector simply by ranking features ie, the features having the lowest 2 score have their weights set to 1, those with the second lowest-scored features have their weight set to 2 and so on. The process goes on until weights are assigned to the highest 2 scored features. In datasets where all m features have different 2 scores, the highest scored feature will have its weight set to m. The second criterion, called (NW), normalizes weights in the interval [..1]. According to this criterion, features with the highest 2 score have their weights set to 1. The other features have their weights linearly established according to their 2 score. Therefore, both criteria follow the idea given in [Wettschereck et al., 1997] that a feature weighting algorithm should assign low weights to features that provide little information for classification and higher weights to features that provide more reliable information. 2 FW algorithm Input: dataset described by features plus class, criterion; Output: Weight Vector (V) Chi: vector with the 2 score for each feature; V: weight vector; begin for each feature f do Chi[f] 2 (class,f,dataset); V create_vector(criterion,chi); end. Figure 3. 2 Feature ( 2FW) algorithm. In addition to the feature weighting process, our experiments performed k-nn classification using a feature domain normalization step and a distance-weighted learning approach. Both of them as described in [Mitchell, 1997]. 5 Experiments Experiments were conducted using fifteen datasets containing discrete or ordinal features to attempt to identify the consistency of the proposed method. An overview of the datasets is given in Table 1. Datasets Balance Scale, Congressional Voting Records, Letter, Optic Digits and Wisconsin Breast Cancer were downloaded from the UCI Machine Learning Repository [Merz and Murphy, 1998]. The others are their extended versions. Extended versions were generated inserting randomly distributed features in the corresponding original dataset. For instance, Balance +1 dataset is the original Balance Scale dataset enlarged with the addition of 1 new attributes (integer-valued) whose values were randomly chosen from interval [..1]. All the extended datasets were obtained using the same process. The only exceptions are the Voting +1 and Voting +2 datasets which were generated with the insertion of binary-valued features, since all the original features are binary-valued. Table 1. Datasets overview. Dataset # instances # features Balance Scale Balance Balance Wisconsin Breast Cancer Breast Breast Congressional Voting Records Voting Voting Letter 2 16 Letter Letter Optic Digits Optic Optic The average-case analysis of simple k-nn given in [Langley and Iba, 1993] as well as results of works using this learning method, such as those described in [Aha, 199 ; Langley and Sage, 1997], indicate that the number of training examples needed to reach a given accuracy grows exponentially with the number of irrelevant attributes. The extended datasets were generated to simulate samples containing large amounts of irrelevant information. Thus, it is possible to verify whether 2FW can contribute to reducing the effect of this undesirable characteristic of k-nn learning or not. All datasets have no missing feature values. 483
4 Proceedings of the 27 IEEE Symposium on Foundations of Computational Intelligence (FOCI 27) The aim of the experiments was to verify the soundness of the proposed feature weighting method articulated to a k-nn classification task. Thus, we consider that the expected behavior of a consistent feature weighting method is to generate higher Average Correct Classification Rates - ACCRs (obtained in classification using the weighting method) than the ACCRs obtained in classification without the weighting method. As it is hard to define the best value for k in k-nn classification, for each dataset the experiments used k=1, k=5 and k=1. Trying to minimize the training data bias, the experiments were conducted in a stratified 1-fold crossvalidation strategy. Tables 2, 3, 4, 5 and 6 show the ACCRs and the corresponding standard deviations (SDs) obtained. The column named k-nn shows results obtained using original k-nn algorithm as discussed in section 3. Columns named (SW) and (NW) present the ACCRs when using the SW and NW criteria respectively (as described in Section 4). As mentioned in the Balance Scale dataset documentation [Merz and Murphy, 1998] the four features are equally relevant for describing the concept this dataset represents. Any consistent weighting method consequently should assign the same weight to the four features. First line of Table 2 shows that this is the case for the proposed weighting method, since the obtained values are exactly the same. Analysis of results in the second and third rows of Table 2 indicate that the weighting method improved the ACCR values; particularly the NW criterion which resulted in better ACCR values for all values of k. In addition, it is interesting to note that original k-nn produced the worst results in the extended datasets. Therefore, the insertion of randomly distributed features contributed to reducing classification accuracy (mainly when using original k-nn). This can be explained by k-nn algorithm sensitivity to irrelevant features [Mitchell, 1997] and has been observed in all experiments conducted in this work. Table 2. Balance Scale Datasets (ACCR SD) Dataset K k-nn Balance Scale Balance Scale Balance Scale Table 3 presents the classification results obtained when using the Wisconsin Breast Cancer dataset and its extensions. In the original dataset, the highest ACCR was achieved using SW criterion (96.63%). Results obtained using the three criteria, however, are very similar and makes it hard to state which method is the best for this dataset. One reason for this behavior may be the fact that most of the original Wisconsin Breast Cancer dataset features are relevant (as showed in [Hruschka Jr. et al., 24]). On the other hand, the experiments using the extended datasets ie, Breast +1 and Breast +2, show that NW criterion delivered slightly better results than the others. Comparing ACCR values between original k-nn and NW, NW tends to perform better when random features are inserted in this dataset. Table 3. Wisconsin Breast Cancer Datasets (ACCR SD) Dataset Wisconsin Breast Cancer K k- NN Breast Breast Tables 2 and 3 reveal that the weighting method did not optimize the classification results when using the original Balance Scale and the original Wisconsin Breast Cancer datasets. Table 4, however, shows that for the Congressional Voting Records domain, the weighting method improved results even when working with the original dataset. This suggests that the weighting method found irrelevant features in the original dataset as well as in the extended ones. Table 4 also shows that NW criterion produced the best results in all experiments using this domain. Table 5 displays the ACCR values achieved using the Optic Digits domain. As with the previous domains, the insertion of irrelevant features has not significantly changed ACCR values when using either weighting criteria (WS and NW). It can be seen, however, that the inserted random features contributed to diminishing ACCR values when using original k-nn. 484
5 Proceedings of the 27 IEEE Symposium on Foundations of Computational Intelligence (FOCI 27) Table 4. Congressional Voting Records Datasets (ACCR SD) Dataset Congress Voting Records k-nn K Voting Voting Table 6. Letter Datasets (ACCR SD) Dataset K k-nn Letter Letter Letter Table 5. Optic Digits Datasets (ACCR SD) Dataset K k-nn Optic Digits Optic Optic It is worth mentioning that the insertion of 1 and 2 random features in the Optic Digits domain tends to generate less impact in the classification results because of the large number of features used for describing this dataset. Inserting 1 new features in a dataset already containing 64 features (Optical Digits) should produce less impact than inserting 1 new features in a dataset containing only 4 features (Balance Scale). The last domain used in this work is the Letter dataset [Merz and Murphy, 1998] and the results of the experiments are in Table 6 and are consistent with what has been said previously. The NW criterion delivered the best ACCR values in eight out of nine experiments. Based on results it is possible to state that when using original k-nn the ACCR values become lower as the number of irrelevant features increases. On the other hand, when using 2FW the ACCR values tend to stabilize even in the presence of many irrelevant features. 6 Conclusions This paper proposes, describes and evaluates a new weighting method, based on the 2 statistical test, for a k- NN classifier. Comparative analyses of results were carried out and the proposed method appears to be very promising. The work shows accuracy values obtained using k-nn (for k=1, k=5 and k=1) on 15 datasets 5 of them are from the UCI Machine Learning Repository [Merz and Murphy, 1998] and the remaining 1 are their extended versions. Results of 45 experiments using a stratified 1-fold crossvalidation strategy are described and discussed. Considering both proposed weighting vector generation criteria ( and ), fortyfour out of forty five experiments favoured the proposed 2FW method. It is interesting to mention that the artificial insertion of irrelevant features in the original datasets, induced greater differences among ACCR values (comparing weighted k- NN versus original k-nn). Therefore, the proposed weighting process tends to have good performance in datasets with a large number of irrelevant features. As commented in [Blum and Langley, 1997], this behavior is an important issue in a feature selection (or feature weighting) algorithm. This paper also proposes two vector weighting generation criteria namely (SW) and (NW). Analysis of data in the previous tables indicates that NW criterion does achieve the best results (compared with either SW or original k-nn) in 39 out of 45 experiments, which can be considered a good performance. We intend next to analyze NW performance in datasets with greater value intervals (for example, [..1] instead of [..1] used in this experiments). Another interesting line of research would be to verify if the dependence between features can influence the behaviour of our proposed method. Therefore, instead of only verifying the relation between each feature and the class, it would be interesting to analyze the relation between 485
6 Proceedings of the 27 IEEE Symposium on Foundations of Computational Intelligence (FOCI 27) each pair of features. Presently we are conducting an empirical comparison with other weighting methods, still using k-nn as the classifier algorithm. References [1] [Cover and Hart, 1967] Cover, T. and Hart, P., Nearest neighbor pattern classification, IEEE Transactions on Information Theory 13, 1967, pp [2] [Mitchell, 1997] Mitchell, T., Machine Learning, The McGraw-Hill Companies, Inc, [3] [Guyon and Elissef, 23] Guyon, I. and Elisseeff, A., An introduction to variable and feature selection, Journal of Machine Learning Research, 3, pp , 23. [4] [Koller and Sahami, 1996] Koller, D. and Sahami, M., Toward optimal feature selection, Proceedings of the 13 th International Conference on Machine Learning, pp , July, [5] [Reunanen, 23] Reunanen, J., Overfitting in making comparisons between variable selection methods, Journal of Machine Learning Research, 3, pp , 23. [ Irvine, CA, University of California. [13] [Langley and Iba, 1993] Langley, P. and Iba, W., Average-case analysis of a nearest neighbor algorithm. In: Proceedings IJCAI-93, Chambery, France, , [14] [Aha, 199] Aha, D., A study of instance-based algorithms for supervised learning tasks: mathematical, empirical and psychological evaluations. Doctoral Dissertation, Department of Information and Computer Science, University of California, Irvine, CA, 199. [15] [Langley and Sage, 1997] Langley, P. and Sage, S., Scaling to domains with many irrelevant features. Computational Learning Theory and Natural Learning Systems. Vol. 4, MIT Press, Cambridge, MA, [16] [Hruschka Jr. et al., 24] Hruschka Jr., E. R., Hruschka, E. R. and Ebecken, N. F. F., Feature Selection by Bayesian Networks, Lecture Notes in Artificial Intelligence. Berlin: Springer-Verlag, v.36. p , 24. [6] [Blum and Langley, 1997] Blum, A. L. and Langley, P., Selection of relevant features and examples in machine learning, Artificial Intelligence, pp , [7] [Liu and Motoda, 1998] Liu, H. and Motoda, H., Feature Selection for Knowledge Discovery and Data Mining. Kluwer Academic, [8] [Witten and Frank, 25] Witten, I. H. and Frank, E., Data Mining Practical Machine Learning Tools and Techniques with Java Implementations, second edition, Morgan Kaufmann Publishers, USA, 25. [9] [Wettschereck et al., 1997] Wettschereck, D., Aha, D. & Mohri, T., A review and empirical evaluation of feature weighting methods for a class of lazy learning algorithms, Artificial Intelligence Review, 11: , [1] [Daelemans and Bosch, 1992] Daelemans, W. and Bosch, A., Generalization performance of backpropagation learning on a syllabification task, Proc. of TWLT3: Connectionism and Natural Language Processing, pp , [11] [Quinlan, 1986] Quinlan, J. R., Induction of decision trees, Machine Learning, 1:81-16, [12] [Merz and Murphy, 1998] Merz, C. J. and Murphy, P. M., UCI Repository of Machine Learning Databases, 486
WEIGHTED K NEAREST NEIGHBOR CLASSIFICATION ON FEATURE PROJECTIONS 1
WEIGHTED K NEAREST NEIGHBOR CLASSIFICATION ON FEATURE PROJECTIONS 1 H. Altay Güvenir and Aynur Akkuş Department of Computer Engineering and Information Science Bilkent University, 06533, Ankara, Turkey
CloNI: clustering of JN -interval discretization
CloNI: clustering of JN -interval discretization C. Ratanamahatana Department of Computer Science, University of California, Riverside, USA Abstract It is known that the naive Bayesian classifier typically
Forward Feature Selection Using Residual Mutual Information
Forward Feature Selection Using Residual Mutual Information Erik Schaffernicht, Christoph Möller, Klaus Debes and Horst-Michael Gross Ilmenau University of Technology - Neuroinformatics and Cognitive Robotics
Encoding Words into String Vectors for Word Categorization
Int'l Conf. Artificial Intelligence ICAI'16 271 Encoding Words into String Vectors for Word Categorization Taeho Jo Department of Computer and Information Communication Engineering, Hongik University,
Using Decision Boundary to Analyze Classifiers
Using Decision Boundary to Analyze Classifiers Zhiyong Yan Congfu Xu College of Computer Science, Zhejiang University, Hangzhou, China yanzhiyong@zju.edu.cn Abstract In this paper we propose to use decision
Chapter 12 Feature Selection
Chapter 12 Feature Selection Xiaogang Su Department of Statistics University of Central Florida - 1 - Outline Why Feature Selection? Categorization of Feature Selection Methods Filter Methods Wrapper Methods
Comparing Univariate and Multivariate Decision Trees *
Comparing Univariate and Multivariate Decision Trees * Olcay Taner Yıldız, Ethem Alpaydın Department of Computer Engineering Boğaziçi University, 80815 İstanbul Turkey yildizol@cmpe.boun.edu.tr, alpaydin@boun.edu.tr
The Role of Biomedical Dataset in Classification
The Role of Biomedical Dataset in Classification Ajay Kumar Tanwani and Muddassar Farooq Next Generation Intelligent Networks Research Center (nexgin RC) National University of Computer & Emerging Sciences
Performance Analysis of Data Mining Classification Techniques
Performance Analysis of Data Mining Classification Techniques Tejas Mehta 1, Dr. Dhaval Kathiriya 2 Ph.D. Student, School of Computer Science, Dr. Babasaheb Ambedkar Open University, Gujarat, India 1 Principal
Constructing X-of-N Attributes with a Genetic Algorithm
Constructing X-of-N Attributes with a Genetic Algorithm Otavio Larsen 1 Alex Freitas 2 Julio C. Nievola 1 1 Postgraduate Program in Applied Computer Science 2 Computing Laboratory Pontificia Universidade
Univariate Margin Tree
Univariate Margin Tree Olcay Taner Yıldız Department of Computer Engineering, Işık University, TR-34980, Şile, Istanbul, Turkey, olcaytaner@isikun.edu.tr Abstract. In many pattern recognition applications,
Concept Tree Based Clustering Visualization with Shaded Similarity Matrices
Syracuse University SURFACE School of Information Studies: Faculty Scholarship School of Information Studies (ischool) 12-2002 Concept Tree Based Clustering Visualization with Shaded Similarity Matrices
Redundancy Based Feature Selection for Microarray Data
Redundancy Based Feature Selection for Microarray Data Lei Yu Department of Computer Science & Engineering Arizona State University Tempe, AZ 85287-8809 leiyu@asu.edu Huan Liu Department of Computer Science
SYMBOLIC FEATURES IN NEURAL NETWORKS
SYMBOLIC FEATURES IN NEURAL NETWORKS Włodzisław Duch, Karol Grudziński and Grzegorz Stawski 1 Department of Computer Methods, Nicolaus Copernicus University ul. Grudziadzka 5, 87-100 Toruń, Poland Abstract:
Efficient Case Based Feature Construction
Efficient Case Based Feature Construction Ingo Mierswa and Michael Wurst Artificial Intelligence Unit,Department of Computer Science, University of Dortmund, Germany {mierswa, wurst}@ls8.cs.uni-dortmund.de
MIT 801. Machine Learning I. [Presented by Anna Bosman] 16 February 2018
MIT 801 [Presented by Anna Bosman] 16 February 2018 Machine Learning What is machine learning? Artificial Intelligence? Yes as we know it. What is intelligence? The ability to acquire and apply knowledge
Fuzzy Partitioning with FID3.1
Fuzzy Partitioning with FID3.1 Cezary Z. Janikow Dept. of Mathematics and Computer Science University of Missouri St. Louis St. Louis, Missouri 63121 janikow@umsl.edu Maciej Fajfer Institute of Computing
CS4445 Data Mining and Knowledge Discovery in Databases. A Term 2008 Exam 2 October 14, 2008
CS4445 Data Mining and Knowledge Discovery in Databases. A Term 2008 Exam 2 October 14, 2008 Prof. Carolina Ruiz Department of Computer Science Worcester Polytechnic Institute NAME: Prof. Ruiz Problem
Weka ( )
Weka ( http://www.cs.waikato.ac.nz/ml/weka/ ) The phases in which classifier s design can be divided are reflected in WEKA s Explorer structure: Data pre-processing (filtering) and representation Supervised
An Empirical Comparison of Spectral Learning Methods for Classification
An Empirical Comparison of Spectral Learning Methods for Classification Adam Drake and Dan Ventura Computer Science Department Brigham Young University, Provo, UT 84602 USA Email: adam drake1@yahoo.com,
COMPARISON OF SUBSAMPLING TECHNIQUES FOR RANDOM SUBSPACE ENSEMBLES
COMPARISON OF SUBSAMPLING TECHNIQUES FOR RANDOM SUBSPACE ENSEMBLES SANTHOSH PATHICAL 1, GURSEL SERPEN 1 1 Elecrical Engineering and Computer Science Department, University of Toledo, Toledo, OH, 43606,
Classification of Hand-Written Numeric Digits
Classification of Hand-Written Numeric Digits Nyssa Aragon, William Lane, Fan Zhang December 12, 2013 1 Objective The specific hand-written recognition application that this project is emphasizing is reading
K-means clustering based filter feature selection on high dimensional data
International Journal of Advances in Intelligent Informatics ISSN: 2442-6571 Vol 2, No 1, March 2016, pp. 38-45 38 K-means clustering based filter feature selection on high dimensional data Dewi Pramudi
Feature Subset Selection Using the Wrapper Method: Overfitting and Dynamic Search Space Topology
From: KDD-95 Proceedings. Copyright 1995, AAAI (www.aaai.org). All rights reserved. Feature Subset Selection Using the Wrapper Method: Overfitting and Dynamic Search Space Topology Ron Kohavi and Dan Sommerfield
A Comparison of Text-Categorization Methods applied to N-Gram Frequency Statistics
A Comparison of Text-Categorization Methods applied to N-Gram Frequency Statistics Helmut Berger and Dieter Merkl 2 Faculty of Information Technology, University of Technology, Sydney, NSW, Australia hberger@it.uts.edu.au
Penalizied Logistic Regression for Classification
Penalizied Logistic Regression for Classification Gennady G. Pekhimenko Department of Computer Science University of Toronto Toronto, ON M5S3L1 pgen@cs.toronto.edu Abstract Investigation for using different
Quantifying the Impact of Learning Algorithm Parameter Tuning
Quantifying the Impact of Learning Algorithm Tuning Niklas Lavesson and Paul Davidsson Dept. of Software and Systems Engineering Blekinge Institute of Technology Box 520, SE-372 25, Ronneby, Sweden Abstract
Improving Classifier Performance by Imputing Missing Values using Discretization Method
Improving Classifier Performance by Imputing Missing Values using Discretization Method E. CHANDRA BLESSIE Assistant Professor, Department of Computer Science, D.J.Academy for Managerial Excellence, Coimbatore,
Cost-sensitive C4.5 with post-pruning and competition
Cost-sensitive C4.5 with post-pruning and competition Zilong Xu, Fan Min, William Zhu Lab of Granular Computing, Zhangzhou Normal University, Zhangzhou 363, China Abstract Decision tree is an effective
Study on Classifiers using Genetic Algorithm and Class based Rules Generation
2012 International Conference on Software and Computer Applications (ICSCA 2012) IPCSIT vol. 41 (2012) (2012) IACSIT Press, Singapore Study on Classifiers using Genetic Algorithm and Class based Rules
Using a genetic algorithm for editing k-nearest neighbor classifiers
Using a genetic algorithm for editing k-nearest neighbor classifiers R. Gil-Pita 1 and X. Yao 23 1 Teoría de la Señal y Comunicaciones, Universidad de Alcalá, Madrid (SPAIN) 2 Computer Sciences Department,
HALF&HALF BAGGING AND HARD BOUNDARY POINTS. Leo Breiman Statistics Department University of California Berkeley, CA
1 HALF&HALF BAGGING AND HARD BOUNDARY POINTS Leo Breiman Statistics Department University of California Berkeley, CA 94720 leo@stat.berkeley.edu Technical Report 534 Statistics Department September 1998
CS570: Introduction to Data Mining
CS570: Introduction to Data Mining Classification Advanced Reading: Chapter 8 & 9 Han, Chapters 4 & 5 Tan Anca Doloc-Mihu, Ph.D. Slides courtesy of Li Xiong, Ph.D., 2011 Han, Kamber & Pei. Data Mining.
Univariate and Multivariate Decision Trees
Univariate and Multivariate Decision Trees Olcay Taner Yıldız and Ethem Alpaydın Department of Computer Engineering Boğaziçi University İstanbul 80815 Turkey Abstract. Univariate decision trees at each
Efficient Pruning Method for Ensemble Self-Generating Neural Networks
Efficient Pruning Method for Ensemble Self-Generating Neural Networks Hirotaka INOUE Department of Electrical Engineering & Information Science, Kure National College of Technology -- Agaminami, Kure-shi,
Feature Selection for Supervised Classification: A Kolmogorov- Smirnov Class Correlation-Based Filter
Feature Selection for Supervised Classification: A Kolmogorov- Smirnov Class Correlation-Based Filter Marcin Blachnik 1), Włodzisław Duch 2), Adam Kachel 1), Jacek Biesiada 1,3) 1) Silesian University
Feature Selection Algorithm with Discretization and PSO Search Methods for Continuous Attributes
Feature Selection Algorithm with Discretization and PSO Search Methods for Continuous Attributes Madhu.G 1, Rajinikanth.T.V 2, Govardhan.A 3 1 Dept of Information Technology, VNRVJIET, Hyderabad-90, INDIA,
Analytical model A structure and process for analyzing a dataset. For example, a decision tree is a model for the classification of a dataset.
Glossary of data mining terms: Accuracy Accuracy is an important factor in assessing the success of data mining. When applied to data, accuracy refers to the rate of correct values in the data. When applied
Dynamic Clustering of Data with Modified K-Means Algorithm
2012 International Conference on Information and Computer Networks (ICICN 2012) IPCSIT vol. 27 (2012) (2012) IACSIT Press, Singapore Dynamic Clustering of Data with Modified K-Means Algorithm Ahamed Shafeeq
Information theory methods for feature selection
Information theory methods for feature selection Zuzana Reitermanová Department of Computer Science Faculty of Mathematics and Physics Charles University in Prague, Czech Republic Diplomový a doktorandský
ASSOCIATIVE CLASSIFICATION WITH KNN
ASSOCIATIVE CLASSIFICATION WITH ZAIXIANG HUANG, ZHONGMEI ZHOU, TIANZHONG HE Department of Computer Science and Engineering, Zhangzhou Normal University, Zhangzhou 363000, China E-mail: huangzaixiang@126.com
Feature Selection Using Modified-MCA Based Scoring Metric for Classification
2011 International Conference on Information Communication and Management IPCSIT vol.16 (2011) (2011) IACSIT Press, Singapore Feature Selection Using Modified-MCA Based Scoring Metric for Classification
Feature Extraction Using ICA
Feature Extraction Using ICA Nojun Kwak, Chong-Ho Choi, and Jin Young Choi School of Electrical Eng., ASRI, Seoul National University San 56-1, Shinlim-dong, Kwanak-ku, Seoul 151-742, Korea {triplea,chchoi}@csl.snu.ac.kr,
Comparison of Various Feature Selection Methods in Application to Prototype Best Rules
Comparison of Various Feature Selection Methods in Application to Prototype Best Rules Marcin Blachnik Silesian University of Technology, Electrotechnology Department,Katowice Krasinskiego 8, Poland marcin.blachnik@polsl.pl
Random Search Report An objective look at random search performance for 4 problem sets
Random Search Report An objective look at random search performance for 4 problem sets Dudon Wai Georgia Institute of Technology CS 7641: Machine Learning Atlanta, GA dwai3@gatech.edu Abstract: This report
Wild Mushrooms Classification Edible or Poisonous
Wild Mushrooms Classification Edible or Poisonous Yulin Shen ECE 539 Project Report Professor: Hu Yu Hen 2013 Fall ( I allow code to be released in the public domain) pg. 1 Contents Introduction. 3 Practicality
1) Give decision trees to represent the following Boolean functions:
1) Give decision trees to represent the following Boolean functions: 1) A B 2) A [B C] 3) A XOR B 4) [A B] [C Dl Answer: 1) A B 2) A [B C] 1 3) A XOR B = (A B) ( A B) 4) [A B] [C D] 2 2) Consider the following
K-Nearest Neighbour Classifier. Izabela Moise, Evangelos Pournaras, Dirk Helbing
K-Nearest Neighbour Classifier Izabela Moise, Evangelos Pournaras, Dirk Helbing Izabela Moise, Evangelos Pournaras, Dirk Helbing 1 Reminder Supervised data mining Classification Decision Trees Izabela
CS249: ADVANCED DATA MINING
CS249: ADVANCED DATA MINING Classification Evaluation and Practical Issues Instructor: Yizhou Sun yzsun@cs.ucla.edu April 24, 2017 Homework 2 out Announcements Due May 3 rd (11:59pm) Course project proposal
Attribute Reduction using Forward Selection and Relative Reduct Algorithm
Attribute Reduction using Forward Selection and Relative Reduct Algorithm P.Kalyani Associate Professor in Computer Science, SNR Sons College, Coimbatore, India. ABSTRACT Attribute reduction of an information
Distribution-free Predictive Approaches
Distribution-free Predictive Approaches The methods discussed in the previous sections are essentially model-based. Model-free approaches such as tree-based classification also exist and are popular for
Using Google s PageRank Algorithm to Identify Important Attributes of Genes
Using Google s PageRank Algorithm to Identify Important Attributes of Genes Golam Morshed Osmani Ph.D. Student in Software Engineering Dept. of Computer Science North Dakota State Univesity Fargo, ND 58105
A Novel Template Matching Approach To Speaker-Independent Arabic Spoken Digit Recognition
Special Session: Intelligent Knowledge Management A Novel Template Matching Approach To Speaker-Independent Arabic Spoken Digit Recognition Jiping Sun 1, Jeremy Sun 1, Kacem Abida 2, and Fakhri Karray
Data Mining. Introduction. Hamid Beigy. Sharif University of Technology. Fall 1394
Data Mining Introduction Hamid Beigy Sharif University of Technology Fall 1394 Hamid Beigy (Sharif University of Technology) Data Mining Fall 1394 1 / 20 Table of contents 1 Introduction 2 Data mining
Lazy Decision Trees Ronny Kohavi
Lazy Decision Trees Ronny Kohavi Data Mining and Visualization Group Silicon Graphics, Inc. Joint work with Jerry Friedman and Yeogirl Yun Stanford University Motivation: Average Impurity = / interesting
University of Florida CISE department Gator Engineering. Data Preprocessing. Dr. Sanjay Ranka
Data Preprocessing Dr. Sanjay Ranka Professor Computer and Information Science and Engineering University of Florida, Gainesville ranka@cise.ufl.edu Data Preprocessing What preprocessing step can or should
Speeding up Logistic Model Tree Induction
Speeding up Logistic Model Tree Induction Marc Sumner 1,2,EibeFrank 2,andMarkHall 2 Institute for Computer Science University of Freiburg Freiburg, Germany sumner@informatik.uni-freiburg.de Department
MANY factors affect the success of data mining algorithms
IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL. 15, NO. 6, NOVEMBER/DECEMBER 2003 1437 Benchmarking Attribute Selection Techniques for Discrete Class Data Mining Mark A. Hall and Geoffrey Holmes
CLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS
CLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS CHAPTER 4 CLASSIFICATION WITH RADIAL BASIS AND PROBABILISTIC NEURAL NETWORKS 4.1 Introduction Optical character recognition is one of
Graph Matching: Fast Candidate Elimination Using Machine Learning Techniques
Graph Matching: Fast Candidate Elimination Using Machine Learning Techniques M. Lazarescu 1,2, H. Bunke 1, and S. Venkatesh 2 1 Computer Science Department, University of Bern, Switzerland 2 School of
Data Preprocessing. Data Preprocessing
Data Preprocessing Dr. Sanjay Ranka Professor Computer and Information Science and Engineering University of Florida, Gainesville ranka@cise.ufl.edu Data Preprocessing What preprocessing step can or should
Global Metric Learning by Gradient Descent
Global Metric Learning by Gradient Descent Jens Hocke and Thomas Martinetz University of Lübeck - Institute for Neuro- and Bioinformatics Ratzeburger Allee 160, 23538 Lübeck, Germany hocke@inb.uni-luebeck.de
Rough Set Approaches to Rule Induction from Incomplete Data
Proceedings of the IPMU'2004, the 10th International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems, Perugia, Italy, July 4 9, 2004, vol. 2, 923 930 Rough
PRIVACY-PRESERVING MULTI-PARTY DECISION TREE INDUCTION
PRIVACY-PRESERVING MULTI-PARTY DECISION TREE INDUCTION Justin Z. Zhan, LiWu Chang, Stan Matwin Abstract We propose a new scheme for multiple parties to conduct data mining computations without disclosing
Time Complexity Analysis of the Genetic Algorithm Clustering Method
Time Complexity Analysis of the Genetic Algorithm Clustering Method Z. M. NOPIAH, M. I. KHAIRIR, S. ABDULLAH, M. N. BAHARIN, and A. ARIFIN Department of Mechanical and Materials Engineering Universiti
K-Nearest Neighbor Classification Approach for Face and Fingerprint at Feature Level Fusion
K-Nearest Neighbor Classification Approach for Face and Fingerprint at Feature Level Fusion Dhriti PEC University of Technology Chandigarh India Manvjeet Kaur PEC University of Technology Chandigarh India
A Novel Feature Selection Framework for Automatic Web Page Classification
International Journal of Automation and Computing 9(4), August 2012, 442-448 DOI: 10.1007/s11633-012-0665-x A Novel Feature Selection Framework for Automatic Web Page Classification J. Alamelu Mangai 1
A Rough Set Approach for Generation and Validation of Rules for Missing Attribute Values of a Data Set
A Rough Set Approach for Generation and Validation of Rules for Missing Attribute Values of a Data Set Renu Vashist School of Computer Science and Engineering Shri Mata Vaishno Devi University, Katra,
COMPARISON OF DIFFERENT CLASSIFICATION TECHNIQUES
COMPARISON OF DIFFERENT CLASSIFICATION TECHNIQUES USING DIFFERENT DATASETS V. Vaithiyanathan 1, K. Rajeswari 2, Kapil Tajane 3, Rahul Pitale 3 1 Associate Dean Research, CTS Chair Professor, SASTRA University,
Index Terms Data Mining, Classification, Rapid Miner. Fig.1. RapidMiner User Interface
A Comparative Study of Classification Methods in Data Mining using RapidMiner Studio Vishnu Kumar Goyal Dept. of Computer Engineering Govt. R.C. Khaitan Polytechnic College, Jaipur, India vishnugoyal_jaipur@yahoo.co.in
Text Classification based on Limited Bibliographic Metadata
Text Classification based on Limited Bibliographic Metadata Kerstin Denecke, Thomas Risse L3S Research Center Hannover, Germany denecke@l3s.de Thomas Baehr German National Library of Science and Technology
SIMILARITY MEASURES FOR MULTI-VALUED ATTRIBUTES FOR DATABASE CLUSTERING
SIMILARITY MEASURES FOR MULTI-VALUED ATTRIBUTES FOR DATABASE CLUSTERING TAE-WAN RYU AND CHRISTOPH F. EICK Department of Computer Science, University of Houston, Houston, Texas 77204-3475 {twryu, ceick}@cs.uh.edu
A Two Stage Zone Regression Method for Global Characterization of a Project Database
A Two Stage Zone Regression Method for Global Characterization 1 Chapter I A Two Stage Zone Regression Method for Global Characterization of a Project Database J. J. Dolado, University of the Basque Country,
UNSUPERVISED STATIC DISCRETIZATION METHODS IN DATA MINING. Daniela Joiţa Titu Maiorescu University, Bucharest, Romania
UNSUPERVISED STATIC DISCRETIZATION METHODS IN DATA MINING Daniela Joiţa Titu Maiorescu University, Bucharest, Romania danielajoita@utmro Abstract Discretization of real-valued data is often used as a pre-processing
Stability Assessment of Electric Power Systems using Growing Neural Gas and Self-Organizing Maps
Stability Assessment of Electric Power Systems using Growing Gas and Self-Organizing Maps Christian Rehtanz, Carsten Leder University of Dortmund, 44221 Dortmund, Germany Abstract. Liberalized competitive
Machine Learning Classifiers and Boosting
Machine Learning Classifiers and Boosting Reading Ch 18.6-18.12, 20.1-20.3.2 Outline Different types of learning problems Different types of learning algorithms Supervised learning Decision trees Naïve
CHAPTER 4 STOCK PRICE PREDICTION USING MODIFIED K-NEAREST NEIGHBOR (MKNN) ALGORITHM
CHAPTER 4 STOCK PRICE PREDICTION USING MODIFIED K-NEAREST NEIGHBOR (MKNN) ALGORITHM 4.1 Introduction Nowadays money investment in stock market gains major attention because of its dynamic nature. So the
Using PageRank in Feature Selection
Using PageRank in Feature Selection Dino Ienco, Rosa Meo, and Marco Botta Dipartimento di Informatica, Università di Torino, Italy {ienco,meo,botta}@di.unito.it Abstract. Feature selection is an important
Kernel Combination Versus Classifier Combination
Kernel Combination Versus Classifier Combination Wan-Jui Lee 1, Sergey Verzakov 2, and Robert P.W. Duin 2 1 EE Department, National Sun Yat-Sen University, Kaohsiung, Taiwan wrlee@water.ee.nsysu.edu.tw
A Closest Fit Approach to Missing Attribute Values in Preterm Birth Data
A Closest Fit Approach to Missing Attribute Values in Preterm Birth Data Jerzy W. Grzymala-Busse 1, Witold J. Grzymala-Busse 2, and Linda K. Goodwin 3 1 Department of Electrical Engineering and Computer
Fuzzy-Rough Feature Significance for Fuzzy Decision Trees
Fuzzy-Rough Feature Significance for Fuzzy Decision Trees Richard Jensen and Qiang Shen Department of Computer Science, The University of Wales, Aberystwyth {rkj,qqs}@aber.ac.uk Abstract Crisp decision
Cluster Analysis. Mu-Chun Su. Department of Computer Science and Information Engineering National Central University 2003/3/11 1
Cluster Analysis Mu-Chun Su Department of Computer Science and Information Engineering National Central University 2003/3/11 1 Introduction Cluster analysis is the formal study of algorithms and methods
MetaData for Database Mining
MetaData for Database Mining John Cleary, Geoffrey Holmes, Sally Jo Cunningham, and Ian H. Witten Department of Computer Science University of Waikato Hamilton, New Zealand. Abstract: At present, a machine
WEKA: Practical Machine Learning Tools and Techniques in Java. Seminar A.I. Tools WS 2006/07 Rossen Dimov
WEKA: Practical Machine Learning Tools and Techniques in Java Seminar A.I. Tools WS 2006/07 Rossen Dimov Overview Basic introduction to Machine Learning Weka Tool Conclusion Document classification Demo
ORT EP R RCH A ESE R P A IDI! " #$$% &' (# $!"
R E S E A R C H R E P O R T IDIAP A Parallel Mixture of SVMs for Very Large Scale Problems Ronan Collobert a b Yoshua Bengio b IDIAP RR 01-12 April 26, 2002 Samy Bengio a published in Neural Computation,
Analyzing Outlier Detection Techniques with Hybrid Method
Analyzing Outlier Detection Techniques with Hybrid Method Shruti Aggarwal Assistant Professor Department of Computer Science and Engineering Sri Guru Granth Sahib World University. (SGGSWU) Fatehgarh Sahib,
Rule extraction from support vector machines
Rule extraction from support vector machines Haydemar Núñez 1,3 Cecilio Angulo 1,2 Andreu Català 1,2 1 Dept. of Systems Engineering, Polytechnical University of Catalonia Avda. Victor Balaguer s/n E-08800
Feature selection has long been an active research topic in machine learning. Beginning
E n h a n c i n g I n f o r m a t i o n Parameter Tuning for Induction-Algorithm- Oriented Feature Elimination Ying Yang and Xindong Wu, University of Vermont Feature selection has long been an active
A Unified Framework to Integrate Supervision and Metric Learning into Clustering
A Unified Framework to Integrate Supervision and Metric Learning into Clustering Xin Li and Dan Roth Department of Computer Science University of Illinois, Urbana, IL 61801 (xli1,danr)@uiuc.edu December
Statistical dependence measure for feature selection in microarray datasets
Statistical dependence measure for feature selection in microarray datasets Verónica Bolón-Canedo 1, Sohan Seth 2, Noelia Sánchez-Maroño 1, Amparo Alonso-Betanzos 1 and José C. Príncipe 2 1- Department
Version Space Support Vector Machines: An Extended Paper
Version Space Support Vector Machines: An Extended Paper E.N. Smirnov, I.G. Sprinkhuizen-Kuyper, G.I. Nalbantov 2, and S. Vanderlooy Abstract. We argue to use version spaces as an approach to reliable
Enhancing K-means Clustering Algorithm with Improved Initial Center
Enhancing K-means Clustering Algorithm with Improved Initial Center Madhu Yedla #1, Srinivasa Rao Pathakota #2, T M Srinivasa #3 # Department of Computer Science and Engineering, National Institute of
Discriminate Analysis
Discriminate Analysis Outline Introduction Linear Discriminant Analysis Examples 1 Introduction What is Discriminant Analysis? Statistical technique to classify objects into mutually exclusive and exhaustive
Using PageRank in Feature Selection
Using PageRank in Feature Selection Dino Ienco, Rosa Meo, and Marco Botta Dipartimento di Informatica, Università di Torino, Italy fienco,meo,bottag@di.unito.it Abstract. Feature selection is an important
4. Feedforward neural networks. 4.1 Feedforward neural network structure
4. Feedforward neural networks 4.1 Feedforward neural network structure Feedforward neural network is one of the most common network architectures. Its structure and some basic preprocessing issues required
Look-Ahead Based Fuzzy Decision Tree Induction
IEEE TRANSACTIONS ON FUZZY SYSTEMS, VOL. 9, NO. 3, JUNE 2001 461 Look-Ahead Based Fuzzy Decision Tree Induction Ming Dong, Student Member, IEEE, and Ravi Kothari, Senior Member, IEEE Abstract Decision
An Initial Seed Selection Algorithm for K-means Clustering of Georeferenced Data to Improve
An Initial Seed Selection Algorithm for K-means Clustering of Georeferenced Data to Improve Replicability of Cluster Assignments for Mapping Application Fouad Khan Central European University-Environmental
Morisita-Based Feature Selection for Regression Problems
Morisita-Based Feature Selection for Regression Problems Jean Golay, Michael Leuenberger and Mikhail Kanevski University of Lausanne - Institute of Earth Surface Dynamics (IDYST) UNIL-Mouline, 115 Lausanne
Building Classifiers using Bayesian Networks
Building Classifiers using Bayesian Networks Nir Friedman and Moises Goldszmidt 1997 Presented by Brian Collins and Lukas Seitlinger Paper Summary The Naive Bayes classifier has reasonable performance
Pattern Classification based on Web Usage Mining using Neural Network Technique
International Journal of Computer Applications (975 8887) Pattern Classification based on Web Usage Mining using Neural Network Technique Er. Romil V Patel PIET, VADODARA Dheeraj Kumar Singh, PIET, VADODARA