PSOk-NN: A Particle Swarm Optimization Approach to Optimize k-nearest Neighbor Classifier

Size: px

Start display at page:

Download "PSOk-NN: A Particle Swarm Optimization Approach to Optimize k-nearest Neighbor Classifier"

Chrystal McKenzie
5 years ago
Views:

1 PSOk-NN: A Particle Swarm Optimization Approach to Optimize k-nearest Neighbor Classifier Alaa Tharwat 1,2,5, Aboul Ella Hassanien 3,4,5 1 Dept. of Electricity- Faculty of Engineering- Suez Canal University, Ismaalia, Egypt. 2 Faculty of Engineering, Ain Shams University, Cairo, Egypt. 3 Faculty of Computers Information, Cairo University, Cairo, Egypt. 4 Faculty of Computers and Information, Beni Suef University - Egypt. 5 Scientific Research Group in Egypt (SRGE) Swarm Work Shop - Nov. 7, 15 Alaa Tharwat 1,2,5, Aboul Ella Hassanien 3,4,5 Swarm Work Shop - Nov. 7, 15 1 /

2 Agenda Introduction Theoretical Background. Proposed Model. Experimental Results. Conclusions and Future Work Alaa Tharwat 1,2,5, Aboul Ella Hassanien 3,4,5 Swarm Work Shop - Nov. 7, 15 2 /

3 Introduction In machine learning field, there are two main learning approaches, namely, supervised and unsupervised learning approaches. There are two main techniques of supervised learning, namely, regression and classification. In the unsupervised approach, the targets or responses of the input data are not required to build the model. There are many types of classifiers, but k-nearest Neighbour (k-nn) classifier is one of the oldest and simplest classifier. Alaa Tharwat 1,2,5, Aboul Ella Hassanien 3,4,5 Swarm Work Shop - Nov. 7, 15 3 /

4 Theoretical Background k-nearest Neighbour (k-nn) Classifier k-nearest Neighbour (k-nn) is one of the most common and simple methods for pattern classification. In k-nn classifier, an unknown pattern is distinguished or classified based on the similarity to the known samples (i.e. labelled or training samples) by computing the distances from the unknown sample to all labelled samples and select the k-nearest samples as the basis for classification. The unknown sample is assigned to the class containing the most samples among the k-nearest samples (i.e. voting), thus, the k parameter must be odd. Alaa Tharwat 1,2,5, Aboul Ella Hassanien 3,4,5 Swarm Work Shop - Nov. 7, 15 4 /

5 Theoretical Background Particle Swarm Optimization (PSO) The main objective of the PSO algorithm is to search in the search space for the positions which are close to the global minimum or maximum solution. In PSO algorithm, a number of particles, agents, or elements which represent the solutions are randomly placed in the search space. The number of particles is determined by a user. The current location or position of each particle is used to calculate the objective or fitness function at that location. Each particle has three values, namely, position (x i R n ), velocity (v i ), the previous best positions (p i ), and (G) which represents the position of the best fitness value achieved. Alaa Tharwat 1,2,5, Aboul Ella Hassanien 3,4,5 Swarm Work Shop - Nov. 7, 15 5 /

6 Theoretical Background Particle Swarm Optimization (PSO) v i (t+1) = wvi (t) + C 1r 1 (p i t x i (t) ) + C 2r 2 (G x i (t) ) (1) The velocity of each particle is adjusted in each iteration as shown in Equation (1). The movement of any particle is then calculated by adding the velocity and the current position of that particle as in Equation (2). v i (t+1) = Current Motion + Particle Memory Influnce + Swarm Influnce x i (t+1) = xi (t) + vi (t+1) (2) where w represents the inertia weight, C 1 is the cognition learning factor, C 2 is the social learning factors, r 1, r 2 are the uniformly generated random numbers in the range of [0, 1]. Alaa Tharwat 1,2,5, Aboul Ella Hassanien 3,4,5 Swarm Work Shop - Nov. 7, 15 6 /

7 Theoretical Background Particle Swarm Optimization (PSO) Particle 1 (Current Position) Particle 1 (Next Position) Particle 2 (Current Position) Particle 2 (Next Position) Original Velocity Velocity to Pbest Velocity to G Resultant Velocity x (t+1) i v (t) i v (t+1) i x (t+1) j x (t) i v G i v(t+1) j P (t) j x (t) i x (t) j v p i P (t) i G v G j v p j x (t) j P (t) i x i (t+1) ` x j (t+1) P (t) j (a) v (t) j Figure: An example to show how two particles are move using PSO algorithm, (a) general movement of the two particles, (b) movement of two particle in one-dimensional space. G (b) Alaa Tharwat 1,2,5, Aboul Ella Hassanien 3,4,5 Swarm Work Shop - Nov. 7, 15 7 /

8 Proposed Model: PSOk-NN Particle Swarm Optimization (PSO) IntializeCPSO ForCEachCParticle NextCParticle UpdateCVelocityCdv i V IfCdFdx i V<FdGVV G=x i UpdateCPositionCdx i V IfCdFdx i V<FdP i VV P i =x i EvaluateCFitnessC FunctionCdFdx i VV TraininigC Samples Testing Samples NextC Iteration No SatisfyC TerminationC Criterion 8 7 f G ClassCBC ClassCGC TestingC Yes BestCSloutionCdGV kcparameter 6? Y <? k=b k=< k=? ClassCG ClassCG ClassCB G B MisclassificationCRate B G < Y? f B Figure: PSOk-NN algorithm searches for the optimal k parameter which minimizes the misclassification rate of the testing samples. Alaa Tharwat 1,2,5, Aboul Ella Hassanien 3,4,5 Swarm Work Shop - Nov. 7, 15 8 /

9 Experimental Results Simulated Example Table: Description of the training data used in our simulated example. Class 1 Class 2 Pattern (ω No. 1 ) (ω 2 ) f 1 f 2 f 1 f Alaa Tharwat 1,2,5, Aboul Ella Hassanien 3,4,5 Swarm Work Shop - Nov. 7, 15 9 /

10 Experimental Results Simulated Example f 2 Class 1 (Training Pattern) Class 1 (Testing Pattern) Class 2 (Training Pattern) Class 2 (Testing Pattern) k=1 k=3 k=5 k=7 k= Value of k k=1 k=3 k=5 k=7 k=9 Predicted Class Label C 2 (false) C 2 (false) C 1 (true) C 1 (true) C 2 (false) f 1 Figure: Example of how k parameter controls the predicted class labels of the unknown sample, hence controls the misclassification rate. Alaa Tharwat 1,2,5, Aboul Ella Hassanien 3,4,5 Swarm Work Shop - Nov. 7, /

11 Experimental Results Simulated Example Table: Description of the testing data used in our simulated example and its predicted class labels using k-nn classifier using different values of k. Testing Samples True Class Predicted Class Labels (ŷ i ) No. of Sample f 1 f 2 Label (y i ) k=1 k=3 k=5 k=7 k= Misclassification Rate (%) The bold values indicate the wrong class label. Alaa Tharwat 1,2,5, Aboul Ella Hassanien 3,4,5 Swarm Work Shop - Nov. 7, /

12 Experimental Results Simulated Example Initial Values Particle Position (x No. ) Velocity (v i Fitness ) Function (F) P i G First Iteration G Second Iteration G G G G Alaa Tharwat 1,2,5, Aboul Ella Hassanien 3,4,5 Swarm Work Shop - Nov. 7, /

13 Experimental Results Simulated Example ParticleS1 ParticleS2S ParticleS3S ParticleS4S MisclassificationSRateS(6) FirstSIteration x 1 F(x 1 )=50 v 1 =5.6 x 4 F(x 4 )=25 v 4 =2.8 x 2 F(x 2 )=25 v 2 =-5.6 x 3 F(x 3 )=0 v 3 =0 k=1 k=3 k=5 k=7 k=9 SecondSIteration MisclassificationSRateS(6) k=1 k=3 k=5 k=7 k=9 Figure: Visualization of how PSO algorithm searches for the best k value which achieves the minimum misclassification rate. Alaa Tharwat 1,2,5, Aboul Ella Hassanien 3,4,5 Swarm Work Shop - Nov. 7, /

14 Experimental Results Experiments Using Real Data Table: Data sets description. Data set Dimension Samples Classes Iris Ionosphere Liver-disorders Ovarian Breast Cancer Wine Sonar Pima Indians Diabetes ORL Yale Alaa Tharwat 1,2,5, Aboul Ella Hassanien 3,4,5 Swarm Work Shop - Nov. 7, /

15 Experimental Results Experiments Using Real Data Dataset PSOk-NN GAk-NN ACOk-NN Misclassification Rate Misclassification Rate Misclassification Rate Iris ± ± ±0 Iono ± ± ± Liver ± ± ± Ovarian ± ± ±0 Breast Cancer ±(0.8037) ± ± Wine ± ± ± Sonar 17.45± ± ± Diabate ± ± ± ORL ±0 9.5±0 8.5±0 Yale ± ± ± Alaa Tharwat 1,2,5, Aboul Ella Hassanien 3,4,5 Swarm Work Shop - Nov. 7, /

16 Experimental Results Experiments Using Real Data Iono Dataset Iris Dataset Sonar Dataset Total Absolute Velocity No. of Iterations Figure: Toal absolute velocity of the PSOk-NN algorithm using Iono, Iris, and Sonar datasets. Alaa Tharwat 1,2,5, Aboul Ella Hassanien 3,4,5 Swarm Work Shop - Nov. 7, /

17 Experimental Results Experiments Using Real Data 10 PSO particles 70 PSO particles 2.5 PSO particles Fitness Function Fitness Function Fitness Function k Value (a) After the first iteration k Value (b) After the second iteration k Value (c) After the tenth iteration Figure: Visualization of the movements of all particles of PSOk-NN algorithm till it reaches to the optimal solution which achieved the minimum misclassification rate. Alaa Tharwat 1,2,5, Aboul Ella Hassanien 3,4,5 Swarm Work Shop - Nov. 7, /

18 Experimental Results Experiments Using Real Data setosa versicolor virginica setosa versicolor virginica 1 1 Second Feature Second Feature First Feature First Feature (a) After the first iteration (b) After the tenth iteration Figure: Misclassification samples after the first and tenth iterations using PSOk-NN algorithm. Alaa Tharwat 1,2,5, Aboul Ella Hassanien 3,4,5 Swarm Work Shop - Nov. 7, /

19 Conclusions PSOk-NN algorithm achieved the minimum misclassification error in eight of the datasets (80%) compared with the other two algorithms. PSOk-NN algorithm converges to the optimal solution faster than the other two algorithms due to the use of linearly decreasing inertia weight in PSO algorithm. GAk-NN fluctuating up and down, while PSOk-NN algorithm is more stable during converging to the optimal solution because in PSO, the best solution gives information to all other particles to move to the optimal solution, while in GA the all agents are changed randomly without any guiding from any agent. Alaa Tharwat 1,2,5, Aboul Ella Hassanien 3,4,5 Swarm Work Shop - Nov. 7, /

20 Thank you Thank You Qurstions Alaa Tharwat 1,2,5, Aboul Ella Hassanien 3,4,5 Swarm Work Shop - Nov. 7, 15 /

Introduction to Artificial Intelligence

Introduction to Artificial Intelligence COMP307 Machine Learning 2: 3-K Techniques Yi Mei yi.mei@ecs.vuw.ac.nz 1 Outline K-Nearest Neighbour method Classification (Supervised learning) Basic NN (1-NN)