Classification using Weka (Brain, Computation, and Neural Learning)

Transcription

1 LOGO Classification using Weka (Brain, Computation, and Neural Learning) Jung-Woo Ha

2 Agenda Classification General Concept Terminology Introduction to Weka Classification practice with Weka Problems: Pima Indians diabetes, handwritten digit recognition Algorithms: Neural Networks, Decision Trees, Support Vector Machines Evaluation criteria Using Experimenter for batch experiments Building committee machine Mini-project 2

3 Machine Classification Sorting fish on a conveyor belt: Salmon ( 연어 ) vs. sea bass ( 농어 ) set up a camera, take images and use some physical differences (length, lightness, width, fin shape, mouth position, etc) to explore. 3

4 Concept of Classification <Notations> n = # training examples x = input variables (features or attributes) y = output variable / target variable (x, y) training example The i-th training example = (x (i), y (i) ) Training Set Learning Algorithm Input features h Output / prediction e.g. pixels in a picture of handwritten digit f (x) w hypothesis w x w x n n 3 or 8 4

5 Terminology Features or Attributes Features are the individual measurable properties of the phenomena being observed Choosing discriminating and independent features is key to any pattern recognition algorithm being successful in classification Training set / Test set Training set: A set of examples used for learning, that is to fit the parameters [i.e., weights] of the classifier Test set: A set of examples used only to assess the performance [generalization] of a fully-specified classifier 5

6 Introduction to Weka Weka: Data Mining Software in Java Weka is a collection of machine learning algorithms for data mining & machine learning tasks What you can do with Weka? data pre-processing, feature selection, classification, regression, clustering, association rules, and visualization Weka is an open source software issued under the GNU General Public License How to get? or just type Weka in google. 6

7 Dataset #1: Pima Indians Diabetes Description Pima Indians have the highest prevalence of diabetes in the world We will build classification models that diagnose if the patient shows signs of diabetes Configuration of the data set 768 instances 8 attributes age, number of times pregnant, results of medical tests/analysis all numeric (integer or real-valued) Also, a discretized set will be provided Class value = 1 (Positive example ) Interpreted as "tested positive for diabetes" 500 instances Class value = 0 (Negative example) 268 instances 7

8 Dataset #2: Handwritten Digits (MNIST) Description The MNIST database of handwritten digits contains digits written by office workers and students We will build a recognition model based on classifiers with the reduced set of MNIST Configuration of the data set Attributes pixel values in gray level in a 28x28 image 784 attributes (all 0~255 integer) Full MNIST set Training set: 60,000 examples Test set: 10,000 examples For our practice, a reduced set with 800 examples is used Class value: 0~9, which represent digits from 0 to 9 8

9 Artificial Neural Networks MLP (Multilayer Perceptron) In Weka, Classifiers-functions-MultilayerPerceptron 9

10 Artificial Neural Networks Reviews on BP algorithm The Number of iterations The number of hidden layers and hidden nodes Learning rate Momentum : Four main parameters for learning MLPs 10

11 Reviews on MLPs Expression power of MLPs 11

12 Decision Trees J48 (Java implementation of C4.5) In Weka, classifiers-trees-j48 12

13 Support Vector Machines SMO (sequential minimal optimization) for training SVM In Weka, classifiers-functions-smo 13

14 Practice Basic Comparing the performances of algorithms MultilayerPerceptron vs. J48 vs. SVM Checking the trained model (structure & parameter) Tuning parameters to get better models Understanding Test options & Classifier output in Weka Advanced Building committee machines using meta algorithms for classification Preprocessing / data manipulation applying Filter Batch experiment with Experimenter Design & run a batch process with KnowledgeFlow 14

15 Dataset for Practice with Weka Pima Indians diabetes Original data: pima_diabetes.arff Discretized data: pima_diabetes_supervised_discretized.arff Handwritten Digit (MNIST) Training/test pair mnist_reduced_training.arff, mnist_reduced_test.arff 800 & 200 instances, respectively Total set (1,000 instances) mnist_reduced_total.arff Can be used for cross-validation 15

16 Data format for Weka heart-disease-simplified Header Data (CSV age sex { female, chest_pain_type { typ_angina, asympt, non_anginal, cholesterol exercise_induced_angina { no, class { present, 63,male,typ_angina,233,no,not_present 67,male,asympt,286,yes,present 67,male,asympt,229,yes,present 38,female,non_anginal,?,no,not_present Note: You can easily generate arff file by adding a header to a usual CSV text file 16

17 Neural Networks in Weka click load a file that contains the training data by clicking Open file button ARFF or CSV formats are readible Click MultilayerPerceptron Set parameters for MLP Set parameters for Test Click Start for learning Click Classify tab Click Choose button Select weka function - MultilayerPerceptron 17

18 Some Notes on the Parameter Setting Parameter Setting = Car Tuning need much experience or many times of trial you may get worse results if you are unlucky Multilayer Perceptron (MLP) Main parameters for learning: hiddenlayers, learningrate, momentum, trainingtime (epoch), seed J48 Main parameters: unpruned, numfolds, minnumobj Many parameters are for controlling the size of the result tree, i.e. confidencefactor, pruning SMO (SVM) Main parameters: c (complexity parameter), kernel, kernel parameters 18

19 Test Options and Classifier Output Setting the data set used for evaluation There are various metrics for evaluation 19

20 How to Evaluate the Performance? (1/2) Usually, build a Confusion Matrix out of given data Evaluation Metrics Accuracy (percent correct) Precision Recall Many other metrics: F-measure, Kappa score, etc. For fare evaluation, the cross-validation scheme is used 20

21 How to Evaluate the Performance? (2/2) Confusion Matrix Real Prediction Positive Negative Positive TP FP Negative FN TN All with positive Test All with Negative Test All with Disease All without Disease Everyone Accuracy Precision TP TN TP FP TN FN TP TP FP Recall TP TP FN As recall precision conversely: As recall precision 21

22 Evaluation Method - Cross Validation K-fold Cross Validation The data set is randomly divided into k subsets. One of the k subsets is used as the test set and the other k-1 subsets are put together to form a training set. k 6-fold cross validation Error D 1 D 2 D 3 D 4 D 5 D D 1 D 2 D 3 D 4 D 6 D k i 1 Error i D 2 D 3 D 4 D 5 D 6 D

23 Committee Machine in Weka Using committee machine / ensemble learning in Weka Boosting: AdaBoostM1 Voting committee: Vote Bagging 23

24 Data Manipulation with Filter in Weka Attribute Selection, discretize Instance Re-sampling, selecting specified folds 24

25 Using Experimenter in Weka Tool for Batch experiments Click New click Select Run tab and click Start If it has finished successfully, click Analyse tab and see the summary Set experiment type/iteration control Set datasets / algorithms 25

26 KnowledgeFlow for Analysis Process Design ( Process Flow Diagram of SAS Enterprise Miner ) 26

27 References Weka Wiki: Weka online documentation: Textbooks Tom Mitchell (1997) Machine Learning, McGraw Hill Christopher M. Bishop (2006) Pattern Recognition and Machine Learning, Springer Richard O. Duda, Peter E. Hart, David G. Stork (2001) Pattern classification (2nd edition), Wiley, New York 27

28 Mini-project Make an arff file Make a csv file with MS Excel. Open the csv file with Weka Save the csv file as an arff file Modify the property value of class to discrete value set with any text editor program Save the arff file Reload the arff file with Weka 28

29 Mini-project click load a file that contains the training data by clicking Open file button ARFF or CSV formats are readible Click MultilayerPerceptron Set parameters for MLP Set parameters for Test Click Start for learning Click Classify tab Click Choose button Select weka function - MultilayerPerceptron 29

30 Mini-project Parameter setting of MLPs More explanations on the parameters 30

31 Test Options and Classifier Output Setting the data set used for evaluation There are various metrics for evaluation 31

32 Mini-project Make a MLP by yourself with GUI option You can make the hidden layers by yourself. When clicking more button, you can get details of explanation for GUI. 32

33 Mini-project J48 33

34 Mini-project Experiments Convenient comparisons on data and methods 34

35 Experiments 35

36 Mini-project Classification problem with Weka Data set 3 different data sets You should include at least one set from UCI ML repository and MNIST set ( Classification methods MLP: iters, learning rate, momentum, # of hidden nodes SVM: will be addressed in next time J48: Default options only 36

37 Mini term-project Contents in the report You should compare the results of various parameter settings for MLPs find optimal parameter setting for MLP and report the classification performance on that setting on all data sets Compare the best MLP result to the result of J48 on three data sets (classification and time) Include discussions At most A4 four pages Due date: 24th Nov. 2011( ) 37