Discriminant analysis in R QMMA

Size: px
Start display at page:

Download "Discriminant analysis in R QMMA"

Transcription

1 Discriminant analysis in R QMMA Emanuele Taufer file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 1/26

2 Default data Get the data set Default library(islr) data(default) head(default) default student balance income 1 No No No Yes No No No No No No No Yes file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 2/26

3 LDA with R The lda() function, present in the MASS library, allows to face classification problems with LDA. The syntax is similar to the one used in the lm and glm functions library(mass) lda.fit=lda(default~balance+income+student,data=default) file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 3/26

4 Output lda.fit Call: lda(default ~ balance + income + student, data = Default) Prior probabilities of groups: No Yes Group means: balance income studentyes No Yes Coefficients of linear discriminants: LD1 balance e-03 income e-06 studentyes e-01 file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 4/26

5 Output analysis Prior probabilities of groups: estimates of probabilities π k, k = 1, 2 (the relative frequencies of the two groups in the data) Group means: estimates of the conditional averages x y: For example, those who did not go to default have an average balance of (dollars), while those who defaulted have balance average of Note also that because student is a qualitative variable, among those who went in default, the percentage of students is approximately 38%. Coefficients of linear discriminant: the linear combination (LD1) used as discriminant. The coefficients give us the importance of the predictors in the classification. balance has a positive effect on LD1 and higher than income. Being a student decreases the value of LD1. Compare the coefficients of LD1 with those obtained in the logistic regression. file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 5/26

6 file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 6/26

7 Plot The distribution of LD1 for the two classification groups: high values of LD1 give greater probability of default. plot(lda.fit) file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 7/26

8 Prediction The classification of training (or test) units can be done with the predict() function The output of predict() contains a series of objects: we use the names() function to see what they are and, in order to analyze and use them, we put everything in a data.frame. lda.pred=predict(lda.fit, Default) names(lda.pred) [1] "class" "posterior" "x" previsione<-as.data.frame(lda.pred) head(previsione) class posterior.no posterior.yes LD1 1 No No No No No No file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 8/26

9 Classification table Let s build a data.frame, Default.LDA, which contains training data and LDA prediction. We also build a classification table Default.LDA<-cbind(Default, "pred.def"=lda.pred$class, "pr.def"=round(lda.pred$pos head(default.lda) default student balance income pred.def pr.def.no pr.def.yes 1 No No No No Yes No No No No No No No No No No No Yes No attach(default.lda) addmargins(table(default,pred.def)) pred.def default No Yes Sum No Yes Sum file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 9/26

10 Change the classification criterion Let s build a new variable, lda.pred.2, that classifies units as default = Yes if the default probability estimated by the LDA model is greater than 0.2 # Build a vector of "No" lda.pred.2=rep("no",nrow(default)) # Change to "Yes" the elements of that vector for which P(Yes)>0.2 lda.pred.2[default.lda$pr.def.yes>0.2]="yes" # Build the classification table addmargins(table(default,lda.pred.2)) lda.pred.2 default No Yes Sum No Yes Sum file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 10/26

11 Build the ROC curve To create the ROC curve it is necessary to install the ROCR library. Steps: 1. Given a classification rule, start by creating a prediction object. This function is used to transform input data. prediction.obj = prediction (predictions, labels) predictions: the expected probability of a unit being positive labels: the classification observed on the data (test or training). It should be provided as factor, the lower level corresponding to the negative class, the upper level the positive class. file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 11/26

12 2. calculate a performance indicator perf.obj=performance(prediction.obj, measure, measure.x) measure: performance measure to use: `fpr` - False positive rate `tpr` - True positive rate `auc` - area under the ROC curve A complete list of performance measures can be found in the R help of the ROCR package measure.x: a second measure of performance. 3. To get the ROC curve plot a perf.obj with measure ="tpr" and measure.x ="fpr" file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 12/26

13 Example with Default data library(rocr) pred <- prediction(default.lda$pr.def.yes,default$default) perf <- performance(pred, measure = "tpr", x.measure = "fpr") plot(perf, colorize=t,lwd=3) file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 13/26

14 AUC In performance() use measure="auc" with the option fpr.stop=0.5 to get the area above the diagonal. AUC is perf.2<-performance(pred,measure="auc",fpr.stop=0.5) ## [1] file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 14/26

15 LOOCV Cross-validation The CV = TRUE option in the lda() function produces a LOOCV on the data. In this case the output contains the classification produced and the a posteriori probabilities (for each unit excluded from the data set). To see only a part of the output, it is appropriate to put the results in a data.frame lda.fit=lda(default~balance+income+student,data=default,cv=t) names(lda.fit) [1] "class" "posterior" "terms" "call" "xlevels" lda.loocv<-data.frame(lda.fit$class,lda.fit$posterior) head(lda.loocv) lda.fit.class No Yes 1 No No No No No No file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 15/26

16 Estimate of the test error rate with LOOCV We calculate a classification table (default = Yes if the probability of default estimated by the LDA model is greater than 0.2) based on the results of the LOOCV lda.cv.pred.2=rep("no",nrow(default)) lda.cv.pred.2[lda.loocv$yes>0.2]="yes" addmargins(table(default$default,lda.cv.pred.2)) lda.cv.pred.2 No Yes Sum No Yes Sum file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 16/26

17 QDA Quadratic discriminant analysis can be performed using the function qda() qda.fit<-qda(default~balance+income+student,data=default) qda.fit Call: qda(default ~ balance + income + student, data = Default) Prior probabilities of groups: No Yes Group means: balance income studentyes No Yes file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 17/26

18 LOOCV for QDA Similarly to what we have seen for the LDA, a LOOCV can be performed simply by inserting the CV = TRUE option qda.fit<-qda(default~balance+income+student,data=default,cv=true) names(qda.fit) [1] "class" "posterior" "terms" "call" "xlevels" qda.loocv<-data.frame(qda.fit$class,qda.fit$posterior) head(qda.loocv) qda.fit.class No Yes 1 No No No No No No qda.cv.pred.2=rep("no",nrow(default)) qda.cv.pred.2[qda.loocv$yes>0.2]="yes" addmargins(table(default$default,qda.cv.pred.2)) qda.cv.pred.2 No Yes Sum No Yes Sum file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 18/26

19 LDA - QDA comparison Using the results obtained, we compare LDA and QDA by comparing the estimates of the error rate test obtained by LOOCV with classification rule: default = Yes if the probability of default estimated by the model of LDA (or QDA) is greater than 0.2 LDA QDA Sensitivity is 57.96% - ( 193/333) Specificity is 97.6% - ( 9435/9667) Sensitivity is 63.66% - ( 212/333) Specificity is 96.58% - ( 9336/9667) file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 19/26

20 Compare LDA - QDA with the ROC curve library(rocr) lda.pred <- prediction(lda.loocv$yes, Default$default) lda.perf <- performance(lda.pred, measure = "tpr", x.measure = "fpr") qda.pred <- prediction(qda.loocv$yes, Default$default) qda.perf <- performance(qda.pred, measure = "tpr", x.measure = "fpr") plot(lda.perf, col="blue",lwd=4) plot(qda.perf, col="red",lwd=4,add=true) legend(0.6,0.6,c('lda','qda'),col=c('blue','red'),lwd=3) file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 20/26

21 Comparison with logistic regression The logistic regression model: glm.fit<-glm(default~., data=default, family=binomial) summary(glm.fit) ## ## Call: ## glm(formula = default ~., family = binomial, data = Default) ## ## Deviance Residuals: ## Min 1Q Median 3Q Max ## ## ## Coefficients: ## Estimate Std. Error z value Pr(> z ) ## (Intercept) e e < 2e-16 *** ## studentyes e e ** ## balance 5.737e e < 2e-16 *** ## income 3.033e e ## --- ## Signif. codes: 0 '***' '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## (Dispersion parameter for binomial family taken to be 1) ## ## Null deviance: on 9999 degrees of freedom ## Residual deviance: on 9996 degrees of freedom ## AIC: ## ## Number of Fisher Scoring iterations: 8 file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 21/26

22 Cross-validation for logistic regression We cross-validate the LGR model and compare it to LDA and QDA using the ROC curves The cv.glm() function of the boot library (seen in previous exercises), directly provides an estimate of the test error using LOOCV or k-fold CV. To calculate the ROC curve we need the probabilities of prediction. The code below calculates the probabilities of default using a LOOCV. The probabilities are inserted in the glm.loocv vector. glm.loocv=vector() #for(i in 1:nrow(Default)){ for(i in 1:10000){ T.Def=Default[-i,] T.glm.fit=glm(default~.,data=T.Def,family=binomial) glm.loocv[i]=predict(t.glm.fit,default[i,],type="response") } To build the ROC curve use library(rocr) glm.pred <- prediction(glm.loocv, Default$default) glm.perf <- performance(glm.pred, measure = "tpr", x.measure = "fpr") file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 22/26

23 file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 23/26

24 Compare LDA - QDA - LGR with ROCs lda.pred <- prediction(lda.loocv$yes, Default$default) lda.perf <- performance(lda.pred, measure = "tpr", x.measure = "fpr") qda.pred <- prediction(qda.loocv$yes, Default$default) qda.perf <- performance(qda.pred, measure = "tpr", x.measure = "fpr") plot(lda.perf, col="blue",lwd=4) plot(qda.perf, col="red",lwd=4,add=true) plot(glm.perf, col="green",lwd=4,add=true) legend(0.6,0.6,c('lda','qda','lgr'),col=c('blue','red','green'),lwd=3) file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 24/26

25 AUC The AUC for the three models, calculated on the cross-validated data with LOOCV, are: LDA: QDA: RLG: In this case, three models are equivalent. The choice of which model to use can be based on purely personal considerations file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 25/26

26 References An Introduction to Statistical Learning, with applications in R. (Springer, 2013) Tobias Sing, Oliver Sander, Niko Beerenwinkel, Thomas Lengauer. ROCR: visualizing classifier performance in R. Bioinformatics 21(20): (2005). file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l4-lda-eng.html#(1) 26/26

Stat 4510/7510 Homework 4

Stat 4510/7510 Homework 4 Stat 45/75 1/7. Stat 45/75 Homework 4 Instructions: Please list your name and student number clearly. In order to receive credit for a problem, your solution must show sufficient details so that the grader

More information

Chapitre 2 : modèle linéaire généralisé

Chapitre 2 : modèle linéaire généralisé Chapitre 2 : modèle linéaire généralisé Introduction et jeux de données Avant de commencer Faire pointer R vers votre répertoire setwd("~/dropbox/evry/m1geniomhe/cours/") source(file = "fonction_illustration_logistique.r")

More information

STAT Statistical Learning. Predictive Modeling. Statistical Learning. Overview. Predictive Modeling. Classification Methods.

STAT Statistical Learning. Predictive Modeling. Statistical Learning. Overview. Predictive Modeling. Classification Methods. STAT 48 - STAT 48 - December 5, 27 STAT 48 - STAT 48 - Here are a few questions to consider: What does statistical learning mean to you? Is statistical learning different from statistics as a whole? What

More information

Binary Regression in S-Plus

Binary Regression in S-Plus Fall 200 STA 216 September 7, 2000 1 Getting Started in UNIX Binary Regression in S-Plus Create a class working directory and.data directory for S-Plus 5.0. If you have used Splus 3.x before, then it is

More information

STENO Introductory R-Workshop: Loading a Data Set Tommi Suvitaival, Steno Diabetes Center June 11, 2015

STENO Introductory R-Workshop: Loading a Data Set Tommi Suvitaival, Steno Diabetes Center June 11, 2015 STENO Introductory R-Workshop: Loading a Data Set Tommi Suvitaival, tsvv@steno.dk, Steno Diabetes Center June 11, 2015 Contents 1 Introduction 1 2 Recap: Variables 2 3 Data Containers 2 3.1 Vectors................................................

More information

Unit 5 Logistic Regression Practice Problems

Unit 5 Logistic Regression Practice Problems Unit 5 Logistic Regression Practice Problems SOLUTIONS R Users Source: Afifi A., Clark VA and May S. Computer Aided Multivariate Analysis, Fourth Edition. Boca Raton: Chapman and Hall, 2004. Exercises

More information

k-nn classification with R QMMA

k-nn classification with R QMMA k-nn classification with R QMMA Emanuele Taufer file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l1-knn-eng.html#(1) 1/16 HW (Height and weight) of adults Statistics

More information

ISyE 6416 Basic Statistical Methods - Spring 2016 Bonus Project: Big Data Analytics Final Report. Team Member Names: Xi Yang, Yi Wen, Xue Zhang

ISyE 6416 Basic Statistical Methods - Spring 2016 Bonus Project: Big Data Analytics Final Report. Team Member Names: Xi Yang, Yi Wen, Xue Zhang ISyE 6416 Basic Statistical Methods - Spring 2016 Bonus Project: Big Data Analytics Final Report Team Member Names: Xi Yang, Yi Wen, Xue Zhang Project Title: Improve Room Utilization Introduction Problem

More information

Statistical Consulting Topics Using cross-validation for model selection. Cross-validation is a technique that can be used for model evaluation.

Statistical Consulting Topics Using cross-validation for model selection. Cross-validation is a technique that can be used for model evaluation. Statistical Consulting Topics Using cross-validation for model selection Cross-validation is a technique that can be used for model evaluation. We often fit a model to a full data set and then perform

More information

URLs identification task: Istat current status. Istat developed and applied a procedure consisting of the following steps:

URLs identification task: Istat current status. Istat developed and applied a procedure consisting of the following steps: ESSnet BIG DATA WorkPackage 2 URLs identification task: Istat current status Giulio Barcaroli, Monica Scannapieco, Donato Summa Istat developed and applied a procedure consisting of the following steps:

More information

Lab #13 - Resampling Methods Econ 224 October 23rd, 2018

Lab #13 - Resampling Methods Econ 224 October 23rd, 2018 Lab #13 - Resampling Methods Econ 224 October 23rd, 2018 Introduction In this lab you will work through Section 5.3 of ISL and record your code and results in an RMarkdown document. I have added section

More information

Cross-Validation Alan Arnholt 3/22/2016

Cross-Validation Alan Arnholt 3/22/2016 Cross-Validation Alan Arnholt 3/22/2016 Note: Working definitions and graphs are taken from Ugarte, Militino, and Arnholt (2016) The Validation Set Approach The basic idea behind the validation set approach

More information

This is called a linear basis expansion, and h m is the mth basis function For example if X is one-dimensional: f (X) = β 0 + β 1 X + β 2 X 2, or

This is called a linear basis expansion, and h m is the mth basis function For example if X is one-dimensional: f (X) = β 0 + β 1 X + β 2 X 2, or STA 450/4000 S: February 2 2005 Flexible modelling using basis expansions (Chapter 5) Linear regression: y = Xβ + ɛ, ɛ (0, σ 2 ) Smooth regression: y = f (X) + ɛ: f (X) = E(Y X) to be specified Flexible

More information

Lecture 25: Review I

Lecture 25: Review I Lecture 25: Review I Reading: Up to chapter 5 in ISLR. STATS 202: Data mining and analysis Jonathan Taylor 1 / 18 Unsupervised learning In unsupervised learning, all the variables are on equal standing,

More information

Evaluating Classifiers

Evaluating Classifiers Evaluating Classifiers Reading for this topic: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website) Evaluating Classifiers What we want: Classifier that best predicts

More information

Generalized Additive Models

Generalized Additive Models Generalized Additive Models Statistics 135 Autumn 2005 Copyright c 2005 by Mark E. Irwin Generalized Additive Models GAMs are one approach to non-parametric regression in the multiple predictor setting.

More information

Tutorials Case studies

Tutorials Case studies 1. Subject Three curves for the evaluation of supervised learning methods. Evaluation of classifiers is an important step of the supervised learning process. We want to measure the performance of the classifier.

More information

Multinomial Logit Models with R

Multinomial Logit Models with R Multinomial Logit Models with R > rm(list=ls()); options(scipen=999) # To avoid scientific notation > # install.packages("mlogit", dependencies=true) # Only need to do this once > library(mlogit) # Load

More information

List of Exercises: Data Mining 1 December 12th, 2015

List of Exercises: Data Mining 1 December 12th, 2015 List of Exercises: Data Mining 1 December 12th, 2015 1. We trained a model on a two-class balanced dataset using five-fold cross validation. One person calculated the performance of the classifier by measuring

More information

Nina Zumel and John Mount Win-Vector LLC

Nina Zumel and John Mount Win-Vector LLC SUPERVISED LEARNING IN R: REGRESSION Logistic regression to predict probabilities Nina Zumel and John Mount Win-Vector LLC Predicting Probabilities Predicting whether an event occurs (yes/no): classification

More information

Poisson Regression and Model Checking

Poisson Regression and Model Checking Poisson Regression and Model Checking Readings GH Chapter 6-8 September 27, 2017 HIV & Risk Behaviour Study The variables couples and women_alone code the intervention: control - no counselling (both 0)

More information

Discussion Notes 3 Stepwise Regression and Model Selection

Discussion Notes 3 Stepwise Regression and Model Selection Discussion Notes 3 Stepwise Regression and Model Selection Stepwise Regression There are many different commands for doing stepwise regression. Here we introduce the command step. There are many arguments

More information

Predictive Checking. Readings GH Chapter 6-8. February 8, 2017

Predictive Checking. Readings GH Chapter 6-8. February 8, 2017 Predictive Checking Readings GH Chapter 6-8 February 8, 2017 Model Choice and Model Checking 2 Questions: 1. Is my Model good enough? (no alternative models in mind) 2. Which Model is best? (comparison

More information

Logistic Regression. (Dichotomous predicted variable) Tim Frasier

Logistic Regression. (Dichotomous predicted variable) Tim Frasier Logistic Regression (Dichotomous predicted variable) Tim Frasier Copyright Tim Frasier This work is licensed under the Creative Commons Attribution 4.0 International license. Click here for more information.

More information

Dynamic Network Regression Using R Package dnr

Dynamic Network Regression Using R Package dnr Dynamic Network Regression Using R Package dnr Abhirup Mallik July 26, 2018 R package dnr enables the user to fit dynamic network regression models for time variate network data available mostly in social

More information

Evaluating Classifiers

Evaluating Classifiers Evaluating Classifiers Reading for this topic: T. Fawcett, An introduction to ROC analysis, Sections 1-4, 7 (linked from class website) Evaluating Classifiers What we want: Classifier that best predicts

More information

Machine Learning: Practice Midterm, Spring 2018

Machine Learning: Practice Midterm, Spring 2018 Machine Learning: Practice Midterm, Spring 2018 Name: Instructions You may use the following resources on the midterm: 1. your filled in "methods" table with the accompanying notation page, 2. a single

More information

Package gbts. February 27, 2017

Package gbts. February 27, 2017 Type Package Package gbts February 27, 2017 Title Hyperparameter Search for Gradient Boosted Trees Version 1.2.0 Date 2017-02-26 Author Waley W. J. Liang Maintainer Waley W. J. Liang

More information

Evaluation Measures. Sebastian Pölsterl. April 28, Computer Aided Medical Procedures Technische Universität München

Evaluation Measures. Sebastian Pölsterl. April 28, Computer Aided Medical Procedures Technische Universität München Evaluation Measures Sebastian Pölsterl Computer Aided Medical Procedures Technische Universität München April 28, 2015 Outline 1 Classification 1. Confusion Matrix 2. Receiver operating characteristics

More information

MACHINE LEARNING TOOLBOX. Logistic regression on Sonar

MACHINE LEARNING TOOLBOX. Logistic regression on Sonar MACHINE LEARNING TOOLBOX Logistic regression on Sonar Classification models Categorical (i.e. qualitative) target variable Example: will a loan default? Still a form of supervised learning Use a train/test

More information

Orange Juice data. Emanuele Taufer. 4/12/2018 Orange Juice data (1)

Orange Juice data. Emanuele Taufer. 4/12/2018 Orange Juice data (1) Orange Juice data Emanuele Taufer file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l10-oj-data.html#(1) 1/31 Orange Juice Data The data contain weekly sales of refrigerated

More information

Classification. Data Set Iris. Logistic Regression. Species. Petal.Width

Classification. Data Set Iris. Logistic Regression. Species. Petal.Width Classification Data Set Iris # load data data(iris) # this is what it looks like... head(iris) Sepal.Length Sepal.Width 1 5.1 3.5 1.4 0.2 2 4.9 3.0 1.4 0.2 3 4.7 3.2 1.3 0.2 4 4.6 3.1 0.2 5 5.0 3.6 1.4

More information

run ld50 /* Plot the onserved proportions and the fitted curve */ DATA SETR1 SET SETR1 PROB=X1/(X1+X2) /* Use this to create graphs in Windows */ gopt

run ld50 /* Plot the onserved proportions and the fitted curve */ DATA SETR1 SET SETR1 PROB=X1/(X1+X2) /* Use this to create graphs in Windows */ gopt /* This program is stored as bliss.sas */ /* This program uses PROC LOGISTIC in SAS to fit models with logistic, probit, and complimentary log-log link functions to the beetle mortality data collected

More information

CS145: INTRODUCTION TO DATA MINING

CS145: INTRODUCTION TO DATA MINING CS145: INTRODUCTION TO DATA MINING 08: Classification Evaluation and Practical Issues Instructor: Yizhou Sun yzsun@cs.ucla.edu October 24, 2017 Learnt Prediction and Classification Methods Vector Data

More information

1 The SAS System 23:01 Friday, November 9, 2012

1 The SAS System 23:01 Friday, November 9, 2012 2101f12HW9chickwts.log Saved: Wednesday, November 14, 2012 6:50:49 PM Page 1 of 3 1 The SAS System 23:01 Friday, November 9, 2012 NOTE: Copyright (c) 2002-2010 by SAS Institute Inc., Cary, NC, USA. NOTE:

More information

SYS 6021 Linear Statistical Models

SYS 6021 Linear Statistical Models SYS 6021 Linear Statistical Models Project 2 Spam Filters Jinghe Zhang Summary The spambase data and time indexed counts of spams and hams are studied to develop accurate spam filters. Static models are

More information

Package MMIX. R topics documented: February 15, Type Package. Title Model selection uncertainty and model mixing. Version 1.2.

Package MMIX. R topics documented: February 15, Type Package. Title Model selection uncertainty and model mixing. Version 1.2. Package MMIX February 15, 2013 Type Package Title Model selection uncertainty and model mixing Version 1.2 Date 2012-06-18 Author Marie Morfin and David Makowski Maintainer Description

More information

Chapter 10: Extensions to the GLM

Chapter 10: Extensions to the GLM Chapter 10: Extensions to the GLM 10.1 Implement a GAM for the Swedish mortality data, for males, using smooth functions for age and year. Age and year are standardized as described in Section 4.11, for

More information

Classification Algorithms in Data Mining

Classification Algorithms in Data Mining August 9th, 2016 Suhas Mallesh Yash Thakkar Ashok Choudhary CIS660 Data Mining and Big Data Processing -Dr. Sunnie S. Chung Classification Algorithms in Data Mining Deciding on the classification algorithms

More information

22s:152 Applied Linear Regression

22s:152 Applied Linear Regression 22s:152 Applied Linear Regression Chapter 22: Model Selection In model selection, the idea is to find the smallest set of variables which provides an adequate description of the data. We will consider

More information

Linear Model Selection and Regularization. especially usefull in high dimensions p>>100.

Linear Model Selection and Regularization. especially usefull in high dimensions p>>100. Linear Model Selection and Regularization especially usefull in high dimensions p>>100. 1 Why Linear Model Regularization? Linear models are simple, BUT consider p>>n, we have more features than data records

More information

Data Mining and Knowledge Discovery: Practice Notes

Data Mining and Knowledge Discovery: Practice Notes Data Mining and Knowledge Discovery: Practice Notes Petra Kralj Novak Petra.Kralj.Novak@ijs.si 2016/11/16 1 Keywords Data Attribute, example, attribute-value data, target variable, class, discretization

More information

Weighted Sample. Weighted Sample. Weighted Sample. Training Sample

Weighted Sample. Weighted Sample. Weighted Sample. Training Sample Final Classifier [ M ] G(x) = sign m=1 α mg m (x) Weighted Sample G M (x) Weighted Sample G 3 (x) Weighted Sample G 2 (x) Training Sample G 1 (x) FIGURE 10.1. Schematic of AdaBoost. Classifiers are trained

More information

CH9.Generalized Additive Model

CH9.Generalized Additive Model CH9.Generalized Additive Model Regression Model For a response variable and predictor variables can be modeled using a mean function as follows: would be a parametric / nonparametric regression or a smoothing

More information

Model Assessment and Selection. Reference: The Elements of Statistical Learning, by T. Hastie, R. Tibshirani, J. Friedman, Springer

Model Assessment and Selection. Reference: The Elements of Statistical Learning, by T. Hastie, R. Tibshirani, J. Friedman, Springer Model Assessment and Selection Reference: The Elements of Statistical Learning, by T. Hastie, R. Tibshirani, J. Friedman, Springer 1 Model Training data Testing data Model Testing error rate Training error

More information

Section 2.1: Intro to Simple Linear Regression & Least Squares

Section 2.1: Intro to Simple Linear Regression & Least Squares Section 2.1: Intro to Simple Linear Regression & Least Squares Jared S. Murray The University of Texas at Austin McCombs School of Business Suggested reading: OpenIntro Statistics, Chapter 7.1, 7.2 1 Regression:

More information

DATA MINING AND MACHINE LEARNING. Lecture 6: Data preprocessing and model selection Lecturer: Simone Scardapane

DATA MINING AND MACHINE LEARNING. Lecture 6: Data preprocessing and model selection Lecturer: Simone Scardapane DATA MINING AND MACHINE LEARNING Lecture 6: Data preprocessing and model selection Lecturer: Simone Scardapane Academic Year 2016/2017 Table of contents Data preprocessing Feature normalization Missing

More information

Stat 342 Exam 3 Fall 2014

Stat 342 Exam 3 Fall 2014 Stat 34 Exam 3 Fall 04 I have neither given nor received unauthorized assistance on this exam. Name Signed Date Name Printed There are questions on the following 6 pages. Do as many of them as you can

More information

Data Mining and Knowledge Discovery Practice notes 2

Data Mining and Knowledge Discovery Practice notes 2 Keywords Data Mining and Knowledge Discovery: Practice Notes Petra Kralj Novak Petra.Kralj.Novak@ijs.si Data Attribute, example, attribute-value data, target variable, class, discretization Algorithms

More information

CS249: ADVANCED DATA MINING

CS249: ADVANCED DATA MINING CS249: ADVANCED DATA MINING Classification Evaluation and Practical Issues Instructor: Yizhou Sun yzsun@cs.ucla.edu April 24, 2017 Homework 2 out Announcements Due May 3 rd (11:59pm) Course project proposal

More information

Using R for Analyzing Delay Discounting Choice Data. analysis of discounting choice data requires the use of tools that allow for repeated measures

Using R for Analyzing Delay Discounting Choice Data. analysis of discounting choice data requires the use of tools that allow for repeated measures Using R for Analyzing Delay Discounting Choice Data Logistic regression is available in a wide range of statistical software packages, but the analysis of discounting choice data requires the use of tools

More information

Evaluation Metrics. (Classifiers) CS229 Section Anand Avati

Evaluation Metrics. (Classifiers) CS229 Section Anand Avati Evaluation Metrics (Classifiers) CS Section Anand Avati Topics Why? Binary classifiers Metrics Rank view Thresholding Confusion Matrix Point metrics: Accuracy, Precision, Recall / Sensitivity, Specificity,

More information

RENR690-SDM. Zihaohan Sang. March 26, 2018

RENR690-SDM. Zihaohan Sang. March 26, 2018 RENR690-SDM Zihaohan Sang March 26, 2018 Intro This lab will roughly go through the key processes of species distribution modeling (SDM), from data preparation, model fitting, to final evaluation. In this

More information

Classification Part 4

Classification Part 4 Classification Part 4 Dr. Sanjay Ranka Professor Computer and Information Science and Engineering University of Florida, Gainesville Model Evaluation Metrics for Performance Evaluation How to evaluate

More information

Data Mining and Knowledge Discovery: Practice Notes

Data Mining and Knowledge Discovery: Practice Notes Data Mining and Knowledge Discovery: Practice Notes Petra Kralj Novak Petra.Kralj.Novak@ijs.si 8.11.2017 1 Keywords Data Attribute, example, attribute-value data, target variable, class, discretization

More information

CS6375: Machine Learning Gautam Kunapuli. Mid-Term Review

CS6375: Machine Learning Gautam Kunapuli. Mid-Term Review Gautam Kunapuli Machine Learning Data is identically and independently distributed Goal is to learn a function that maps to Data is generated using an unknown function Learn a hypothesis that minimizes

More information

Package rocc. February 20, 2015

Package rocc. February 20, 2015 Package rocc February 20, 2015 Title ROC based classification Version 1.2 Depends ROCR Author Martin Lauss Description Functions for a classification method based on receiver operating characteristics

More information

Gelman-Hill Chapter 3

Gelman-Hill Chapter 3 Gelman-Hill Chapter 3 Linear Regression Basics In linear regression with a single independent variable, as we have seen, the fundamental equation is where ŷ bx 1 b0 b b b y 1 yx, 0 y 1 x x Bivariate Normal

More information

Package parcor. February 20, 2015

Package parcor. February 20, 2015 Type Package Package parcor February 20, 2015 Title Regularized estimation of partial correlation matrices Version 0.2-6 Date 2014-09-04 Depends MASS, glmnet, ppls, Epi, GeneNet Author, Juliane Schaefer

More information

Generalized Additive Model

Generalized Additive Model Generalized Additive Model by Huimin Liu Department of Mathematics and Statistics University of Minnesota Duluth, Duluth, MN 55812 December 2008 Table of Contents Abstract... 2 Chapter 1 Introduction 1.1

More information

22s:152 Applied Linear Regression

22s:152 Applied Linear Regression 22s:152 Applied Linear Regression Chapter 22: Model Selection In model selection, the idea is to find the smallest set of variables which provides an adequate description of the data. We will consider

More information

Performance Estimation and Regularization. Kasthuri Kannan, PhD. Machine Learning, Spring 2018

Performance Estimation and Regularization. Kasthuri Kannan, PhD. Machine Learning, Spring 2018 Performance Estimation and Regularization Kasthuri Kannan, PhD. Machine Learning, Spring 2018 Bias- Variance Tradeoff Fundamental to machine learning approaches Bias- Variance Tradeoff Error due to Bias:

More information

GLM II. Basic Modeling Strategy CAS Ratemaking and Product Management Seminar by Paul Bailey. March 10, 2015

GLM II. Basic Modeling Strategy CAS Ratemaking and Product Management Seminar by Paul Bailey. March 10, 2015 GLM II Basic Modeling Strategy 2015 CAS Ratemaking and Product Management Seminar by Paul Bailey March 10, 2015 Building predictive models is a multi-step process Set project goals and review background

More information

Lecture 24: Generalized Additive Models Stat 704: Data Analysis I, Fall 2010

Lecture 24: Generalized Additive Models Stat 704: Data Analysis I, Fall 2010 Lecture 24: Generalized Additive Models Stat 704: Data Analysis I, Fall 2010 Tim Hanson, Ph.D. University of South Carolina T. Hanson (USC) Stat 704: Data Analysis I, Fall 2010 1 / 26 Additive predictors

More information

Manuel Oviedo de la Fuente and Manuel Febrero Bande

Manuel Oviedo de la Fuente and Manuel Febrero Bande Supervised classification methods in by fda.usc package Manuel Oviedo de la Fuente and Manuel Febrero Bande Universidade de Santiago de Compostela CNTG (Centro de Novas Tecnoloxías de Galicia). Santiago

More information

Product Catalog. AcaStat. Software

Product Catalog. AcaStat. Software Product Catalog AcaStat Software AcaStat AcaStat is an inexpensive and easy-to-use data analysis tool. Easily create data files or import data from spreadsheets or delimited text files. Run crosstabulations,

More information

CS4491/CS 7265 BIG DATA ANALYTICS

CS4491/CS 7265 BIG DATA ANALYTICS CS4491/CS 7265 BIG DATA ANALYTICS EVALUATION * Some contents are adapted from Dr. Hung Huang and Dr. Chengkai Li at UT Arlington Dr. Mingon Kang Computer Science, Kennesaw State University Evaluation for

More information

Linear Methods for Regression and Shrinkage Methods

Linear Methods for Regression and Shrinkage Methods Linear Methods for Regression and Shrinkage Methods Reference: The Elements of Statistical Learning, by T. Hastie, R. Tibshirani, J. Friedman, Springer 1 Linear Regression Models Least Squares Input vectors

More information

2. On classification and related tasks

2. On classification and related tasks 2. On classification and related tasks In this part of the course we take a concise bird s-eye view of different central tasks and concepts involved in machine learning and classification particularly.

More information

Package scorecard. December 17, 2017

Package scorecard. December 17, 2017 Type Package Title Credit Risk Scorecard Package scorecard December 17, 2017 Makes the development of credit risk scorecard easily and efficiently by providing functions such as information value, variable

More information

Data Mining Classification: Alternative Techniques. Imbalanced Class Problem

Data Mining Classification: Alternative Techniques. Imbalanced Class Problem Data Mining Classification: Alternative Techniques Imbalanced Class Problem Introduction to Data Mining, 2 nd Edition by Tan, Steinbach, Karpatne, Kumar Class Imbalance Problem Lots of classification problems

More information

Using the SemiPar Package

Using the SemiPar Package Using the SemiPar Package NICHOLAS J. SALKOWSKI Division of Biostatistics, School of Public Health, University of Minnesota, Minneapolis, MN 55455, USA salk0008@umn.edu May 15, 2008 1 Introduction The

More information

Package EBglmnet. January 30, 2016

Package EBglmnet. January 30, 2016 Type Package Package EBglmnet January 30, 2016 Title Empirical Bayesian Lasso and Elastic Net Methods for Generalized Linear Models Version 4.1 Date 2016-01-15 Author Anhui Huang, Dianting Liu Maintainer

More information

3 Feature Selection & Feature Extraction

3 Feature Selection & Feature Extraction 3 Feature Selection & Feature Extraction Overview: 3.1 Introduction 3.2 Feature Extraction 3.3 Feature Selection 3.3.1 Max-Dependency, Max-Relevance, Min-Redundancy 3.3.2 Relevance Filter 3.3.3 Redundancy

More information

Metrics for Performance Evaluation How to evaluate the performance of a model? Methods for Performance Evaluation How to obtain reliable estimates?

Metrics for Performance Evaluation How to evaluate the performance of a model? Methods for Performance Evaluation How to obtain reliable estimates? Model Evaluation Metrics for Performance Evaluation How to evaluate the performance of a model? Methods for Performance Evaluation How to obtain reliable estimates? Methods for Model Comparison How to

More information

Package optimus. March 24, 2017

Package optimus. March 24, 2017 Type Package Package optimus March 24, 2017 Title Model Based Diagnostics for Multivariate Cluster Analysis Version 0.1.0 Date 2017-03-24 Maintainer Mitchell Lyons Description

More information

Package hmeasure. February 20, 2015

Package hmeasure. February 20, 2015 Type Package Package hmeasure February 20, 2015 Title The H-measure and other scalar classification performance metrics Version 1.0 Date 2012-04-30 Author Christoforos Anagnostopoulos

More information

2017 ITRON EFG Meeting. Abdul Razack. Specialist, Load Forecasting NV Energy

2017 ITRON EFG Meeting. Abdul Razack. Specialist, Load Forecasting NV Energy 2017 ITRON EFG Meeting Abdul Razack Specialist, Load Forecasting NV Energy Topics 1. Concepts 2. Model (Variable) Selection Methods 3. Cross- Validation 4. Cross-Validation: Time Series 5. Example 1 6.

More information

COSC160: Detection and Classification. Jeremy Bolton, PhD Assistant Teaching Professor

COSC160: Detection and Classification. Jeremy Bolton, PhD Assistant Teaching Professor COSC160: Detection and Classification Jeremy Bolton, PhD Assistant Teaching Professor Outline I. Problem I. Strategies II. Features for training III. Using spatial information? IV. Reducing dimensionality

More information

Data Mining: Classifier Evaluation. CSCI-B490 Seminar in Computer Science (Data Mining)

Data Mining: Classifier Evaluation. CSCI-B490 Seminar in Computer Science (Data Mining) Data Mining: Classifier Evaluation CSCI-B490 Seminar in Computer Science (Data Mining) Predictor Evaluation 1. Question: how good is our algorithm? how will we estimate its performance? 2. Question: what

More information

Regression trees with R

Regression trees with R Regression trees with R Emanuele Taufer file:///c:/users/emanuele.taufer/google%20drive/2%20corsi/5%20qmma%20-%20mim/0%20labs/l7-regression-trees.html#(1) 1/17 Implement decision trees with R To adapt

More information

Package oasis. February 21, 2018

Package oasis. February 21, 2018 Type Package Package oasis February 21, 2018 Title Multiple Sclerosis Lesion Segmentation using Magnetic Resonance Imaging (MRI) Version 3.0.4 Date 2018-02-20 Trains and makes predictions from the OASIS

More information

Feature Selection. CE-725: Statistical Pattern Recognition Sharif University of Technology Spring Soleymani

Feature Selection. CE-725: Statistical Pattern Recognition Sharif University of Technology Spring Soleymani Feature Selection CE-725: Statistical Pattern Recognition Sharif University of Technology Spring 2013 Soleymani Outline Dimensionality reduction Feature selection vs. feature extraction Filter univariate

More information

Logistic Regression

Logistic Regression Logistic Regression ddebarr@uw.edu 2016-05-26 Agenda Model Specification Model Fitting Bayesian Logistic Regression Online Learning and Stochastic Optimization Generative versus Discriminative Classifiers

More information

The brlr Package. March 22, brlr... 1 lizards Index 5. Bias-reduced Logistic Regression

The brlr Package. March 22, brlr... 1 lizards Index 5. Bias-reduced Logistic Regression The brlr Package March 22, 2006 Version 0.8-8 Date 2006-03-22 Title Bias-reduced logistic regression Author David Firth URL http://www.warwick.ac.uk/go/dfirth Maintainer David Firth

More information

Cross- Valida+on & ROC curve. Anna Helena Reali Costa PCS 5024

Cross- Valida+on & ROC curve. Anna Helena Reali Costa PCS 5024 Cross- Valida+on & ROC curve Anna Helena Reali Costa PCS 5024 Resampling Methods Involve repeatedly drawing samples from a training set and refibng a model on each sample. Used in model assessment (evalua+ng

More information

Stat 5303 (Oehlert): Response Surfaces 1

Stat 5303 (Oehlert): Response Surfaces 1 Stat 5303 (Oehlert): Response Surfaces 1 > data

More information

Package FWDselect. December 19, 2015

Package FWDselect. December 19, 2015 Title Selecting Variables in Regression Models Version 2.1.0 Date 2015-12-18 Author Marta Sestelo [aut, cre], Nora M. Villanueva [aut], Javier Roca-Pardinas [aut] Maintainer Marta Sestelo

More information

Regression on the trees data with R

Regression on the trees data with R > trees Girth Height Volume 1 8.3 70 10.3 2 8.6 65 10.3 3 8.8 63 10.2 4 10.5 72 16.4 5 10.7 81 18.8 6 10.8 83 19.7 7 11.0 66 15.6 8 11.0 75 18.2 9 11.1 80 22.6 10 11.2 75 19.9 11 11.3 79 24.2 12 11.4 76

More information

Package citools. October 20, 2018

Package citools. October 20, 2018 Type Package Package citools October 20, 2018 Title Confidence or Prediction Intervals, Quantiles, and Probabilities for Statistical Models Version 0.5.0 Maintainer John Haman Functions

More information

As a reference, please find a version of the Machine Learning Process described in the diagram below.

As a reference, please find a version of the Machine Learning Process described in the diagram below. PREDICTION OVERVIEW In this experiment, two of the Project PEACH datasets will be used to predict the reaction of a user to atmospheric factors. This experiment represents the first iteration of the Machine

More information

The ROCR Package. January 30, 2007

The ROCR Package. January 30, 2007 The ROCR Package January 30, 2007 Title Visualizing the performance of scoring classifiers. Version 1.0-2 Date 2007-01-27 Depends gplots Author Tobias Sing, Oliver Sander, Niko Beerenwinkel, Thomas Lengauer

More information

Data Mining and Knowledge Discovery Practice notes: Numeric Prediction, Association Rules

Data Mining and Knowledge Discovery Practice notes: Numeric Prediction, Association Rules Keywords Data Mining and Knowledge Discovery: Practice Notes Petra Kralj Novak Petra.Kralj.Novak@ijs.si Data Attribute, example, attribute-value data, target variable, class, discretization Algorithms

More information

CREDIT RISK MODELING IN R. Finding the right cut-off: the strategy curve

CREDIT RISK MODELING IN R. Finding the right cut-off: the strategy curve CREDIT RISK MODELING IN R Finding the right cut-off: the strategy curve Constructing a confusion matrix > predict(log_reg_model, newdata = test_set, type = "response") 1 2 3 4 5 0.08825517 0.3502768 0.28632298

More information

Statistics & Analysis. Fitting Generalized Additive Models with the GAM Procedure in SAS 9.2

Statistics & Analysis. Fitting Generalized Additive Models with the GAM Procedure in SAS 9.2 Fitting Generalized Additive Models with the GAM Procedure in SAS 9.2 Weijie Cai, SAS Institute Inc., Cary NC July 1, 2008 ABSTRACT Generalized additive models are useful in finding predictor-response

More information

CSSS 510: Lab 2. Introduction to Maximum Likelihood Estimation

CSSS 510: Lab 2. Introduction to Maximum Likelihood Estimation CSSS 510: Lab 2 Introduction to Maximum Likelihood Estimation 2018-10-12 0. Agenda 1. Housekeeping: simcf, tile 2. Questions about Homework 1 or lecture 3. Simulating heteroskedastic normal data 4. Fitting

More information

Package EFS. R topics documented:

Package EFS. R topics documented: Package EFS July 24, 2017 Title Tool for Ensemble Feature Selection Description Provides a function to check the importance of a feature based on a dependent classification variable. An ensemble of feature

More information

Package smbinning. December 1, 2017

Package smbinning. December 1, 2017 Title Scoring Modeling and Optimal Binning Version 0.5 Author Herman Jopia Maintainer Herman Jopia URL http://www.scoringmodeling.com Package smbinning December 1, 2017 A set of functions

More information

MIPE: Model Informing Probability of Eradication of non-indigenous aquatic species. User Manual. Version 2.4

MIPE: Model Informing Probability of Eradication of non-indigenous aquatic species. User Manual. Version 2.4 MIPE: Model Informing Probability of Eradication of non-indigenous aquatic species User Manual Version 2.4 March 2014 1 Table of content Introduction 3 Installation 3 Using MIPE 3 Case study data 3 Input

More information

Package fbroc. August 29, 2016

Package fbroc. August 29, 2016 Type Package Package fbroc August 29, 2016 Title Fast Algorithms to Bootstrap Receiver Operating Characteristics Curves Version 0.4.0 Date 2016-06-21 Implements a very fast C++ algorithm to quickly bootstrap

More information