Feb 27, 2014 CSE 6242 / CX Classification. How to predict a discrete variable? Based on Parishit Ram s slides

Size: px

Start display at page:

Download "Feb 27, 2014 CSE 6242 / CX Classification. How to predict a discrete variable? Based on Parishit Ram s slides"

Bruce Cunningham
5 years ago
Views:

1 Feb 27, 2014 CSE 6242 / CX 4242 Classificatin Hw t predict a discrete variable? Based n Parishit Ram s slides

2 Hw will I rate "Chpin's 5th Symphny"? Sngs Label Sme nights Skyfall Cmfrtably numb We are yung Chpin's 5th???

3 Classificatin What tls d yu need fr classificatin? 1.Data S = {(x i, y i )} i = 1,...,n x i represents each example with d attributes y i represents the label f each example 2. Classificatin mdel f (a,b,c,...) with sme parameters a, b, c,... a mdel/functin maps examples t labels 3.Lss functin L(y, f(x)) hw t penalize mistakes

4 Features Sng name Label Artist Length... Sme nights Fun 4:23... Skyfall Adele 4:00... Cmf. numb Pink Fl. 6:13... We are yung Fun 3: Chpin's 5th?? Chpin 5:32...

5 Training a classifier (building the mdel ) Q: Hw d yu learn apprpriate values fr parameters a, b, c,... such that (Part I) y i = f (a,b,c,...) (x i ), i = 1,..., n Lw/n errr n the training set (Part II) y = f (a,b,c,...) (x), fr any new x Lw/n errr n future queries (sngs) Pssible A: Minimize with respect t a, b, c,...

6 Classificatin lss functin Mst cmmn lss: 0-1 lss functin Mre general lss functins are defined by a m x m cst matrix C such that where y = a and f(x) = b Class T0 T1 P0 0 C 10 P1 C T0 (true class 0), T1 (true class 1) 01 0 P0 (predicted class 0), P1 (predicted class 1)

7 k-nearest-neighbr Classifier The classifier: f(x) = majrity label f the k nearest neighbrs (NN) f x Mdel parameters: number f neighbrs k distance functin d(.,.)

8 k-nearest-neighbr Classifier If k and d(.,.) are fixed Things t learn:? Hw t learn them:? If d(.,.) is fixed, but yu can change k Things t learn:? Hw t learn them:?

9 k-nearest-neighbr Classifier If k and d(.,.) are fixed Things t learn: Nthing Hw t learn them: N/A If d(.,.) is fixed, but yu can change k Things t learn: Nthing Hw t learn them: N/A Selecting k: Try different values f k n sme hld-ut set

10 Crss-validatin Find the best perfrming k 1. Hld ut a part f the training data 2. Train n the rest f the data 3. Evaluate the training errr n the hldut set 4. D this multiple times, nce fr each k, and pick the k with best perfrmance with respect t the errr (n hld-ut set) averaged ver all hld-ut sets

11 Crss-validatin: Hldut sets Leave-ne-ut crss-validatin (LOO-CV) hld ut sets f size 1 K-fld crss-validatin hld sets f size (n / K) K = 10 is mst cmmn (i.e., 10 fld CV)

12 k-nearest-neighbr Classifier If k is fixed, but yu can change d(.,.) Things t learn:? Hw t learn them:? Crss-validatin:? Pssible distance functins: Euclidean distance: Manhattan distance:

13 k-nearest-neighbr Classifier If k is fixed, but yu can change d(.,.) Things t learn: distance functin d(.,.) Hw t learn them: ptimizatin Crss-validatin: any regularizer yu have n yur distance functin

14 Summary n k-nn classifier Advantages N learning (unless yu are learning the distance functins) quite pwerful in practice (and has theretical guarantees as well) Caveats Cmputatinally expensive at test time Reading material: ESL bk, Chapter Le Sng's slides n knn classifier

15 Pints abut crss-validatin Requires extra cmputatin, but gives yu infrmatin abut expected test errr LOO-CV: Advantages Unbiased estimate f test errr (especially fr small n) Lw variance Caveats Extremely time cnsuming

16 Pints abut crss-validatin K-fld CV: Advantages Mre efficient than LOO-CV Caveats K needs t be large fr lw variance T small K leads t under-use f data, leading t higher bias Usually accepted value K = 10 Reading material: ESL bk, Chapter Le Sng's slides n CV

17 Decisin trees (DT) The classifier: f T (x) is the majrity class in the leaf in the tree T cntaining x Mdel parameters: The tree structure and size

18 Decisin trees Things t learn:? Hw t learn them:? Crss-validatin:?

19 Decisin trees Things t learn: the tree structure Hw t learn them: (greedily) minimize the verall classificatin lss Crss-validatin: finding the best sized tree with K-fld crss-validatin

20 Learning the tree structure Pieces: 1. best split n the chsen attribute 2. best attribute t split n 3. when t stp splitting 4. crss-validatin

21 Chsing the split Split types fr a selected attribute j: 1. Categrical attribute (e.g. `genre') x 1j = Rck, x 2j = Classical, x 3j = Pp 2. Ordinal attribute (e.g. `achievement') x 1j =Gld, x 2j =Platinum, x 3j =Silver 3. Cntinuus attribute (e.g. sng length) x 1j = 235, x 2j = 543, x 3j = 378 x 1,x 2,x 3 x 1,x 2,x 3 x 1,x 2,x 3 Rck Classical Pp Plat. Gld Silver x 1 x 2 x 3 x 1 x 2 x 3 x 1,x 3 x 2 Split n genre Split n achievement Split n length

22 Chsing the split At a nde T fr a given attribute d, select a split s as fllwing: min s lss(t L ) + lss(t R ) where lss(t) is the lss at nde T Nde lss functins: Ttal lss: Crss-entrpy: where p ct is the prprtin f class c in nde T

23 Chsing the attribute Chice f attribute: 1. Attribute prviding the maximum imprvement in training lss 2. Attribute with maximum infrmatin gain

24 When t stp splitting? 1. Hmgenus nde (all pints in the nde belng t the same class OR all pints in the nde have the same attributes) 2. Nde size less than sme threshld 3. Further splits prvide n imprvement in training lss (lss(t) <= lss(t L ) + lss(t R ))

25 Cntrlling tree size In mst cases, yu can drive training errr t zer (hw? is that gd?) What is wrng with really deep trees? Really high "variance What can be dne t cntrl this? Regularize the tree cmplexity Penalize cmplex mdels and prefers simpler mdels (why?) Lk at Le Sng's slides n the decmpsitin f errr in bias and variance f the estimatr

26 Summary n decisin trees Advantages Easy t implement Interpretable Very fast test time Can wrk seamlessly with mixed attributes ** Wrks quite well in practice Caveats Can be t simplistic Training can be very expensive Crss-validatin is hard (and plain annying)

27 Final wrds n decisin trees Reading material: ESL bk, Chapter Le Sng's slides

How to predict a discrete variable?

How to predict a discrete variable? CSE 6242 / CX 4242 Classificatin Hw t predict a discrete variable? Based n Parishit Ram s slides. Pari nw at SkyTree. Graduated frm PhD frm GT. Als based n Alex Gray s slides. Hw will I rate "Chpin's 5th