How to predict a discrete variable?

Size: px

Start display at page:

Download "How to predict a discrete variable?"

Homer Austin
5 years ago
Views:

1 CSE 6242 / CX 4242 Classificatin Hw t predict a discrete variable? Based n Parishit Ram s slides. Pari nw at SkyTree. Graduated frm PhD frm GT. Als based n Alex Gray s slides.

2 Hw will I rate "Chpin's 5th Symphny"? Sngs Label Sme nights Skyfall Cmfrtably numb We are yung Chpin's 5th???

3 Classificatin What tls d yu need fr classificatin? 1.Data S = {(x i, y i )} i = 1,...,n x i represents each example with d attributes y i represents the label f each example 2. Classificatin mdel f (a,b,c,...) with sme parameters a, b, c,... a mdel/functin maps examples t labels 3.Lss functin L(y, f(x)) hw t penalize mistakes

4 Features Sng name Label Artist Length... Sme nights Fun 4:23... Skyfall Adele 4:00... Cmf. numb Pink Fl. 6:13... We are yung Fun 3: Chpin's 5th?? Chpin 5:32...

5 Training a classifier (building the mdel ) Q: Hw d yu learn apprpriate values fr parameters a, b, c,... such that y i = f (a,b,c,...) (x i ), i = 1,..., n Lw/n errr n training data (sngs) y = f (a,b,c,...) (x), fr any new x Lw/n errr n test data (sngs) Pssible A: Minimize with respect t a, b, c,...

6 Classificatin lss functin Mst cmmn lss: 0-1 lss functin Mre general lss functins are defined by a m x m cst matrix C such that where y = a and f(x) = b Class T0 T1 P0 0 C 10 T0 (true class 0), T1 (true class 1) P1 C 01 0 P0 (predicted class 0), P1 (predicted class 1)

7 k-nearest-neighbr Classifier The classifier: f(x) = majrity label f the k nearest neighbrs (NN) f x Mdel parameters: Number f neighbrs k Distance/similarity functin d(.,.)

8 But KNN is s simple! It can wrk really well! Pandra uses it: (frm the bk Data Mining fr Business Intelligence )

9 k-nearest-neighbr Classifier If k and d(.,.) are fixed Things t learn:? Hw t learn them:? If d(.,.) is fixed, but yu can change k Things t learn:? Hw t learn them:?

10 k-nearest-neighbr Classifier If k and d(.,.) are fixed Things t learn: Nthing Hw t learn them: N/A If d(.,.) is fixed, but yu can change k Selecting k: Try different values f k n sme hld-ut set

12 Hw t find the best k in K-NN? Use crss validatin.

13 Example, evaluate k = 1 (in K-NN) using 5-fld crss-validatin

14 Crss-validatin (C.V.) 1.Divide yur data int n parts 2.Hld 1 part as test set r hld ut set 3.Train classifier n remaining n-1 parts training set 4.Cmpute test errr n test set 5.Repeatabve steps n times, nce fr each n-th part 6.Cmpute the average test errr ver all n flds (i.e., crss-validatin test errr)

15 Crss-validatin variatins Leave-ne-ut crss-validatin (LOO-CV) test sets f size 1 K-fld crss-validatin Test sets f size (n / K) K = 10 is mst cmmn (i.e., 10 fld CV)

16 k-nearest-neighbr Classifier If k is fixed, but yu can change d(.,.) Things t learn:? Hw t learn them:? Crss-validatin:? Pssible distance functins: Euclidean distance: Manhattan distance:

17 k-nearest-neighbr Classifier If k is fixed, but yu can change d(.,.) Things t learn: distance functin d(.,.) Hw t learn them: ptimizatin Crss-validatin: any regularizer yu have n yur distance functin

18 Summary n k-nn classifier Advantages Little learning (unless yu are learning the distance functins) quite pwerful in practice (and has theretical guarantees as well) Caveats Cmputatinally expensive at test time Reading material: ESL bk, Chapter 13.3http://wwwstat.stanfrd.edu/~tibs/ElemStatLearn/printings/ESLII_print10.pdf Le Sng's slides n knn classifierhttp:// f

19 Pints abut crss-validatin Requires extra cmputatin, but gives yu infrmatin abut expected test errr LOO-CV: Advantages Unbiased estimate f test errr (especially fr small n) Lw variance Caveats Extremely time cnsuming

20 Pints abut crss-validatin K-fld CV: Advantages Mre efficient than LOO-CV Caveats K needs t be large fr lw variance T small K leads t under-use f data, leading t higher bias Usually accepted value K = 10 Reading material: ESL bk, Chapter 7.10http://wwwstat.stanfrd.edu/~tibs/ElemStatLearn/printings/ESLII_print10.pdf Le Sng's slides n CV

21 Decisin trees (DT) Weather? The classifier: f T (x) is the majrity class in the leaf in the tree T cntaining x Mdel parameters: The tree structure and size

22 Decisin trees Things t learn:? Hw t learn them:? Crss-validatin:?

23 Decisin trees Things t learn: the tree structure Hw t learn them: (greedily) minimize the verall classificatin lss Crss-validatin: finding the best sized tree with K-fld crss-validatin

24 Learning the tree structure Pieces: 1. best split n the chsen attribute 2. best attribute t split n 3. when t stp splitting 4. crss-validatin

25 Chsing the split Split types fr a selected attribute j: 1. Categrical attribute (e.g. genre ) x 1j = Rck, x 2j = Classical, x 3j = Pp 2. Ordinal attribute (e.g. `achievement') x 1j =Platinum, x 2j =Gld, x 3j =Silver 3. Cntinuus attribute (e.g. sng length) x 1j = 235, x 2j = 543, x 3j = 378 x 1,x 2,x 3 x 1,x 2,x 3 x 1,x 2,x 3 Rck Classical Pp Plat. Gld Silver x 1 x 2 x 3 x 1 x 2 x 3 x 1,x 3 x 2 Split n genre Split n achievement Split n length

26 Chsing the split At a nde T fr a given attribute d, select a split s as fllwing: min s lss(t L ) + lss(t R ) where lss(t) is the lss at nde T Nde lss functins: Ttal lss: Crss-entrpy: where p ct is the prprtin f class c in nde T

27 Chsing the attribute Chice f attribute: 1. Attribute prviding the maximum imprvement in training lss 2. Attribute with maximum infrmatin gain (Recall that entrpy ~= uncertainty)

28 When t stp splitting? 1. Hmgenus nde (all pints in the nde belng t the same class OR all pints in the nde have the same attributes) 2. Nde size less than sme threshld 3. Further splits prvide n imprvement in training lss (lss(t) <= lss(t L ) + lss(t R ))

29 Cntrlling tree size In mst cases, yu can drive training errr t zer (hw? is that gd?) What is wrng with really deep trees? Really high "variance What can be dne t cntrl this? Regularize the tree cmplexity Penalize cmplex mdels and prefers simpler mdels Lk at Le Sng's slides n the decmpsitin f errr in bias and variance f the estimatr

30 Summary n decisin trees Advantages Easy t implement Interpretable Very fast test time Can wrk seamlessly with mixed attributes ** Wrks quite well in practice Caveats Can be t simplistic (but OK if it wrks) Training can be very expensive Crss-validatin is hard (nde-level CV)

31 Final wrds n decisin trees Reading material: ESL bk, Chapter 9.2http://wwwstat.stanfrd.edu/~tibs/ElemStatLearn/printings/ESLII_print10.pdf Le Sng's slideshttp://

How to predict a discrete variable?

How to predict a discrete variable? CSE 6242 / CX 4242 Classificatin Hw t predict a discrete variable? Based n Parishit Ram s slides. Pari nw at SkyTree. Graduated frm PhD frm GT. Als based n Alex Gray s slides. Hw will I rate "Chpin's 5th