Designing a learning system

Transcription

1 CS 75 Machie Learig Lecture Desigig a learig system Milos Hauskrecht milos@cs.pitt.edu 539 Seott Square, x-5 people.cs.pitt.edu/~milos/courses/cs75/ Admiistrivia No homework assigmet this week Please try to obtai a copy of Matlab: A brief tutorial o Matlab ext week

2 Learig: first look Assume we get a dataset D that cosists of pairs (x, y) Goal: lear the mappig f : X Y to is able to predict well y for some future x. Questio: How do we lear f? y x Learig: first look. Data: D { d, d,.., d}. Model selectio: Select a model or a set of models (with parameters) E.g. y ax b 3. Choose the objective (error) fuctio Squared error Error( D, ( axi ) i. Learig: Fid the set of parameters ( optimizig the error fuctio a, b ) arg max Error( D, ( ( 5. Applicatio Apply the leared model to ew data E.g. predict ys for the ew iput x f ( x) a x b

3 Learig: first look. Data: D { d, d,.., d}. Model selectio: Select a model or a set of models (with parameters) E.g. y ax y b 3. Choose the objective (error) fuctio Squared error Error( D, ( axi ) i -. Learig: - Fid the set of parameters ( optimizig the error - fuctio - ( a, b ) - arg max ( Error( D, x Applicatio Apply the leared model to ew data E.g. predict ys for the ew iput x f ( x) a x b Learig: first look. Data: D { d, d,.., d}. Model selectio: Select a model or a set of models (with parameters) E.g. y ax b 3. Choose the objective (error) fuctio Squared error y Error( D, ( axi ) i. Learig: Fid the set of parameters ( optimizig the error fuctio a, b ) -arg max Error( D, ( ( - 5. Applicatio - Apply the leared - model to ew data E.g. predict ys for the ew iput x f ( x) a x b

4 Learig: first look. Data: D { d, d,.., d}. Model selectio: Select a model or a set of models (with parameters) E.g. y ax b 3. Choose the objective (error) fuctio Squared error Error( D, ( axi ) i. Learig: Fid the set of parameters ( optimizig the error fuctio a, b ) arg max Error( D, ( ( 5. Applicatio - - Apply the leared model to ew data - - E.g. predict ys for the ew iput x - f ( x) a x b Learig: first look. Data: D { d, d,.., d}. Model selectio: Select a modelor a set of models (with parameters) - - E.g. y ax b Choose the objective (error) fuctio - Squared error Error( D, ( axi ) i. Learig: Fid the set of parameters ( optimizig the error fuctio a, b ) arg max Error( D, ( ( 5. Applicatio Apply the leared model to ew data E.g. predict ys for the ew iput x f ( x) a x b

5 Learig: first look. Data: D { d, d,.., d}. Model selectio: Select a model or a set of models (with parameters) E.g. y ax b 3. Choose the objective (error) fuctio Squared error Error( D, ( axi ) i. Learig: x - - Fid the set of parameters - ( optimizig the error - fuctio a, b ) arg max Error( D,.5 ( ( 5. Applicatio Apply the leared model to ew data E.g. predict ys for the ew iput x f ( x) a x b Learig: first look. Data: D { d, d,.., d}. Model selectio: Select a model or a set of models (with parameters) E.g. y ax b 3. Choose the objective (error) fuctio Squared error Error( D, ( axi ) i. Learig: Fid the set of parameters ( optimizig the error fuctio ( a, b ) arg max ( Error( D, 5. Applicatio * Apply the leared model to ew data f ( x) a x b Looks straightforward, but there are problems. * 5

6 Learig: geeralizatio error We fit the model based o past examples observed i D Traiig data: Data used to fit the parameters of the model Traiig error: Error( D, ( f ( xi )) Problem: Ultimately we are iterested i learig the mappig that performs well o the whole populatio of examples i True (geeralizatio) error (over the whole populatio): Error( E( x, y) [( y f ( x)) ] Mea squared error Traiig error tries to approximate the true error!!!! Does a good traiig error imply a good geeralizatio error? Overfittig Assume we have a set of poits ad we cosider polyomial fuctios as our possible models

7 Overfittig Fittig a liear fuctio with the square error Error is ozero. Why? Overfittig Fittig a liear fuctio with the square error Error is ozero: Error( D, f ) ( f ( xi )) i

8 Overfittig Assume i additio to a liear model: y f ( x) ax b 3 also: y f ( x) a3x ax ax b Which model would give us a smaller error for the least squares fit? Overfittig Liear vs. cubic polyomial Higher order polyomial leads to a better fit, smaller error

9 Overfittig Is it always good to miimize the error of the observed data? Overfittig For data poits, the degree 9 polyomial gives a perfect fit (Lagrage iterpolatio). Error is zero. Is it always good to miimize the traiig error?

10 Overfittig For data poits, degree 9 polyomial gives a perfect fit (Lagrage iterpolatio). Error is zero. Is it always good to miimize the traiig error? NO!! More importat: How do we perform o the usee data? Overfittig Situatio whe the traiig error is low ad the geeralizatio error is high. Causes of the pheomeo: Model with a large umber of parameters (degrees of freedom) Small data size (as compared to the complexity of the model)

11 How to evaluate the learer s performace? Geeralizatio error is the true error for the populatio of examples we would like to optimize E( x, y )[( y f ( x)) ] But it caot be computed exactly Sample mea oly approximates the true mea Optimizig the traiig error ca lead to the overfit, i.e. traiig error may ot reflect properly the geeralizatio error i,.. ( y So how to assess the geeralizatio error? i f ( x )) i How to evaluate the learer s performace? Geeralizatio error is the true error for the populatio of examples we would like to optimize Sample mea oly approximates it Two ways to assess the geeralizatio error is: Theoretical: Law of Large umbers statistical bouds o the differece betwee true ad sample mea errors Practical: Use a separate data set with m data samples to test the model (Average) test error Error( Dtest, f ) ( y j f ( x j )) m j,.. m

12 Assessmet of the geeralizatio performace Simple holdout method Divide the data ito the disjoit traiig ad test data Dataset Traiig set Testig set Evaluate Lear (fit) Predictive model Typically /3 traiig ad /3 testig Testig of models: regressio Data set Traiig set Test set Lear o the traiig set The model Evaluate o the test set

13 Testig of models: classificatio Data set Traiig set Test set case cotrol case cotrol Lear o the traiig set The model Evaluate o the test set Evaluatio measures Easiest way to evaluate the model: Error fuctio used i the optimizatio is adopted also i the evaluatio Advatage: may help us to see model overfittig. Simply compare the error o the traiig ad testig data. Evaluatio of the models ofte cosiders: Other aspects or statistics of the model ad its performace Moreover the Error fuctio used for the optimizatio may be a coveiet approximatio of the quality measure we would really like to optimize 3

14 Evaluatio measures Classificatio: Predictio Case Cotrol Case TP.3 FN. Actual Cotrol FP. TN. Misclassificatio error: E FP FN Sesitivity: TP SN TP FN Specificity: TN SP TN FP A learig system: basic cycle. Data: D { d, d,.., d}. Model selectio: Select a model or a set of models (with parameters) E.g. y ax b 3. Choose the objective fuctio Squared error ( f ( xi )) i. Learig: Fid the set of parameters optimizig the error fuctio The model ad parameters with the smallest error 5. Testig/validatio: Evaluate o the test data. Applicatio Apply the leared model to ew data f (x) CS 75 Machie Learig

15 A learig system: basic cycle. Data: D { d, d,.., d}. Model selectio: Select a y model ax or b a set of models (with parameters) E.g. x 3. Choose the objective fuctio ( f ( xi )) x x i x x - Squared error - x - x. Learig: Fid the set of parameters optimizig the error fuctio The model ad parameters with the smallest error Testig/validatio: Evaluate o the test data. Applicatio Apply the leared model to ew data f (x) A learig system: basic cycle. Data: D { d, d,.., d}. Model selectio: Select a model or a set of models (with parameters) E.g. y ax b 3. Choose the objective fuctio Squared error ( f ( xi )) i. Learig: Fid the set of parameters optimizig the error fuctio The model ad parameters with the smallest error 5. Testig/validatio: Evaluate o the test data. Applicatio Apply the leared model to ew data f (x) CS 75 Machie Learig 5

16 Steps take whe desigig a ML system Data Model selectio Choice of Error fuctio Learig/optimizatio Evaluatio Applicatio Add some complexity Data Data cleaig/preprocessig Feature selectio/dimesioality reductio Model selectio Choice of Error fuctio Learig/optimizatio Evaluatio Applicatio

17 Desigig a ML solutio Data Data cleaig/preprocessig Feature selectio/dimesioality reductio Model selectio Choice of Error fuctio Learig/optimizatio Evaluatio Applicatio Desigig a ML solutio Data Data cleaig/preprocessig Feature selectio/dimesioality reductio Model selectio Choice of Error fuctio Learig/optimizatio Evaluatio Applicatio 7

18 Data source ad data biases Uderstad the data source Uderstad the data your models will be applied to Watch out for data biases: Make sure the data we make coclusios o are the same as data we used i the aalysis It is very easy to derive uexpected results whe data used for aalysis ad learig are biased Results (coclusios) derived for a biased dataset do ot hold i geeral!!! CS 75 Machie Learig Data biases Example: Assume you wat to build a ML program for predictig the stock behavior ad for choosig your ivestmet strategy Data extractio: pick compaies that are traded o the stock market o Jauary 7 Go back 3 years ad extract all the data for these compaies Use the data to build a ML model supportig your future ivestmets Questio: Would you trust the model? Are there ay biases i the data? CS 75 Machie Learig

19 Steps take whe desigig a ML system Data Data cleaig/preprocessig Feature selectio/dimesioality reductio Model selectio Choice of Error fuctio Learig/optimizatio Evaluatio Applicatio Data cleaig ad preprocessig Data you receive may ot be perfect: Cleaig Preprocessig (coversios) Cleaig: Get rid of errors, oise, Removal of redudacies Preprocessig: Reamig Rescalig (ormalizatio) Discretizatios Abstractio Aggregatio New attributes CS 75 Machie Learig 9

20 Data preprocessig Reamig (relabelig) categorical values to umbers dagerous i cojuctio with some learig methods umbers will impose a order that is ot warrated High Normal Low True False Ukow Red Blue Gree Problem: How to safely represet the differet categories as umbers whe o order exists? Solutio: Use idicator vector (or oe-hot) represetatio. Example: Red, Blue, Gree colors 3 categories use a vector of biary (,) values of size 3 Ecodig: Red: (,,); Blue: (,,); ad Gree: (,,) Data preprocessig Rescalig (ormalizatio): cotiuous values trasformed to some rage, typically [-, ] or [,]. 5 - Why ormalizatio? Some learig algorithms are sesitive to the values recorded i the specific iput field ad its magitude

21 Data preprocessig Discretizatio (biig): cotiuous values to a fiite set of discrete values Example: Group Group Group 3 Example : Data preprocessig Abstractio: merge together categorical values Aggregatio: summary or aggregatio operatios, such miimum value, maximum value, average etc. New attributes: example: obesity-factor = weight/height