Designing a learning system

Transcription

1 CS 75 Itro to Machie Learig Lecture Desigig a learig system Milos Hauskrecht milos@pitt.edu 539 Seott Square, -5 people.cs.pitt.edu/~milos/courses/cs75/ Admiistrivia No homework assigmet this week Please try to obtai a copy of Matlab: Net week: Recitatios: Matlab tutorial Tuesday: Review of algebra ad probability

2 Learig: first look Assume we see eamples of pairs (, y) i D ad we wat to lear the mappig f : X Y to predict y for some future We get the data D - what should we do? y Learig: first look Problem: may possible fuctios f : X Y eists for represetig the mappig betwee ad y Which oe to choose? May eamples still usee! y

3 Learig: first look Solutio: make a assumptio about the model, say, f ( ) a b y Learig: first look Choosig a parametric model or a set of models is ot eough Still too may fuctios f ( ) a b Oe for every pair of parameters b y

4 Learig: first look We wat the best set of model parameters reduce the misfit betwee the model M ad observed data D Or, (i other words) eplai the data the best How to measure the misfit? y Learig: first look We wat the best set of model parameters reduce the misfit betwee the model M ad observed data D Or, (i other words) eplai the data the best How to measure the misfit? y - The differece i the observed value of y ad model predictio

5 Learig: first look We wat the best set of model parameters reduce the misfit betwee the model M ad observed data D Or, (i other words) eplai the data the best How to measure the misfit? y Learig: first look We wat the best set of model parameters reduce the misfit betwee the model M ad observed data D Or, (i other words) eplai the data the best How to measure the misfit? Objective fuctio: Error (loss) fuctio: Measures the misfit betwee D ad M Eamples of error fuctios: Average Square Error Average Absolute Error i i ( y y i i f ( )) i i f ( ) 5

6 Learig: first look Liear regressio Miimizes the squared error fuctio for the liear model ( yi f ( i )) i y Learig: first look. Data: D { d, d,.., d}. Model selectio: Select a model or a set of models (with parameters) E.g. y a b 3. Choose the objective (error) fuctio Squared error Error( D, ( yi ai ) i. Learig: Fid the set of parameters ( optimizig the error fuctio * * a, b ) arg ma Error( D, ( ( 5. Applicatio Apply the leared model to ew data E.g. predict ys for the ew iput f * * ( ) a b

7 Learig: first look. Data: D { d, d,.., d}. Model selectio: Select a model or a set of models (with parameters) E.g. y 3. Choose the objective (error) fuctio Squared error Error( D, ( yi ai ) i -. Learig: - Fid the set of parameters ( optimizig the error - fuctio - * * ( a, b ) - arg ma ( Error( D, y a b Applicatio Apply the leared model to ew data E.g. predict ys for the ew iput f * * ( ) a b Learig: first look. Data: D { d, d,.., d}. Model selectio: Select a model or a set of models (with parameters) E.g. y a b 3. Choose the objective (error) fuctio Squared error y Error( D, ( yi ai ) i. Learig: Fid the set of parameters ( optimizig the error fuctio * * a, b ) -arg ma Error( D, ( ( - 5. Applicatio - Apply the leared - model to ew data E.g. predict ys for the ew iput f * * ( ) a b

8 Learig: first look. Data: D { d, d,.., d}. Model selectio: Select a model or a set of models (with parameters) E.g. 3. Choose the objective (error) fuctio Squared error Error( D, ( yi ai ) i. Learig: Fid the set of parameters ( optimizig the error fuctio * * a, b ) arg ma Error( D, y a b ( ( 5. Applicatio - - Apply the leared model to ew data - - E.g. predict ys for the ew iput - f * * ( ) a b Learig: first look. Data: D { d, d,.., d}. Model selectio: Select a modelor a set of models (with parameters) - - E.g. y a b Choose the objective (error) fuctio - Squared error Error( D, ( yi ai ) i. Learig: Fid the set of parameters ( optimizig the error fuctio * * a, b ) arg ma Error( D, ( ( 5. Applicatio Apply the leared model to ew data E.g. predict ys for the ew iput f * * ( ) a b

9 Learig: first look. Data: D { d, d,.., d}. Model selectio: Select a model or a set of models (with parameters) E.g. 3. Choose the objective (error) fuctio Squared error Error( D, ( yi ai ) i. Learig: - - Fid the set of parameters - ( optimizig the error - fuctio * * a, b ) arg ma Error( D,.5 y a b ( ( 5. Applicatio Apply the leared model to ew data E.g. predict ys for the ew iput f * * ( ) a b Learig: first look. Data: D { d, d,.., d}. Model selectio: Select a model or a set of models (with parameters) E.g. y a b 3. Choose the objective (error) fuctio Squared error Error( D, ( yi ai ) i. Learig: Fid the set of parameters ( optimizig the error fuctio * * ( a, b ) arg ma ( Error( D, 5. Applicatio * Apply the leared model to ew data f ( ) a b Looks straightforward, but there are problems. * 9

10 Learig: geeralizatio error We fit the model based o past eamples observed i D Traiig data: Data used to fit the parameters of the model Traiig error: Error( D, ( yi f ( i )) Problem: Ultimately we are iterested i learig the mappig that performs well o the whole populatio of eamples i True (geeralizatio) error (over the whole populatio): Error( E [( y f ( (, y) )) ] Mea squared error Traiig error tries to approimate the true error!!!! Does a good traiig error imply a good geeralizatio error? Traiig vs Geeralizatio error Assume we have a set of poits ad we cosider polyomial fuctios as our possible models

11 Traiig vs Geeralizatio error Fittig a liear fuctio with the square error Error is ozero Traiig vs Geeralizatio error Liear vs. cubic polyomial

12 Traiig vs Geeralizatio error Liear vs. cubic polyomial Higher order polyomial leads to a better fit, smaller error Traiig vs Geeralizatio error Is it always good to miimize the error of the observed data? Remember: our goal is to optimize future errors CS 75 Machie Learig

13 Traiig vs Geeralizatio error For data poits, the degree 9 polyomial gives a perfect fit (Lagrage iterpolatio). Error is zero. Is it always good to miimize the traiig error? CS 75 Machie Learig Overfittig For data poits, degree 9 polyomial gives a perfect fit (Lagrage iterpolatio). Error is zero. Is it always good to miimize the traiig error? NO!! More importat: How do we perform o the usee data? CS 75 Machie Learig 3

14 Overfittig Situatio whe the traiig error is low ad the geeralizatio error is high. Causes of the pheomeo: Model with a large umber of parameters (degrees of freedom) Small data size (as compared to the compleity of the model) CS 75 Machie Learig How to evaluate the learer s performace? Geeralizatio error is the true error for the populatio of eamples we would like to optimize E(, y )[( y f ( )) ] But it caot be computed eactly Sample mea oly approimates the true mea Optimizig the traiig error ca lead to the overfit, i.e. traiig error may ot reflect properly the geeralizatio error i,.. ( y So how to test the geeralizatio error? i f ( )) i

15 How to evaluate the learer s performace? Geeralizatio error is the true error for the populatio of eamples we would like to optimize Sample mea oly approimates it Two ways to assess the geeralizatio error is: Theoretical: Law of Large umbers statistical bouds o the differece betwee true geeralizatio ad sample mea errors Practical: Use a separate data set with m data samples to test the model (Average) test error Error( Dtest, f ) ( y j f ( j )) m j,.. m Evaluatio of the geeralizatio performace Split available data D ito two disjoit sets: traiig set D trai testig set D test Dataset Traiig set Testig set Optimize trai error Lear (fit) Also called: Simple holdout method Typically /3 traiig ad /3 testig Predictive model Evaluate Calculate test error 5

16 Assessmet of model performace Assessmet of the geeralizatio performace of the model: Basic rule: Never ever touch the test data durig the learig/model buildig process Test data should be used for the fial evaluatio oly Testig of models: regressio Data set Traiig set Test set Lear o the traiig set The model Evaluate o the test set

17 Testig of models: classificatio Data set Traiig set Test set case cotrol case cotrol Lear o the traiig set The model Evaluate o the test set Evaluatio measures Easiest way to evaluate the model: Error fuctio used i the optimizatio is adopted also i the evaluatio Advatage: may help us to see model overfittig. Simply compare the error o the traiig ad testig data. Evaluatio of the models ofte cosiders: Other aspects or statistics of the model ad its performace Moreover the Error fuctio used for the optimizatio may be a coveiet approimatio of the quality measure we would really like to optimize 7

18 Evaluatio measures: classificatio Biary classificatio: Predictio Case Cotrol Case TP.3 FN. Actual Cotrol FP. TN. Misclassificatio error: E FP FN Sesitivity: TP SN TP FN Specificity: TN SP TN FP A learig system: basic cycle. Data: D { d, d,.., d}. Model selectio: Select a model or a set of models (with parameters) E.g. y a b 3. Choose the objective fuctio Squared error ( yi f ( i )) i. Learig: Fid the set of parameters optimizig the error fuctio The model ad parameters with the smallest error 5. Testig/validatio: Evaluate o the test data. Applicatio Apply the leared model to ew data f ()

19 A learig system: basic cycle. Data: D { d, d,.., d}. Model selectio: Select a model or a set of models (with parameters) E.g. 3. Choose the objective fuctio ( yi f ( i )) i - Squared error - -. Learig: Fid the set of parameters optimizig the error fuctio The model ad parameters with the smallest error Testig/validatio: Evaluate o the test data. Applicatio Apply the leared model to ew data f () y a b A learig system: basic cycle. Data: D { d, d,.., d}. Model selectio: Select a model or a set of models (with parameters) E.g. y a b 3. Choose the objective fuctio Squared error ( yi f ( i )) i. Learig: Fid the set of parameters optimizig the error fuctio The model ad parameters with the smallest error 5. Testig/validatio: Evaluate o the test data. Applicatio Apply the leared model to ew data f () CS 75 Machie Learig 9

20 Steps take whe desigig a ML system Data Model selectio Choice of Error fuctio Learig/optimizatio Evaluatio Applicatio Add some compleity Data Data cleaig/preprocessig Feature selectio/dimesioality reductio Model selectio Choice of Error fuctio Learig/optimizatio Evaluatio Applicatio

21 Desigig a ML solutio Data Data cleaig/preprocessig Feature selectio/dimesioality reductio Model selectio Choice of Error fuctio Learig/optimizatio Evaluatio Applicatio Desigig a ML solutio Data Data cleaig/preprocessig Feature selectio/dimesioality reductio Model selectio Choice of Error fuctio Learig/optimizatio Evaluatio Applicatio

22 Data source ad data biases Uderstad the data source Uderstad the data your models will be applied to Watch out for data biases: Make sure the data we make coclusios o are the same as data we used i the aalysis It is very easy to derive uepected results whe data used for aalysis ad learig are biased Results (coclusios) derived for a biased dataset do ot hold i geeral!!! Data biases Eample: Assume you wat to build a ML program for predictig the stock behavior ad for choosig your ivestmet strategy Data etractio: pick compaies that are traded o the stock market o Jauary 7 Go back 3 years ad etract all the data for these compaies Use the data to build a ML model supportig your future ivestmets Questio: Would you trust the model? Are there ay biases i the data? CS 75 Machie Learig

23 Steps take whe desigig a ML system Data Data cleaig/preprocessig Feature selectio/dimesioality reductio Model selectio Choice of Error fuctio Learig/optimizatio Evaluatio Applicatio Data cleaig ad preprocessig Data you receive may ot be perfect: Cleaig Preprocessig (coversios) Cleaig: Get rid of errors, oise, Removal of redudacies Preprocessig: Reamig Rescalig (ormalizatio) Discretizatios Abstractio Aggregatio New attributes CS 75 Machie Learig 3

24 Data preprocessig Reamig (relabelig) categorical values to umbers dagerous i cojuctio with some learig methods umbers will impose a order that is ot warrated Eample: assume the followig ecodig of values High, Normal, Low High Normal Low > implies High > Normal: Is it OK? > implies Normal > Low: Is it OK? > implies High > Low: Is it OK?? Data preprocessig Reamig (relabelig) categorical values to umbers dagerous i cojuctio with some learig methods umbers will impose a order that is ot warrated Eample: assume the followig ecodig of values High, Normal, Low High Normal Low > implies High > Normal: Is it OK? > implies Normal > Low: Is it OK? > implies High > Low: Is it OK?

25 Data preprocessig Reamig (relabelig) categorical values to umbers dagerous i cojuctio with some learig methods umbers will impose a order that is ot warrated High Normal Low True False Ukow? Data preprocessig Reamig (relabelig) categorical values to umbers dagerous i cojuctio with some learig methods umbers will impose a order that is ot warrated High Normal Low True False Ukow Red Blue Gree? 5

26 Data preprocessig Reamig (relabelig) categorical values to umbers dagerous i cojuctio with some learig methods umbers will impose a order that is ot warrated High Normal Low True False Ukow Red Blue Gree Data preprocessig Reamig (relabelig) categorical values to umbers Problem: How to safely represet the differet categories as umbers whe o order eists? Solutio: Use idicator vector (or oe-hot) represetatio. Eample: Red, Blue, Gree colors 3 categories use a vector of size 3 with biary values Ecodig: Red: (,,); Blue: (,,); Gree: (,,)