Performance Evaluation

Transcription

1 Performance Evaluation Dan Lizotte Evaluating Performance Which do ou prefer and wh?

2 Evaluating Performance Which do ou prefer and wh? Evaluating Performance Performance of a Fied Hpothesis (HTF , JWHT., 5)

3 Define the loss (error) of the hpothesis on an eample (, ) as L(h(), ) Suppose (X, Y ) is a vector-valued random variable. Then what is L(h(X), Y ) Performance of a Fied Hpothesis Given a model h, (which could have come from anwhere), its generalization error is: E[L(h(X), Y )] Given a set of data points ( i, i ) that are realizations of (X, Y ), we can compute the empirical error l h,n = n What is l h,n? n L(h( i ), i ) i= Generalization error of hpotheses from last da 5 log(generalization Error) Degree of Polnomial

4 Estimates of generalization error using points 5 log(generalization Error) Degree of Polnomial Sample Mean Given a dataset (collection of realizations),,..., n of X, the sample mean is: n = n Given a dataset, n is a fied number. We use X n to denote the random variable corresponding to the sample mean computed from a randoml drawn dataset of size n. i i Datasets and sample means Datasets of size n = 5, sample means plotted in red. 4

5 Statistics, Parameters, and Estimation A statistic is an summar of a dataset. (E.g. function applied to a dataset. n, sample median.) A statistic is the result of a A parameter is an summar of the distribution of a random variable. (E.g. µ X, median.) A parameter is the result of a function applied to a distribution. Estimation uses a statistic (e.g. n ) to estimate a parameter (e.g. µ X ) of the distribution of a random variable. Estimate: value obtained from a specific dataset Estimator: function (e.g. sum, divide b n) used to compute the estimate Estimand: parameter of interest Sampling Distributions (AoS, p.6, q.9) Given an estimate, how good is it? 5

6 The distribution of an estimator is called its sampling distribution. Bias (AoS, p.9) The epected difference between estimator and parameter. For eample, If, estimator is unbiased. E[ X n µ X ] Sometimes, n > µ X, sometimes n < µ X, but the long run average of these differences will be zero. Variance The epected squared difference between estimator and its mean Positive for all interesting estimators. For an unbiased estimator E[( X n E[ X n ]) ] E[( X n µ X ) ] Sometimes, n > µ X, sometimes n < µ X, but the squared differences are all positive and do not cancel out. 6

7 Normal (Gaussian) Distribution (AoS, p.8) f X () = e ( µ X ) σ X σ X π Normal distribution is defined b two parameters: µ X, σ X. The normal distribution is special (among other reasons) because man estimators have approimatel normal sampling distributions or have sampling distributions that are closel related to the normal. For an estimator like X n, if we know µ Xn and σ Xn, then we can sa a lot about how good it is. Central Limit Theorem (AoS, p.77) Informall: The sampling distribution of Xn is approimatel normal if n is big enough. More formall, for X with finite variance: where F Xn ( ) e ( µ X ) σ n σ n π σ n = σ n is called the standard error and σ is the variance of X. Who cares? Eruptions dataset has n = 7 observations. Our estimate of the mean of eruption times is 7 = What is the probabilit of observing an 7 that is within seconds of the true mean? Who cares? B the C.L.T., Pr(.7 X 7 µ X.7) =.7 =.7 e ( µ X ) σ n πσn =.986 Note! I estimated σ X here. (Look up t-test for details.) 7

8 .7 =.7 e ( µ X ) σ n =.986 πσn 6 4 Densit..... z Confidence Intervals (AoS, p.9) Tpicall, we specif confidence given b α Use the sampling distribution to get an interval that traps the parameter (estimand) with probabilit α. 95% C.I. for eruption mean is (.5,.6) 8

9 6 95% Confidence Region 4 Densit z 9

10 What a Confidence Interval Means

11 Effect of n on width Performance Evaluation - Test Sets Training error underestimates generalization error. It is a biased estimator. If ou reall want a good estimate of generalization error, ou need to hold out a separate test set of data not used for training. Possibl of size n = (.96) σ L d where σl is the variance of the loss (which has to be guessed or estimated from training) and d is half-width of a 95% confidence interval. Could report test the error, but then deplo whatever ou train on the whole data. (Probabl won t be worse.)

12 Eample - linear model Training Data ## [] "Estimated variance of errors: " ## [] "Sample required for CI width of. (+-.): 65" Eample - linear model Training Data Testing Data ## TestMSE VarOfErrors StdOfSquaredErrors n StandardError CI_left ## ## CI_right

13 ##.885 Choosing Performance Measures for Regression: Mean Errors MSE = n n i= RMSE = n MAE = n (ŷ i i ) n (ŷ i i ) i= n i= ŷ i i I find MAE easier to interpret. (How far am I from the correct value, on average?) RMSE is at least in the same units as the. Choosing Performance Measures for Regression: Mean Relative Error MRE = n n i= ŷ i i i Scales error according to magnitude of true. E.g., if MRE=., then regression is wrong b % of the value of, on average. If this is appropriate for our problem then linear regression, which assumes additive error, ma not be appropriate. Options include using a different model or regression on log rather than on. Etra slides - The Bootstrap The Bootstrap (AoS, p.) CLT gives theoretical approimate sampling distribution of Xn. We could also estimate the sampling distribution of X n b drawing man datasets of size n, computing X n on each, constructing histogram. This is impossible. But we can use the data we have as a surrogate. The Bootstrap Call our dataset D. Draw B new datasets b sampling observations with replacement from D. (B is often at least ) Compute X (b) n for each of the datasets. Use the histogram/empirical distribution of these pretend X to determine confidence limits.

14 Bootstrap eample librar(boot) bootstraps <- boot(faithful$eruptions,function(d,i){mean(d[i])},r=5) bootdata = data.frame(bars=bootstraps$t); limits = quantile(bootdata$bars,c(.5,.975)) ggplot(bootdata, aes(=bars)) + labs(="prop.") + geom_histogram(aes( =..densit..)) + geom_errorbarh(aes(min=limits[[]], ma=limits[[]], =c()),height=.5,colour="red",size=) 6 4 Prop bars 4

15 Realit Check.8.6 Prop eruptions 5