Variations on Regression Models. Prof. Bennett Math Models of Data Science 2/02/06

Transcription

1 Variations on Regression Models Prof. Bennett Math Models of Data Science 2/02/06

2 Outline Steps in modeling Review of Least Squares model Model in E & K pg Aqualsol version of E&K Other loss functions Other regularization

3 Modeling Process Gather data: input and output pairs Represent data mathematically: (X,Y) Select parametric model: g(x) Select loss function: Loss Select regularization Optimize parameters with respect to given loss Estimate out-of-sample accuracy

4 Predict Drug Bioavailability Aqua solubility = Aquasol 525 descriptors generated Electronic TAE Traditional y R molecules with tested solubility x i R = 197

5 Modeling Process Gather data: input and output pairs Represent data mathematically: (X,Y) Select parametric model: g(x) Select loss function: Loss Select regularization Optimize parameters with respect to given loss Estimate out-of-sample accuracy

6 b=2 1-d Regression with bias w, x + b y x

7 Modeling Process Gather data: input and output pairs Represent data mathematically: (X,Y) Select parametric model: g(x) Select loss function: Loss Select regularization Optimize parameters with respect to given loss Estimate out-of-sample accuracy

8 Linear Regression Given training data: ( x, ),( x, ),,( x, ),,( x, ) ( ) S = y y y y n points x R and labels y R i Construct linear function: g( x) = x, w = x' w = wi( x) i + b Goal for future data (x,y) with y unknown g( x) y i i n i= 1 i

9 Least Squares Approximation Want g( x) y Define error f( x, y) = y g( x) = ξ Minimize loss i= 1 ( x ) 2 Lgs (, ) = yi g( i)

10 Least Squares Loss L( w) = ( x ' w+ b y ) i= 1 i = Xw + eb y 2 i 2 2 norm = ( Xw+ eb y)'( Xw+ eb y)

11 Convex Functions A function f is (strictly) convex on a convex set S, if and only if for any x,y S, f(λx+(1- λ)y)(<) λf(x)+ (1- λ)f(y) for all 0 λ 1. f(x+(1- λ)y) f(y) f(x) x x+(1- λ)y y

12 Theorem y Consider problem min f(x) unconstrained. If f ( x) = 0 and f is convex, then x is a global minimum. Proof: f( y) f( x) + ( y x)' f ( x) by convexity of f = f( x) since f( x) = 0.

13 Stationary Points Note that this condition is not sufficient f ( x*) = 0 Also true for local max and saddle points

14 Modeling Process Gather data: input and output pairs Represent data mathematically: (X,Y) Select parametric model: g(x) Select loss function: Loss Select regularization Optimize parameters with respect to given loss Estimate out-of-sample accuracy

15 Optimal Solution Want: y Xw+ be eis a vector of ones Mathematical Model: Optimality Conditions: 2 2 min w L( w, bs, ) = y ( Xw+ eb) + w L( w, b, S) = 2 X'( y Xw eb) + 2w = 0 w L( w, b, S) = 2 e'( y Xw eb) = 0 b

16 Optimal Solution Thus : ee ' b = ey ' exw ' ey ' exw ' b = = mean( y) mean( X)' w ( XX ' + λiw ) = Xy ' Xe ' b Idea: Scale data so means are 0, e.g ey ' = 0 ex ' = 0

17 Recenter Data Shift y by mean 1 µ = yi yi : = y µ i i= 1 Shift x by mean 1 x = xi xi : = xi x i= 1

18 Ridge Regression with bias Center data by Calculate w x and µ X= X eµ ' y = y µ e 1 w = ( X'X+λI) X' y Calculate b = ( µ xw ' ) To predict new point g(x)=x w+b

19 Modeling Process Gather data: input and output pairs Represent data mathematically: (X,Y) Select parametric model: g(x) Select loss function: Loss Select regularization Optimize parameters with respect to given loss Estimate out-of-sample accuracy

20 Generalization To estimate generalization error: Divide test into training set= Xtrain 100 points in Aquasol and test set = Xtest 97 points in Aquasol Create g(x) using Xtrain Evaluate on Xtest

21 Train and Test for λ test error train error 60 squared error lambda

22 Could we do better Other loss functions? Other types of regularization?

23 Nonlinear time series Read E&K pg What are the modeling tricks?

24 Linear Programming Matlab can solve linear programs of the following form for give f,a,aeq,lb,ub min x f' x A* x b A * x= b eq eq lb x ub

25 Sample Problem T=[ ] R=[ ]

26 Matlab Let ith row of B be Problem becomes 2 i [ t t 1] i min [1000]*[ wabc]' wabc,,, w e B a R e B b R + c

27 Comparison of norms w =1 2-norm 1-norm -norm

28 Inconsistent System of equations Want Ax b? How does this correspond to our framework? What are the loss functions?

29 Review of norms 2-norm y 2 = i = 1 1-norm -norm n y 2 i y = = yi n 1 i 1 y max y = i i

30 Translate the ideas to our framework Using the ideas in E&K, formulate the following problem as a linear program min Xw+ b y w, b

31 Translate the ideas to our framework Using the ideas in E&K, formulate the following problem as a linear program min Xw+ b y b w, 1

32 Translate the ideas to our framework Using the ideas in E&K, formulate the following problem as a linear program min Xw+ b y +λ w b w, 1 1

33 Challenge problem Popular loss function ε-insensitive min max( x ' w+ b y ε,0) wb, i i -ε ε

34 Discussion Questions Sketch each type of loss function and discuss advantages and disadvantages of different loss functions? (1-norm, 2- norm, -norm, ε-insensitive) What would be the ramification of using these same functions for regularization?