Practical Machine Learning in R

Transcription

1 Practical Machine Learning in R Tuning Lars Kotthoff 12 larsko@uwo.edu 1 with slides from Bernd Bischl and Michel Lang 2 slides available at 1

2 Tuning Frank Hutter and Marius Lindauer, Algorithm Configuration: A Hands on Tutorial, AAAI

3 Hperparameter Tuning ud to find best hperparameters for a method in a data-dependent wa important to achieve good performance in practice esntial for some methods, e.g. SVMs 3

4 Grid and Random Search Bergstra, James, and Yoshua Bengio. Random Search for Hper-Parameter Optimization. J. Mach. Learn. Res. 13, no. 1 (Februar 2012):

5 Population-Bad Methods e.g. Racing and Genetic Algorithms start with population of random configurations eliminate weak individuals generate new population from strong individuals iterate 5

6 Model-Bad Search currentl considered state-of-the-art build surrogate model of parameter-respon surface evaluate cheap model instead of expensive target function u model to o next point to evaluate target function at iterate 6

7 Model-Bad Search Components learner for surrogate model method for generating t of initial obrvations infill criterion how to get next evaluation point termination criterion 7

8 Model-Bad Search Example 1D Iter = 1, Gap = e init x 8

9 Model-Bad Search Example 1D Iter = 2, Gap = e init 1 0 x 9

10 Model-Bad Search Example 1D Iter = 3, Gap = e init x 10

11 Model-Bad Search Example 1D Iter = 4, Gap = e e 04 6e 04 4e 04 init 2e 04 0e+00 x 11

12 Model-Bad Search Example 1D Iter = 5, Gap = e e 04 init 1e 04 0e+00 x 12

13 Model-Bad Search Example 1D Iter = 6, Gap = e init x 13

14 Model-Bad Search Example 1D Iter = 7, Gap = e e 05 4e 05 3e 05 init 2e 05 1e 05 0e+00 x 14

15 Model-Bad Search Example 1D Iter = 8, Gap = e e 05 e 05 e 05 init 5.0e 06 e+00 x 15

16 Model-Bad Search Example 1D Iter = 9, Gap = e e e e 06 init 2.5e 06 e+00 x 16

17 Model-Bad Search Example 1D Iter = 10, Gap = e e 07 3e 07 2e 07 init 1e 07 0e+00 x 17

18 Model-Bad Search Example 2D 2.0 init init init init

19 Model-Bad Search Example 2D 2.0 init init init init

20 Model-Bad Search Example 2D 2.0 init init init init 20

21 Model-Bad Search Example 2D 2.0 init init init init

22 Model-Bad Search Example 2D 2.0 init init init init

23 Model-Bad Search Example 2D 2.0 init init init init

24 Model-Bad Search Example 2D 2.0 init init init init

25 Model-Bad Search Example 2D 2.0 init init init init 25

26 Model-Bad Search Example 2D 2.0 init init init init

27 Model-Bad Search Example 2D 2.0 init init init init

28 When are we done? most approaches incomplete cannot prove optimalit, not guaranteed to find optimal solution (in finite time) performance highl dependent on configuration space How do we know when to stop? 28

29 Time Budget How much time/how man function evaluations? too much wasted resources too little suboptimal result experiment with different ttings run veral times with different random initializations 29

30 Evaluation repeated evaluation with same train/test split statisticall unsound violates independence assumption example: parameters have no real effect, onl random variation still one parameter tting will win solution: different train/test splits 30

31 Nested Resampling 31

32 In mlr tuning with different methods available as wrapper model-bad optimization available in mlrmbo package nested resampling available as resampling method 32

33 Exercis 07-tuning-exercis.Rmd 33