Using Bad Learners to find Good Configurations

Transcription

1 Using Bad Learners to find Good Configurations Vivek Nair Tim Menzies North Carolina State University Norbert Siegmund Sven Apel Bauhaus-Universität Weimar University of Passau

2 Configurable Systems and Variability System

3 Configurable Systems and Variability t_2k_480p24.y4m System

4 Configurable Systems and Variability t_2k_480p24.y4m System trailer_480p24.x264

5 Configurable Systems and Variability Configuration options c1 c2 c3 c4 t_2k_480p24.y4m... System cn trailer_480p24.x264

6 Configurable Systems and Variability Configuration options c1 c2 c3 c4 t_2k_480p24.y4m... System cn trailer_480p24.x264 Non-functional behavior: response time, throughput, etc.

7 Configurable Systems and Variability Configuration options c1 c2 c3 c4 t_2k_480p24.y4m... System cn trailer_480p24.x264 Non-functional behavior: response time, throughput, etc.

8 Performance Optimization is Necessary!

9 Performance Optimization is getting more Complex! Necessary [1] Xu et. al Hey, you have given me too many knobs!: understanding and dealing with over-designed configuration in system software.fse 2015 [2] Van Aken, Dana, et al. "Automatic Database Management System Tuning Through Large-scale Machine Learning." International Conference on Management of Data. ACM, 2017.

10 Performance Optimization is required since Default Configuration is Bad! Necessary Complex [1] Van Aken, Dana, et al. "Automatic Database Management System Tuning Through Large-scale Machine Learning." International Conference on Management of Data. ACM, [2] Herodotou, Herodotos, et al. "Starfish: A Self-tuning System for Big Data Analytics." CIDR

11 Performance Optimization can be Expensive! Evaluation of single instance of their hardware/software co-design problem can take weeks[1] Necessary Complex Default is bad Rolling Sort use-case required 21 days, within a total experimental time of about 2.5 months[2] Test suite generation using Evolutionary Algorithm can take weeks[3] Image recognition workload and speech recognition workload, jobs ran for many hours or days[4] [1] Zuluaga, Marcela, et al. "Active learning for multi-objective optimization." International Conference on Machine Learning [2] Jamshidi, Pooyan, and Giuliano Casale. "An uncertainty-aware approach to optimal configuration of stream processing systems."mascots-2016 [3] Wang, Tiantian, et al. "Searching for better configurations: a rigorous approach to clone evaluation." FSE-2013 [4] Venkataraman, Shivaram, et al. "Ernest: Efficient Performance Prediction for Large-Scale Advanced Analytics." NSDI

12 Existing Solutions [Guo 13] [Siegmund 12] [Sarkar 15] [Nair 17] Configuration Configuration F1= True F2= False F3= True Response Time = 2100 ms Request F1= True F2= False F3= True Response Time ~ 2100 ms Request Software System [Siegmund 12] Siegmund, Norbert, et al. "Predicting performance via automated feature-interaction detection." ICSE [Guo 13] Guo, Jienmei, et al. Variability-aware performance prediction: A statistical learning approach. ASE-2013 [Sarkar 15] Sarkar, Atri, et al. "Cost-efficient sampling for performance prediction of configurable systems (t)." ASE-2015 [Nair 17] Nair, Vivek, et al. "Faster discovery of faster system configurations with spectral learning." ASE Journal to appear. Surrogate System

13 Existing Solutions 2012 [Siegmund 12] 2013 [Guo 13] 2015 [Sarkar 15] Configuration Space [Siegmund 12] Siegmund, Norbert, et al. "Predicting performance via automated feature-interaction detection." ICSE [Guo 13] Guo, Jienmei, et al. Variability-aware performance prediction: A statistical learning approach. ASE-2013 [Sarkar 15] Sarkar, Atri, et al. "Cost-efficient sampling for performance prediction of configurable systems (t)." ASE-2015 [Nair 17] Nair, Vivek, et al. "Faster discovery of faster system configurations with spectral learning." ASE Journal to appear [Nair 17]

14 Existing Solutions 2012 [Siegmund 12] 2013 [Guo 13] [Sarkar 15] 1. [Nair 17] Divide the configuration space into training and testing sets; Train 1 Test Configuration Space [Siegmund 12] Siegmund, Norbert, et al. "Predicting performance via automated feature-interaction detection." ICSE [Guo 13] Guo, Jienmei, et al. Variability-aware performance prediction: A statistical learning approach. ASE-2013 [Sarkar 15] Sarkar, Atri, et al. "Cost-efficient sampling for performance prediction of configurable systems (t)." ASE-2015 [Nair 17] Nair, Vivek, et al. "Faster discovery of faster system configurations with spectral learning." ASE Journal to appear.

15 Existing Solutions [Siegmund 12] [Guo 13] [Sarkar 15] [Nair 17] Divide the configuration space into training and testing sets; Measure all the configurations in the testing set; Train 1 Test 2 Configuration Space [Siegmund 12] Siegmund, Norbert, et al. "Predicting performance via automated feature-interaction detection." ICSE [Guo 13] Guo, Jienmei, et al. Variability-aware performance prediction: A statistical learning approach. ASE-2013 [Sarkar 15] Sarkar, Atri, et al. "Cost-efficient sampling for performance prediction of configurable systems (t)." ASE-2015 [Nair 17] Nair, Vivek, et al. "Faster discovery of faster system configurations with spectral learning." ASE Journal to appear.

16 Existing Solutions [Siegmund 12] [Sarkar 15] [Guo 13] Train [Nair 17] Divide the configuration space into training and testing sets; Measure all the configurations in the testing set; Iteratively sampling configuration from training set to build a model and test the model against testing set; Test 2 Configuration Space [Siegmund 12] Siegmund, Norbert, et al. "Predicting performance via automated feature-interaction detection." ICSE [Guo 13] Guo, Jienmei, et al. Variability-aware performance prediction: A statistical learning approach. ASE-2013 [Sarkar 15] Sarkar, Atri, et al. "Cost-efficient sampling for performance prediction of configurable systems (t)." ASE-2015 [Nair 17] Nair, Vivek, et al. "Faster discovery of faster system configurations with spectral learning." ASE Journal to appear.

17 Existing Solutions [Siegmund 12] [Sarkar 15] [Guo 13] Train 1 4 Test [Nair 17] Divide the configuration space into training and testing sets; Measure all the configurations in the testing set; Iteratively sampling configuration from training set to build a model and test the model against testing set; Exit when an accurate model is built (e.g., error = 0.1) 2 Configuration Space [Siegmund 12] Siegmund, Norbert, et al. "Predicting performance via automated feature-interaction detection." ICSE [Guo 13] Guo, Jienmei, et al. Variability-aware performance prediction: A statistical learning approach. ASE-2013 [Sarkar 15] Sarkar, Atri, et al. "Cost-efficient sampling for performance prediction of configurable systems (t)." ASE-2015 [Nair 17] Nair, Vivek, et al. "Faster discovery of faster system configurations with spectral learning." ASE Journal to appear.

18 Limitation of Existing Solutions

19 Limitation of Existing Solutions

20 Limitation of Existing Solutions

21 Limitation of Existing Solutions

22 Limitation of Existing Solutions

23 Limitation of Existing Solutions

24 Core Insight

25 Rank Preserving Model

26 Rank Preserving Model

27 Rank Preserving Model

28 Rank Preserving Model

29 Rank Preserving Model

30 Rank Preserving Model

31 Rank Preserving Model

32 Rank Preserving Model Train 1 Test 2 Configuration Space Divide the configuration space into training and testing sets; Measure all the configurations in the testing set; Iteratively sampling configuration from training set to build a model and test the model against testing set;

33 Rank Preserving Model Train 1 4 Test 2 Configuration Space 4. Divide the configuration space into training and testing sets; Measure all the configurations in the testing set; Iteratively sampling configuration from training set to build a model and test the model against testing set; Calculate accuracy model should get progressively more accurate

34 Rank Preserving Model Train 1 4 Test 2 Configuration Space Divide the configuration space into training and testing sets; Measure all the configurations in the testing set; Iteratively sampling configuration from training set to build a model and test the model against testing set; Calculate accuracy model should get progressively more accurate Exit when a model built does not improve (accuracy plateau)

35 Evaluation

36 Baselines Progressive Sampling[1] Sequentially (randomly) sample configuration to build a decision tree till threshold accuracy is reached Projective Sampling[2] Using minimal set of initial sample configurations to project the sampling cost based on a threshold accuracy [1] Guo, Jianmei, et al. Variability-aware performance prediction: A statistical learning approach. ASE-2013 [2] Sarkar, Atri, et al. "Cost-efficient sampling for performance prediction of configurable systems (t)." ASE-2015

37 Research Questions RQ1 - Can inaccurate models accurately rank configurations? RQ2 - How expensive is a rank-based method?

38 Subject Software Systems Video Encoder Databases Utility Compression Numerical Grid benchmark Web server Data processing

39 Subject Software Systems Pooyan Jamshidi Sven Apel Norbert Siegmund Combined effort = 6 computational months

40 Experimental Settings Data (100)

41 Experimental Settings Data (100) Training Pool (40) Prune Pool (20) Testing Pool (40)

42 Experimental Settings Data (100) Training Pool (40) Prune Pool (20) Testing Pool (40)

43 Results

44 RQ1: Can inaccurate models accurately rank configurations?

45 RQ1: Can inaccurate models accurately rank configurations?

46 RQ1: Can inaccurate models accurately rank configurations?

47 RQ1: Can inaccurate models accurately rank configurations? The lower the better

48 RQ1: Can inaccurate models accurately rank configurations? The lower the better

49 RQ1: Can inaccurate models accurately rank configurations? 6 predictedoptima l 1 actualoptimal The lower the better

50 RQ1: Can inaccurate models accurately rank configurations? RD = 1-6 =5 6 predictedoptimal 1 actualoptimal The lower the better

51 RQ1: Can inaccurate models accurately rank configurations?

52 RQ1: Can inaccurate models accurately rank configurations?

53 RQ1: Can inaccurate models accurately rank configurations? 8/1535

54 RQ1: Can inaccurate models accurately rank configurations?

55 RQ2: How expensive is a rank-based approach?

56 RQ2: How expensive is a rank-based approach? The lower the better

57 RQ2: How expensive is a rank-based approach?

58 RQ2: How expensive is a rank-based approach?

59 Conclusion

60