Automated smart test design! and its applications in! software transplantation, improvement and android testing!

Transcription

1 Automated smart test design! and its applications in! software transplantation, improvement and android testing! Mark Harman Talk by Mark Harman based on PhD work by Ke Mao, Alex Marginean Jointly supervised by Yue Jia University College London

2 Madame Tussaud s Sherlock Holmes Museum 20 mins walk Marble Arch National History Museum Eros National Gallery Westminster Abbey Nelson s Column British Museum London Eye House of Parliament Covent Garden Market Tate Modern RoyalCourts of Justice Globe Theatre St. Paul s

3 COWs CREST Open Workshop Roughly one per month! Discussion based Recorded and archived

4 COWs CREST Open Workshop Roughly one per month! Discussion based! Recorded and archived

5 COWs CREST Open Workshop Roughly one per month! Discussion based Recorded and archived

6 COWs

7 COWs #Total Registrations 1690 #Unique Attendees 721 #Unique Institutions 257 #Countries 44 #Talks 455! (Last updated on September 2016)!

8 Search Based Optimization S B S T Software Testing

9 In SBST we apply search techniques to search large search spaces, guided by a fitness function that captures natural counterparts as test objectives. Tabu Search Ant Colonies Particle Swarm Optimization Hill Climbing Genetic Algorithms Genetic Programming Simulated Annealing Greedy LP Random Estimation of Distribution Algorithms

10 Search Based Software Engineering In SBSE we apply search techniques to search large search spaces, guided by a fitness function that captures natural counterparts as test objectives. Tabu Search Ant Colonies Particle Swarm Optimization Hill Climbing Genetic Algorithms Genetic Programming Simulated Annealing Greedy LP Random Estimation of Distribution Algorithms

11 SBSE Tutorial and Survey

12 SBSE Tutorial and Survey Mark Harman, Phil McMinn, Jerffeson Teixeira de Souza and Shin Yoo. Search Based Software Engineering: Techniques, Taxonomy, Tutorial. Springer, 2012.! google: SBSE tutorial!!! Mark Harman, Afshin Mansouri and Yuanyuan Zhang. Search Based Software Engineering: Trends, Techniques and Applications ACM Computing Surveys. 45(1): Article 11, 2012.! google: SBSE survey

13 SBSE Tutorial and Survey Mark Harman, Phil McMinn, Jerffeson Teixeira de Souza and Shin Yoo. Search Based Software Engineering: Techniques, Taxonomy, Tutorial. Springer, 2012.! google: SBSE tutorial!!! Mark Harman, Afshin Mansouri and Yuanyuan Zhang. Search Based Software Engineering: Trends, Techniques and Applications ACM Computing Surveys. 45(1): Article 11, 2012.! google: SBSE survey

14 SBSE Tutorial and Survey Mark Harman, Phil McMinn, Jerffeson Teixeira de Souza and Shin Yoo. Search Based Software Engineering: Techniques, Taxonomy, Tutorial. Springer, 2012.! google: SBSE tutorial!!! Mark Harman, Afshin Mansouri and Yuanyuan Zhang. Search Based Software Engineering: Trends, Techniques and Applications ACM Computing Surveys. 45(1): Article 11, 2012.! google: SBSE survey

15 SBSE Tutorial and Survey Mark Harman, Phil McMinn, Jerffeson Teixeira de Souza and Shin Yoo. Search Based Software Engineering: Techniques, Taxonomy, Tutorial. Springer, 2012.! google: SBSE tutorial!!! Mark Harman, Afshin Mansouri and Yuanyuan Zhang. Search Based Software Engineering: Trends, Techniques and Applications ACM Computing Surveys. 45(1): Article 11, 2012.! google: SBSE survey

16 SBSE Tutorial and Survey Mark Harman, Phil McMinn, Jerffeson Teixeira de Souza and Shin Yoo. Search Based Software Engineering: Techniques, Taxonomy, Tutorial. Springer, 2012.! google: SBSE tutorial!!! Mark Harman, Afshin Mansouri and Yuanyuan Zhang. Search Based Software Engineering: Trends, Techniques and Applications ACM Computing Surveys. 45(1): Article 11, 2012.! google: SBSE survey

17

18 800" Accumulated*Number*of*SBST*Publica5ons* 700" 600" 500" 400" 300" 200" 100" y"="0.0013x 4 "*"0.061x 3 "+"1.0008x 2 "*"5.8636x"+"10.443" Polynomial yearly rise in the number of papers Search Based Software Testing 0" 1975" 1977" 1979" 1981" 1983" 1985" 1987" 1989" 1991" 1993" 1995" 1997" 1999" 2001" 2003" 2005" 2007" 2009" 2011" 2013"

19 2014# 2011# 2008# 2005# 2002# 1999# 1996# 1993# 1990# 1987# 1984# 1981# 1978# 1975# 0%# 10%# 20%# 30%# 40%# 50%# 60%# 70%# 80%# 90%# 100%# The changing ratio SBSE to SBST SBST$ Other$SBSE$Publica2ons$

20 S B S T

21 Structural

22 Structural find tests to! cover! branches,! statements &! dataflow, etc.

23 Integration

24 Integration find! best component! ordering

25 Temporal

26 Temporal find worst case! execution time

27 CIT

28 CIT find 2-way, 3-way! n-way! interaction tests

29 SPLs

30 Augment

31 Augment find new tests! from old tests

32 Regression

33 Regression find good! subsets and! orders of tests

34 Functional

35 Mutation

36 State! based

37 Model! based

38 Black box

39 Failure! Analysis

40 Security

41 Web/! Services

42 Agents

43

44

45 Joachim Wegener and Oliver Bühler. GECCO 2004

46 Wasif Afzal, Richard Torkar, Robert Feldt and Greger Wikstrand. SSBSE 2010

47 Nikolai Tillmann, Jonathan de Halleux and Tao Xie. ASE 2014

48 AUSTIN applied to real-world embedded automotive industry: Daimler, B&M Systemtechnik. Recommended for testing C. Kiran Lakhotia,Mark Harman,and Hamilton Gross. I&ST 2013

49 EvoSuite automatically generates test cases for Java code. An excellent and high recommended tool. Gordon Fraser and Andrea Arcuri. ESEC/FSE 2011

50 NEW KID: SAPIENZ for fully-automated Android testing K. Mao, M. Harman, and Y. Jia. Sapienz: Multi-objective automated testing for Android applications. In ISSTA 16, to appear. Sapienz Ke Mao - Automated Mobile Testing: Dumb Monkeys, Smart Monkeys and Sapienz

51 Unfortunately, Facebook has stopped. Report OK Sapienz Ke Mao - Automated Mobile Testing: Dumb Monkeys, Smart Monkeys and Sapienz

52 ANDROID IS NO TOY Does crash matter? A crash is a fatal failure: - Lost your data/progress - Fatal for domains e.g., medical In this work we report crashes, but Mobile Medical Augmented Reality App for the Apple ipad Interview with Prof. Hans-Peter Meinzer, medicalaugmentedreality.com/2012/03/mobilemedical-augmented-reality-app-for-the-apple-ipadinterview-with-prof-hans-peter-meinzer Sapienz Ke Mao - Automated Mobile Testing: Dumb Monkeys, Smart Monkeys and Sapienz

53 THE STATE OF THE ART fully-automated Android testing K. Mao, M. Harman, and Y. Jia. Sapienz: Multi-objective automated testing for Android applications. In ISSTA 16, to appear. Sapienz Ke Mao - Automated Mobile Testing: Dumb Monkeys, Smart Monkeys and Sapienz

54 THE STATE OF THE ART fully-automated Android testing K. Mao, M. Harman, and Y. Jia. Sapienz: Multi-objective automated testing for Android applications. In ISSTA 16, to appear. Sapienz Ke Mao - Automated Mobile Testing: Dumb Monkeys, Smart Monkeys and Sapienz

55 THE STATE OF THE ART fully-automated Android testing K. Mao, M. Harman, and Y. Jia. Sapienz: Multi-objective automated testing for Android applications. In ISSTA 16, to appear. Sapienz Ke Mao - Automated Mobile Testing: Dumb Monkeys, Smart Monkeys and Sapienz

56 THE STATE OF THE ART Are we there yet? S. R. Choudhary, A. Gorla, and A. Orso. Automated test input generation for Android: Are we there yet? In Proc. of ASE 15, pages , Sapienz Ke Mao - Automated Mobile Testing: Dumb Monkeys, Smart Monkeys and Sapienz

57 THE STATE OF THE ART Are we there yet? Definitely NOT Android Monkey! performs best S. R. Choudhary, A. Gorla, and A. Orso. Automated test input generation for Android: Are we there yet? In Proc. of ASE 15, pages , Sapienz Ke Mao - Automated Mobile Testing: Dumb Monkeys, Smart Monkeys and Sapienz

58 MOTIVATION EXAMPLE monkey testing Sapienz Ke Mao - Automated Mobile Testing: Dumb Monkeys, Smart Monkeys and Sapienz

59 SAPIENZ WORKFLOW SRC/APK SAPIENZ Instrumented APK Multi-level Instrumenter Decompiler Static Strings AUT Android Device States Logger DB Report Generator Gene Interpreter MOTIFCORE Test Replayer Fitness Extractor Evaluate Crash Report Coverage Report Replay Video Atomic Genes Motif Genes Test Generator Initialiser Select GA Vary Solutions (Test Suites) Sapienz Ke Mao - Automated Mobile Testing: Dumb Monkeys, Smart Monkeys and Sapienz

60 SAPIENZ WORKFLOW SRC/APK SAPIENZ Instrumented APK Multi-level Instrumenter Decompiler Static Strings AUT Android Device States Logger DB Report Generator Gene Interpreter MOTIFCORE Test Replayer Fitness Extractor Evaluate Crash Report Coverage Report Replay Video Atomic Genes Motif Genes Test Generator Initialiser Select GA Vary Solutions (Test Suites) Sapienz Ke Mao - Automated Mobile Testing: Dumb Monkeys, Smart Monkeys and Sapienz

61 EMULATOR MODE Sapienz Ke Mao - Automated Mobile Testing: Dumb Monkeys, Smart Monkeys and Sapienz

62 REAL DEVICE MODE System-level Testing! Mobile App Testing! Event-driven App Testing! Automated Exploratory Testing Sapienz Ke Mao - Automated Mobile Testing: Dumb Monkeys, Smart Monkeys and Sapienz

63 THREE EVALUATIONS 68 Benchmarks Statistical Significance Top 1000 GooglePlay Apps COVERAGE Sapienz Dynodroid Monkey 53% 44% 48% FAULTS Sapienz Dynodroid Monkey bugs LENGTH Sapienz Dynodroid Monkey , /27 confirmed bugs Sapienz Ke Mao - Automated Mobile Testing: Dumb Monkeys, Smart Monkeys and Sapienz

64 RANK-CRASH #Crashes Rank Sapienz Ke Mao - Automated Mobile Testing: Dumb Monkeys, Smart Monkeys and Sapienz

65 Education,,9 Personalisation,,38 Transport,,8 Weather,,6 Communication,,23 Puzzle,,71 Books,&,Reference,,6 Tools,, 43 Casual,,106 Educational,,27 Productivity,,30 News,&,Magazines,,6 Entertainment,, 61 Sports,,42 Board,,2 Word,,12 NULL,,13 Racing,,38 Trivia,, 15 Shopping,,25 Arcade,, 61 Media,&,Video,,24 Finance,, 13 Strategy,, 28 Health,&,Fitness,,10 Sapienz Action,, 54 Social,,19 Simulation,, 76 Casino,,15 Role,Playing,,16 Card,,10 Ke Mao - Automated Mobile Testing: Dumb Monkeys, Smart Monkeys and Sapienz Business,,5 Photography,,29 Adventure,, 29 Lifestyle,,27 Travel, &, Local,,21 Music,&,Audio,,25

66 Automated Android Testing custom fit service fit tailored into your development process give us the app; get detailed test report non flakey minimised length test commitment to our community repeatable fault revelation minimised debugging effort thought leadership; open source systems Sapienz Ke Mao - Automated Mobile Testing: Dumb Monkeys, Smart Monkeys and Sapienz

67 How much do you really trust testing? Refuse: I didn t write it! Wait: Until I prove it correct Experiment: Generate tests

68 How much do you really trust testing? Experiment: Generate tests

69 How much do you really trust testing? Experiment: Generate tests

70

71

72 Genetic Improvement of Programs Sensitivity Bowtie2 Analysis GP Programs Programs Programs Bowtie2 Test Improved data Fitness Non-functional property Test harness 70 times faster 30+ interventions HC clean up: 7 slight semantic improvement W. B. Langdon and M. Harman Optimising Existing Software with Genetic Programming. TEC 2015

73 Genetic Improvement of Programs Sensitivity Analysis GP Programs Test data Fitness Non-functional property Test harness

74 Genetic Improvement of Programs Sensitivity Cuda Analysis GP Programs Programs Programs Cuda Test Improved data Fitness Non-functional property Test harness 7 times faster updated for new hardware automated updating W. B. Langdon and M. Harman Genetically Improved CUDA C++ Software, EuroGP 2014

75 Inter version transplantation Sensitivity Analysis GP Programs Test data Fitness Non-functional property Test harness

76 Inter version transplantation v1 MiniSat Sensitivity v2 MiniSat Analysis GP Programs Programs Programs MiniSat Improved vn Test MiniSat data Fitness Non-functional Multi-doner transplant property Test harness Specialized for CIT 17% faster Justyna Petke, Mark Harman, William B. Langdon and Westley Weimer Using Genetic Improvement & Code Transplants to Specialise a C++ program to a Problem Class (EuroGP 14) GECCO Humie! silver medal

77 Real world cross system transplantation Sensitivity Analysis GP Programs Test data Fitness Non-functional property Test harness

78 Real world cross system transplantation Idct Pidgin dependence Mytar analysis regression tests GP CFow CFlow acceptance tests unit tests SOX Web server Trux Crypt 3x5 = 15 experiments 12 were successful Automated Software Transplantation! Earl Barr, Mark Harman, Yue Jia, Alexandru Marginean and Justyna Petke ISSTA Distinguished paper award. Submitted to ICSE 2014.

79 Real world cross system transplantation Doner feature Sensitivity Analysis GP Host feature Host Test data Fitness Successfully autotransplanted new Non-functional functionality and passed all Automated Software Transplantation! Earl Barr, Mark Harman, Yue Jia, Alexandru Marginean and Justyna Petke ISSTA Submitted to ICSE property Test harness regression tests for 12 out of 15 real world systems ACM Distinguished paper award

80 Memory speed trade offs Sensitivity Analysis GP Programs Test data Fitness Non-functional property Test harness

81 Memory speed trade offs System System Sensitivity malloc Analysis GP optimised malloc Test data Fitness Improve execution time by Non-functional property Test harness 12% or achieve a 21% memory consumption reduction Fan Wu, Westley Weimer, Mark Harman, Yue Jia and Jens Krinke Deep Parameter Optimisation Conference on Genetic and Evolutionary Computation (GECCO 15).

82 Reducing energy consumption Sensitivity Analysis GP Programs Test data Fitness Non-functional property Test harness

83 Reducing energy consumption MiniSat Improved MiniSat CIT CIT Sensitivity MiniSat Ensemble Analysis GP Improved MiniSat Ensemble MiniSat Test Improved MiniSat AProVE data Fitness AProVE Non-functional property Test harness Energy consumption can be reduced by as much as 25% Bobby R. Bruce Justyna Petke Mark Harman Reducing Energy Consumption Using Genetic Improvement Conference on Genetic and Evolutionary Computation (GECCO 15).

84 Grow and graft new functionality Sensitivity Analysis? GP Programs Test data Fitness Non-functional property Test harness

85 Grow and graft new functionality Grow Graft Human! Knowledge GP Feature Sensitivity Analysis GP Host System Feature Test Test data Fitness data Fitness Non-functional Non-functional property Test harness Mark Harman, Yue Jia and Bill Langdon, property Test harness Babel Pidgin: SBSE can grow and graft entirely new functionality into a real world system Symposium on Search-Based Software Engineering SSBSE (Challenge track) Challenge Track! Award

86 AUTOMATED TESTING IS MATURING 68 Benchmarks Statistical Significance Top 1000 GooglePlay Apps COVERAGE Sapienz Dynodroid Monkey 53% 44% 48% FAULTS Sapienz Dynodroid Monkey bugs LENGTH Sapienz Dynodroid Monkey , /27 confirmed bugs Sapienz Ke Mao - Automated Mobile Testing: Dumb Monkeys, Smart Monkeys and Sapienz