Using Models to Automate Testing: Experiences & Challenges

Transcription

1 Using Models to Automate Testing: Experiences & Challenges Dr. Muhammad Uzair Khan Senior Member IEEE Quest Lab, FAST-NU Islamabad Pakistan Software Quality Conference, PSQC 17 April 08, 2017

2 Assistant FAST NU About me! PhD from University of Nice, France and INRIA in 2011 Head QUEST Lab Founding Member, Pakistan Software Testing Board Treasurer and Secretary IEEE Islamabad Section Over 20 high quality international research publications Over 14 years of industry and academia experience Expertise: Software Testing, UML, Empirical Software Engineering, Model refactoring, Aspect Oriented Engineering Principle Investigator of ICT R&D Funded project AGCT 2017 (26 Million PKR) Co-Principle Investigator of ICT R&D Funded project MBT-Toolset (15 Million PKR) 2

3 Software Quality Engineering and Testing (QUEST) Lab Quest Lab, Est. June 2013, Industry-driven research on software quality engineering: testing, model-driven development Targeting real industrial problems Several national and international industry partners International academic collaborations: Interdisciplinary Centre for Security, Reliability and Trust (SnT) Luxembourg Simula Research Laboratory, Norway Several local and foreign funded projects Worth over Rs. 50 Million Research being published at high quality international venues Have offered 12+ fully funded MS scholarships 3

4 Applied Research in Software Engineering Useful research must have impact: Research-driven innovation Close industrial involvement Conscious effort to understand the problems and practice of industry Focus on pain points in industry Realistic Evaluation Long term industrial collaboration => impact Impact on Academia Should contribute to the learning of students Introduction of new courses/specializations Producing high quality graduates Producing high quality publications 4

5 Our Industry Collaboration Model Unlike other stereotyped academic research, at QUEST lab research is driven by practical problems faced by industry Strong industrial collaborations to develop and evaluate candidate solutions 5

6 Software Testing Testing is the process of executing a program with the intent of finding faults The challenge is to identify scenarios that trigger the faulty functionality of software

7 Major steps of Testing Identifying Test Paths/Sequences Identifying Test Data Test Execution Test Evaluation new Calculator(..).add( ) Windows, 2, 2 4 7

8 Identifying Test Sequences What sequences of function calls to test? Out of billions of possible function call sequences for a class, which call sequences to select? Important Decision A number of heuristics are typically used, e.g., important function sequences based on application logic data-flow sequences checking the use-case flows checking the misuse-cases

9 Identifying Test Data What test data to execute? Consider a program taking two integers as input Possible number of input combinations: 2 64 * 2 64 Which combinations to test? Decision is typically based on test adequacy criteria To maximize code s control flow coverage To maximize code s data flow coverage Test based on equivalence classes and boundaries

10 Computing Test Oracle How to obtain a test oracle (expected results)? Computing test oracle is challenging Simpler at unit testing level Expected outputs are based on test data Well documented specifications can help

11 Test Execution How to execute test cases? Manual execution laborious A number of frameworks support automated test execution Record and replay tools (e.g., for web applications) xunit framework tools

12 Automated Test Execution How to execute test cases without manual intervention? A number of frameworks support automated test execution Record and replay tools (e.g., for web applications) xunit framework tools Industry practice Large part of local industry struggling to achieve automated test execution.. 12

13 Automated Software Testing Industry Practice Problems? Human intensive effort, Not scalable, not repeatable, not maintainable Limited confidence in testing, How to evaluate effectiveness of testing? Typical solution: Testing is driven by availability and capability of the resource 13

14 Why not automated other steps? Useful Test data, sequences, and oracle are application specific Cannot be generalized Need to capture specification of the system under test in a machine analyzable form Domain/application specific methodologies and strategies for testing are required One widely accepted solution: Model-based testing Our expertise at QUEST lab Test Path/Sequence Test Data Test Oracle 14

15 Model-Based Testing Model-based Testing (MBT) uses models of the system to completely automate the testing process The models typically describe some aspects of system under test MBT allows automated testing solutions for complex industrial software systems Early Systematic Scalable 15

16 Test Case Generation and Execution What modeling strategy? SW Model Tailored Modeling Methodology What test data & sequence generation strategy? SW Code Test cases Application specific/ control Test and cases data flow analysis What oracle strategy? Oracle Expected Results Results Compare Application domain specific 16

17 Vending Machine Control Software - State Machine off on power-on/ light-on; m:=0 power-off/ light-off coffee idle coffee[m>0]/start;dec after(5)/stop busy money empty inc/m:=1 dec[m=1]/m:=0 inc/m:=m+1 not empty dec[m>1]/m:=m-1 17

18 State coverage off on power-on/ light-on; m:=0 power-off/ light-off coffee idle coffee[m>0]/start;dec after(5)/stop busy money empty inc/m:=1 dec[m=1]/m:=0 inc/m:=m+1 not empty dec[m>1]/m:=m-1 18

19 Applied Research in MBT MBT is not something new! Well-accepted by international industry IBM, Microsoft, Cisco, Schlumberger, etc Next, some of the industrial applications of MBT we at Quest lab have worked with 19

20 Automated Software Testing with Models 20

21 I - Video Game Testing Game Industry worth around $100 billion* Platform games a famous type of games Games are highly interactive, state-based, and are typically realtime software Highly competitive market * 21

22 Video Game Testing Current Practice Manual Testing A common approach of testing games in industry Testers play various game scenarios manually Preferably over long periods of time Observing various game scenarios Various game play rules need to be taken into account for testing Focus of functional testing: Functional correctness of games Identifying crash points 22

23 Video Game Testing - Problem Problem Lack of a scalable strategy to test functionality of platform games Issues with Current Practices Labor intensive: Manual testing by playing games for hours Non-systematic: Highly dependent on individual skills Boredom effect: Task gets non-rewarding over time Non-repeatable: playing the same game scenarios for every change in the game Error prone: A number of interesting scenarios are missed 23

24 Video Game Testing Automated Approach We developed an automated platform game testing approach Focuses on testing the game based on avatar s behavior by simulating the inputs required for playing games. Testers need to model the game play from users perspective We developed a detailed modeling methodology A UML profile for modeling platform games for testing A set of guidelines for modeling game concepts and behavior for testing purposes Automated Model-based Testing Approach for Testing Games, published in ACM/IEEE 18th International Conference on Model Driven Engineering Languages and Systems (MODELS 15), 2015, Canada. 24

25 Modeling Mario Brothers 25

26 Video Game Testing Models for Mario Bros Fig. 3. Mario s avatar state machine 26

27 Video Game Testing Test Sequences (the user game play events) are automatically obtained by traversing the models as a graph Test Data generated by solving constraints on guards Test Execution is automated by developing a simulator that can execute user events Test Oracle System-generated events State transitions in the state machines (due to a change, signal, or a time event) that are triggered as a result of the user-generated events System crashes Fig. 3. Mario s avatar state machine 27

28 A Sample Bug 28

29 II - Testing Real time Embedded Software Problem: RTES execute in complex environments A large number of possible environmental scenarios Practice: Manually writing simulators for SiL testing Manually system level test cases Bottle Recycling Machine Marine Seismic Acquisition System 29

30 Testing RTES - Solution Environment Models Independent Black-box Behavior driven by environment Environment model One model for Environment simulator Test cases and oracles Notation: Class Diagrams, State machines, OCL Environment Simulator Test cases Test oracle * *Environment Modeling and Simulation for Automated Testing of Soft Real-Time Embedded Software, in Software & Systems Modeling Journal (Sosym),

31 Domain Model - Example

32 Behavioral Model - Example 32

33 Automated RTES Testing Test Sequence Search algorithms guide the selection of test sequences that take the RTES closer to error states Test Data Test data generated based on search using constraint solvers Test Oracle Error states (states that should never happen) were added in the environment state machine Test Execution Simulator is automatically generated from the environment models 33

34 III - Mobile Application Generation & Testing Problem Mobile applications need to support a large number of platforms Very tedious task to develop and maintain numerous versions Manually evaluating non-functional properties across various platform takes significant effort 34

35 Mobile Application Generation & Testing Solution: Capture the core business logic with models A detailed modeling methodology specific to mobile applications UML Profile for modeling Mobile Applications UML Class Diagrams and State-machines Action Language: ALF Native application code generated for each platform Performance related properties are modeled and their conformance automatically tested *A product-line model-driven engineering approach for generating feature-based mobile applications, Journal of Systems and Software, vol. 123, January

36 Mobile Application Generation & Testing

37 Mobile Application Generation & Testing Android Application Code Windows Application Code

38 Mobile Application Modeling Performance 38

39 Mobile Application Modeling Performance 39

40 Mobile Application Performance Testing Test Sequences Search-based approaches and fuzzy logic are used to identify test sequence that stress the SUT to violate the performance properties Test Data Test data is generated to support the generated test sequences Test Oracle The performance properties modeled are considered as oracle Test Execution Tests are automatically executed across various platforms for the versions of the mobile applications 40

41 IV - Automotive System Testing A safety-critical automotive software developed Monitors a car seat via an electrical field sensor to classify the occupant disables the airbag for children and unoccupied seats ensures airbag deployment for adults Need to support ISO automotive standard for functional safety All requirements must be tested with test cases * Automatic Generation of System Test Cases from Use Case Specifications, in International Symposium on Software Testing and Analysis (ISSTA 2015), USA

42 Automotive System Testing Solution: Use case modeling with RUCM and OCL constraints RUCM: Restricted Use case Modeling Language Allows to specify scenario in a structured language Natural Language Processing to generate test cases OCL Constraints solving to generate test data

43 RUCM Use case

44 Derived OCL Constraints

45 Automotive System Testing - Automation Test Sequences Various scenarios of the use cases are automatically identified to form the test sequence Test Data The OCL constraints are automatically solved to generate scenarios to test the system Test Oracle The system actions of the use case are treated as oracle Test Execution Test scripts are automatically generated and executed on preexisting simulators 45

46 When to Automate? A large number of testing scenarios are automatable ROI? Large maintenance cycle Regression Testing Data driven Testing, e.g., web application Load and Performance Testing When the effort of manual testing is huge and redundant Systematic testing is a need Systems with complex nature and large environment variations Critical systems Learning curve should not hinder automation Need to start from somewhere 46

47 What to Automate? Everything that can be automated, should be automated? A large number of testing tools support automated test execution xunit framework tools: JUnit, GoogleTest, nunit, etc Record & replay tools: Selenium, Test Complete, QTP, Watir, etc Should be used to reduce rework and support systematic test execution and management Load testing is only practical when automated: LoadRunner, JMeter Repeated and labor intensive tasks should be automated, e.g., manual white-box testing Useful Test data, sequences, and oracle are application specific Given the ROI and nature of the project, investment should be made in exploring such strategies 47

48 Open Challenges How to automate usability testing? Test sequence, data and oracle generation approaches are tailored to the application needs Tool support for Test Data Generation is improving EvoSuite, SBOS OCL Solver Test oracle in most cases is the difficult part Model-based testing approaches are promising Conformiq, Smartesting Still a huge potential for improvement Regression Testing approaches have room for a lot of improvement 48

49 Future of Software Testing Current Software Testing Industry: Over $60 Billion 10% growth every year - a few of the industries having a double figure growth rate for last decade Software Testing Job ranked among the top 10 In-demand jobs of 2016 (fortune.com) The role of software tester is changing More development, Developers are Testers, Test Driven Development and DevOps, demonstrable skillset to be rewarded Cloud and IoT increasing the importance of software testing Tools gaining momentum Automated testing for tedious and repetitive tasks - A must A lot of research prospective! 49

50 Thanks! Dr. Muhammd Uzair Khan Quest Lab, FAST house, Islamabad 50