Test Automation. Fundamentals. Mikó Szilárd

Size: px

Start display at page:

Download "Test Automation. Fundamentals. Mikó Szilárd"

Angela Watts
6 years ago
Views:

1 Test Automation Fundamentals Mikó Szilárd 2016

2 EPAM 2

3 Blue-chip clients rely on EPAM 3

4 SCHEDULE 9.12 Intro 9.19 Unit testing Unit testing Continuous integration Continuous integration Selenium WebDriver Selenium WebDriver Selenium WebDriver Selenium WebDriver Mobile Testing Mobile Testing Mobile Testing 3 Monday 12:15 4 homeworks 4

5 Agenda 1 Introduction 2 Approaches 3 Tools and frameworks 4 Test Execution 5

6 Introduction 1 History 2 Motives 3 Advantages 4 Best Practices 5 Expectations and Reality 6 Two disciplines 6

7 Feedback Control Mechanism 7

8 40s Concept of Automation 8

9 Future of Automation 9

10 Million Motives Source lines of code history Car Operating system 10

11 Motives RISK 11

12 Motives 12

13 Repeatable & schedulable Reliable Fast test execution Increased functional coverage Automatic reporting 13

14 Best Practices Automation is recommended for Static and Repetitive testing Data driven testing High complexity testing Load and Performance testing Smoke testing Regression testing 14

15 Best Practices Automation is NOT recommended for One-time, ad-hoc or exploratory testing Usability or look-and-feel testing Testing a feature under development Testing 3rd party services Cases where time and cost investment is too high 15

time savings Automate everything Keep coverage and quality Every

16 Expectations and Reality Automate at once Automate gradually Cut testing budget Increase test coverage Reduce testing time Reinvest time savings Automate everything Keep coverage and quality Every team can use automation tools Skills and training needed for success 16

17 Two disciplines Test and automation are two different disciplines Test Automation Application expertise Development expertise What to test How to automate Test cases Test code 17

18 Approaches 1 Linear scripting Record and Playback 2 Structured scripting - Library Architecture 3 Data Driven Testing 4 Keyword/Table Driven Testing 5 Hybrid Testing 6 Test Driven Development 7 Acceptance Test Driven Development 8 Behavior Driven Development 18

19 Linear scripting Record and Playback Manual recording of each test step Playing recorded test in the future Two types of verification Differences reported as errors Insert verification checkpoints Automatic record of all responses Simplest approach 19

20 Linear scripting Record and Playback Advantages Little training and setup time Technical expertise not mandatory Develop new tests on the fly Able to capture steps for Later diagnosis Bug reproduction Could help understanding test scripting 20

21 Linear scripting Record and Playback Disadvantages Manual capture is unavoidable Hard or impossible to maintain Test data is hard coded Higher chance of redundancy Higher chance of false positives 21

22 Linear scripting Record and Playback Improvements For windowed applications Use fixed screen format Verify only a selected area Ensuring the same input data Use a database backup Use stubbed data Use variables to store dynamic data 22

23 Structured scripting - Library Architecture Create an abstraction layer for testing Use control structures (conditionals, loops) Each module or function has its own test Create a common library from reusable parts scripts Build test cases using test functions 23

24 Structured scripting - Library Architecture Advantages Scalable High level of code reuse Easy to adapt functionality changes Lower maintenance cost 24

25 Structured scripting - Library Architecture Disadvantages Test data is hard coded Technical expertise required Longer planning time Longer test script preparation Deeper knowledge of the application 25

26 Data Driven Testing Test data shouldn t be hard coded Divide test data and test code Storage options Database, XML, Excel or CSV file Read inputs and expected outputs as key-value pairs Run your tests against multiple sets of data 26

27 Data Driven Testing Advantages Early test data creation is possible Script changes doesn t affect test data One script can run with multiple sets of data Manipulating test data could mean Adding, removing or modifying test cases Lower maintenance costs 27

28 Data Driven Testing Disadvantages Technical expertise required Longer planning time Longer test script preparation Test data should be maintained carefully 28

29 Keyword/Table Driven Testing Store test data and actions separately from code Create keywords for actions Create keyword parameters from test data Create test cases with a series of keywords 29

30 Keyword/Table Driven Testing Main process stages Test preparation Defining test basis Test design Analyzing test basis Test case design Test data design Manual execution Automating test execution Implementing keyword actions Automated execution 30

31 Keyword/Table Driven Testing Advantages Early test design creation is possible High level of code reuse Independence from test tools Readable test cases for stakeholders Testing work can be distributed Test design needs application expertise Implementation needs automation expertise 31

32 Keyword/Table Driven Testing Disadvantages Very high initial investment High automation expertise required Creating and maintaining the framework Not feasible for smaller projects 32

33 Hybrid Testing Combination of other approaches Over time it s a natural evolution step Structured data driven testing Structured keyword driven testing Aims to have all their advantages Has most of their disadvantages Initial investment and needed expertise are high 33

34 If it's not worth testing, why are you wasting your time working on it? - Scott Ambler 34

Test Driven Development Introduced a new evolutionary approach Test first development Refactoring Break down requirements to a test case Improve on

35 Test Driven Development Introduced a new evolutionary approach Test first development Refactoring Break down requirements to a test case Improve on software only that much to pass the new test Refactor design flaws and rerun all tests to catch regression When all test passed you can start it over 35

36 Test Driven Development 36

37 Test Driven Development Focuses on developers Developers doing TDD need Strong understanding of requirements Knowledge at least in unit testing Good unit test characteristics are Runs fast Runs in isolation Easy to read and understand Uses real data Represents one step towards your overall goal 37

requirements and developer tests Helping to communicate and understand customer

38 Acceptance Test Driven Development Extension of test driven development Acceptance tests drive the development process Creating a layer between requirements and developer tests Helping to communicate and understand customer needs Through them all stakeholders could be involved Customers, Business, Testers, Developers 38

39 Instead of What should I test?, you might hear What s the intended behavior?. - Neel Lakshminarayan 39

common language between stakeholders Through the use of a simple

40 Behavior Driven Development Improvement of acceptance test driven development Business goals drive the development process Enables a common language between stakeholders Through the use of a simple domain-specific language Domain specific phrases in English sentences 40

41 Behavior Driven Development Acceptance tests should be written as a user story As a [role] I want [feature] so that [benefit] Acceptance criteria should be written as scenarios Given [initial context] When [event occurs] Then [ensure some outcomes] 41

42 Behavior Driven Development User story example Title: Customer withdraws cash As a customer, I want to withdraw cash from an ATM, so that I don t have to wait in line at the bank. Scenario example Scenario: Account is in credit Given the account is in credit And the card is valid And the dispenser contains cash When the customer requests cash Then ensure the account is debited And ensure the card is returned And ensure cash is dispensed 42

43 Tools and Frameworks Test automation tools can help in automating such task as Product installation Test data creation API interaction GUI interaction Test execution Problem detection Defect logging They serve as a driving agent for the automation process 43

Frameworks put automation approaches into practice

44 An automation framework is not a tool which performs a specific task It is an infrastructure that provides Frameworks put automation approaches into practice Building a framework is a a unified solution strategic decision 44

45 Return on Investment A profitability measure that evaluates the performance of a business by dividing net profit by investment. ROI = Net profit Investment = Gross profit Investment Investment Good metric to measure return of money per period Could be used as an indicator to compare different alternatives However it is risky since a pre calculation is based on assumptions 45

46 Test Execution 1 Test Log 2 Error Log 3 Analyzing Results 4 Reporting 5 Metrics 46

47 Analyzing Results Classify false positives Test environment Application changes Test error Avoid reporting the same failure Keep on eye on false negatives Test execution finishes surprisingly fast Verified message appears in a different place 47

48 Metrics 48

49 Questions 49

Test Automation Practice STC 2012

Test Automation Practice STC 2012 Test Automation Practice in Agile Projects STC 2012 www.harbinger-systems.com Abstract Principles of agile Challenges in Test Automation What we should and shouldn t automate Test Automation Frameworks