Automated Acceptance Testing

Transcription

1 Automated Acceptance Testing Björn Beskow Callista Enterprise AB CADEC , Automated Acceptance Testing, Slide 1

2 Target audience and Objectives Target audience Developers, Designers, Architects, Project Managers and Project Sponsors interested in lean and mean ways to achieve good-enough quality without paying an excessive price Objectives Provide an overview of some Acceptance Testing strategies, tools and frameworks Non-Objectives To cover Unit Testing, Performance Testing, Stress Testing, CADEC , Automated Acceptance Testing, Slide 2

3 Agenda Background: The case for automated tests Classification of automated tests Tools and Frameworks for Acceptance Tests Data-driven Tests CADEC , Automated Acceptance Testing, Slide 3

4 Testing - a critical success factor for agility Enables true iterativity Enables relentless refactoring Shows expectations and elicits feedback Demonstrates visible progress CADEC , Automated Acceptance Testing, Slide 4

5 Quality Assurance precedes Quality Assessment Testing is about Quality Assurance, not just Quality Assessment Quality Assessment only indirectly affect quality Testing reveals information Testing helps focus project activity CADEC , Automated Acceptance Testing, Slide 5

6 Test Automation Goals Tests should be S.M.A.R.T: Self Checking Maintainable Act as documentation Repeatable and Robust To the point provide defect triangulation CADEC , Automated Acceptance Testing, Slide 6

7 Manual Tests are Repetitive Error-prone Difficult to test other units than the User Interface yet a Manual Test Process must be present in order to automate it! CADEC , Automated Acceptance Testing, Slide 7

8 Automated Tests must be easy to write easy to find easy to run easy to maintain otherwise they will slow you down they will get left behind you ll go back to manual testing CADEC , Automated Acceptance Testing, Slide 8

9 Economy of Tests The number of possible defects is infinite Time and resources are usually finite Hence every test strategy involves a trade off! Don t automate the overly complex cases (remember the rule) Only automate tests that are going to be repeated Choose a methodology which maximizes both ease of writing and maintainability CADEC , Automated Acceptance Testing, Slide 9

10 Some Terminology Unit tests Integration tests System tests Acceptance tests Functional tests Black-box tests White-box tests CADEC , Automated Acceptance Testing, Slide 10

11 Classifying Automated Tests Granularity Entire system Subsystems Unit System R & PB Program Individual units Point of Contact API Existing User Interface Testability API GUI Test Case Production Record and Play Back Hand Written (programmatic) CADEC , Automated Acceptance Testing, Slide 11

12 Unit Tests Tests a logical unit in isolation Written and executed by developers as part of the development process Written in a programming language Once written, always must run 100% correct at all times before integrating any changes CADEC , Automated Acceptance Testing, Slide 12

13 Acceptance Tests Tests a system or part of a system from the outside, in terms of observable behavior Typically structured around a comprehensible chunk of system functionality Ideally written by the customer Either written in terms of User Interface actions, or in terms of words and facts Ideally run near 100% correct at the end of a release/project CADEC , Automated Acceptance Testing, Slide 13

14 Record/Playback A Test Automation tool records events that make up a Test Case into a Test Script Events can be User Interface interactions or API calls The scripts can be played back later for regression testing System Unit R & PB Program GUI API CADEC , Automated Acceptance Testing, Slide 14

15 Example: Astra QuickTest Unit System R & PB Program API GUI Astra QuickTest CADEC , Automated Acceptance Testing, Slide 15

16 Example: MaxQ System Unit R & PB Program API MaxQ GUI CADEC , Automated Acceptance Testing, Slide 16

17 Drawbacks of Record/Playback approaches Tests tend to be fragile Maintenance tends to be expensive Complex, expensive tools Tests cannot be pre-built CADEC , Automated Acceptance Testing, Slide 17

18 Hand-written, Programmatic Tests Pros: Tests can be made more robust Tests can be made more maintainable Simple, cheap tools and frameworks Can be built before production code Cons: Higher initial cost/effort More skills required Unit System API R & PB Program JUnit GUI CADEC , Automated Acceptance Testing, Slide 18

19 Leveraging JUnit beyond Unit Tests TestCase Command Template method TestResult Collecting Parameter TestSuite Composite CADEC , Automated Acceptance Testing, Slide 19

20 JUnit Add-ons for testing Web Applications Framework which emulates browser behavior form submission JavaScript execution http authentication cookies page redirections Allows Java test code to examine and validate returned pages jwebunit.sourceforge.net higher level abstraction on top of HttpUnit that facilitates creation of acceptance tests for web applications CADEC , Automated Acceptance Testing, Slide 20

21 jwebunit example public class TestStore extends WebTestCase { } public void testlogin() throws Exception { beginat("/store/faces/home.jsp"); clicklinkwithtext("log in"); setformelement("_id0:username", "test"); setformelement("_id0:edtpassword", "test"); submit(); asserttextpresent("welcome to Callista Store"); asserttextpresent("test"); } CADEC , Automated Acceptance Testing, Slide 21

22 JUnit Add-ons for database testing JUnit extension targeted for database-driven projects Can be used to put a database into a known state between test runs Enables assertions to verify that the database content match some expected values. Can export and import database content to and from XML datasets CADEC , Automated Acceptance Testing, Slide 22

23 dbunit example public class TestStore extends WebTestCase { public void setup () throws Exception { IDatabaseConnection connection = getconnection(); try { IDataSet dataset = new FlatXmlDataSet(new FileInputStream(xmlFile)); DatabaseOperation.CLEAN_INSERT.execute(connection, dataset); } finally { connection.close(); } gettestcontext().setbaseurl(" } } private IDatabaseConnection getconnection() {... } CADEC , Automated Acceptance Testing, Slide 23

24 Data-driven Acceptance Tests Tests written in terms of words (operations) and facts (data) that make sense to the customer Software modules adapt the words and facts to the underlying software system, interpret them and give them formal semantics CADEC , Automated Acceptance Testing, Slide 24

25 Framework for Integrated Test (FIT) Simplistic framework for data-driven tests Tests are viewed as documents containing tabular data Executing tests are viewed as annotating the test documents with test results FIT Runner Adapter CADEC , Automated Acceptance Testing, Slide 25

26 CADEC , Automated Acceptance Testing, Slide 26 FIT Example

27 Finally don t forget the people! Managing Resistance to Change The tool will replace the testers? The tool will be too expensive? The tool will be too difficult to use? QA team Development Team = true? Co-operative relationship Open to learning Management Buy-In Early Payback CADEC , Automated Acceptance Testing, Slide 27

28 Summary Establish clear and reasonable expectations as to what can and cannot be accomplished with automated testing Adopt a viable, cost-effective test strategy and methodology Select tools and frameworks that allows you to automate your tests in accordance with your test strategy Gain approval for the test strategy both within the project team and from the project sponsor CADEC , Automated Acceptance Testing, Slide 28

29 How can Callista help you? Seminars, Workshops and hands-on tutorials Initial tool smithing to establish a lightweight, efficient testing platform Architectural consulting to assure testability Technical management and architectural mentoring for Unit-Test and Acceptance Test patterns Project architects, test engineers and mentors to secure a successful testing strategy CADEC , Automated Acceptance Testing, Slide 29

30 Resources Unit Testing and Unit Testing Extensions jwebunit.sourceforge.net jfcunit.sourceforge.net GUI Testing MaxQ.tigris.org FIT fit.c2.com CADEC , Automated Acceptance Testing, Slide 30

31 CADEC , Automated Acceptance Testing, Slide 31 Questions