Breaking the mould in Automated Testing. TMF London: 28th of July 2010 Richard Roy, Michael Wagner

Size: px

Start display at page:

Download "Breaking the mould in Automated Testing. TMF London: 28th of July 2010 Richard Roy, Michael Wagner"

Leslie Lionel Thomas
5 years ago
Views:

1 Breaking the mould in Automated Testing TMF London: 28th of July 2010 Richard Roy, Michael Wagner

2 Just two of the Evils in Functional Test Automation Reliance on Scripting (In any guise) which is a Technical Discipline. An enduring need for Test Data that is meaningful, comprehensive and in the correct state. This often requires extensive technical work, forcing testers to be distracted from the functionality they wish to test. Copyright by TRICENTIS, all rights reserved 2

3 Scripting: Scripting itself developed as a result of the limited options available in the early approaches available to Automation of Functional Testing. Straight forward Capture and Replay, to quickly build Scripts, does not work! Now Frameworks themselves have/are being developed to overcome the challenges presented by Scripting (which is in effect a new development project in itself). Some of these frameworks can help to mask some of the negative impact of Scripts, especially at a user level, but they certainly cannot overcome them all at a technical level. It is the script itself which is the cause of inflexibility in testing this is true no matter how the script is dressed up. Scripting requires staff to focus on technical issues that have nothing to do with the quality of the application under test Thus scripting itself has become the overhead and the roadblock to test automation. The typical results of a need to Script are: Greater Risk Taking in a project High Complexity High requirements for technically skilled resources High requirements for allocations of Test Time Depressingly low Automation Levels Copyright by TRICENTIS, all rights reserved 3

4 How is the need for Scripting Overcome? Business driven Dynamic Steering Hypothesis: Stable test cases become reality if the technical steering is being generated from the business-view of an application at runtime and not vice versa. Copyright by TRICENTIS, all rights reserved 4

5 Business driven Dynamic Steering Cognitive Test Cases: business language - not scripts! Requires only business SME skills Highly stable from a technical as well as from a business point of view Independent from technologies, HTML, Java,.NET, SAP, Manual execution, Copyright by TRICENTIS, all rights reserved 5

Business driven Dynamic Steering A test case can

10 Advantages of Business driven Dynamic Steering Eradicates completely the complexity and inflexibility of scripts Clear, stable and reusable test cases. What technology work there is done by one (or two) for the many. Manual and automated tests run in same environment. (One click Automation when appropriate). Can be mixed as and where required. Complete coverage of environments (all handled the same clear way) (GUI, Messaging, DB). Simple creation, automation and management of test cases. Opens up a wealth of opportunity for new approaches to truly overcome some of the more stubborn and traditional challenges associated with Automation of Functional Testing. Copyright by TRICENTIS, all rights reserved 10

11 Test Data Most testing initiatives involve significant overheads to build, manage and maintain Test Data Data sometimes comes from multiple sources (silo s) and obtaining it can be difficult for many reasons. (political /technical) Without correct and available test data it is not possible to achieve true automation. Production data sources of test data are sensitive : Should be properly anonimised Restrictions on locations Highly contentious Legal implications Use of vast quantities of test data (often done to attempt to maximise code coverage) can be both resource expensive and time consuming. Achieving the right State for data is often a challenge (time and time again). Some tests can take DAYS to run. As the data changes across the test one fault can grind the entire test to a halt, meaning a complete restart of the test is required. Copyright by TRICENTIS, all rights reserved 11

12 Advantages of synthetic test data Synthetic test data Perfect accuracy Each test data object has a specific test purpose or is the result of a specific test process and covers an exact defined precondition. Production data Incomplete data material Restricted data material as regards content, data are derived only from the past! Hidden data material Finding adequate data is difficult at times impossible! High test coverage Low test coverage Low data volume High data volume Extractors are complex often complete extractions of production data Costs for maintenance of data basis are proportional to the data volume in Mainframe (keyword: MIPS consumption!) Low operating costs No anonymization required High operating costs Anonymization ex post required Copyright by TRICENTIS, all rights reserved 12

13 Two approaches to obtaining test data Synthetic test data Production data Perfect accuracy Each test data object has a specific test purpose or is the result of a specific test process. Low data volume Incomplete data material Restricted data material as regards content, data are derived only from the past! Hidden data material Finding adequate data is difficult at If synthetic test data have times indeed impossible! so many advantages why do all companies use High test coverage production data? Low test coverage High data volume Extractors are complex often complete extractions of production data Costs for maintenance of data basis are proportional to the data volume in Mainframe (keyword: MIPS consumption!) Low operating costs No anonymization required High operating costs Anonymization ex post required Copyright by TRICENTIS, all rights reserved 13

14 Difficulties - synthetic test data The following situations occur: Test executions consume or exhaust test data. Example: When a valid contract is canceled, it will be marked as canceled. This contract cannot be used again for further test executions. The tests require a (complex) process history in the test data objects. Example: To test a calculation of commission or charges of a checking account, specific booking histories over a potentially long time period for the respective checking account are required. The generation or adaptation of data objects requires non-gui Input and cannot be performed manually. Example: Account transactions cannot be entered via the interface of the SuT but only via WebSphere MQ data records. Copyright by TRICENTIS, all rights reserved 14

That is potentially a lot of data to handle Now we got potentially thousands of test data objects physically created in the data base of our SuT: e.g. Account holders, Insurants, Customers, Vendors, Contracts, Partners, Products.

17 That is potentially a lot of data to handle Now we got potentially thousands of test data objects physically created in the data base of our SuT: e.g. Account holders, Insurants, Customers, Vendors, Contracts, Partners, Products.. After that, other test cases have started batch jobs with the SuT that process or aggregate our test data objects: e.g. create the required booking history at a bank account e.g. create a certain history into a live-insurance contract e.g. create a sales transaction history with a vendor/partner e.g. create stock at a warehouse. Somebody or something needs to keep track about this. There must be a central unit that exactly knows about all test data objects: Quantity, Status. Otherwise we re lost again! Copyright by TRICENTIS, all rights reserved 17

18 However somebody needs to maintain control and that somebody is: TOR the Test Data Object Registry create register get use New register status changes TOR consume register consumption Copyright by TRICENTIS, all rights reserved 18

19 Project overview pilot project Project scope: Solution for the test data problem for a large bank Main objectives: Reduce test data volume to 3 per cent of current data volume while maintaining and/or increasing test coverage Anonimisation of production data Project landscape: Core banking system with partner management, accounts management, payment management and security management, Technologies: Siebel: CRM System (Partner Management and Account Management, Security Trading) Avaloq: (Security Management) 3270 Terminal Application: Payment Management WebSphere MQ: non-gui Interface for all involved applications Copyright by TRICENTIS, all rights reserved 19

21 Implemented solution Synthetic dynamic test data Massive usage of test automation (especially in the non-gui area to create the necessary test data objects) Copyright by TRICENTIS, all rights reserved 21

The Future of Testing: Continuous Enterprise Testing

The Future of Testing: Continuous Enterprise Testing ANZTB Test Conference, Canberra, 1 June 2018 Thomas Hadorn. Dev Ops Years Months Months Weeks Delivery Cycle Time Weeks Days Perceived Disruption Software