Experiences with Action Based Testing a Case Study. Flavio Kubagawa, QAD Inc Hans Buwalda, LogiGear Corporation

Experiences with Action Based Testing a Case Study Flavio Kubagawa, QAD Inc Hans Buwalda, LogiGear Corporation

Action Based Testing (ABT) Test Planning and Control Global Test Design Breakdown Analysis Clustering "Test Clusters" Navigation Scheme Physical Navigation Report End users Management QA/Auditors System Development Input data Expected outcomes Documentation SEPARATION System(s) Under Test Actual results Comparison with expectations Management information (c) 2002, QAD, LogiGear, all rights reserved 3

Test Lines cluster EXAMPLE OF A TEST CLUSTER version 1.0 author Hans Buwalda section 1. Entering clients and balances last name first name account nr balance enter client Green John 458473948 1500 enter client Wood Anna 422087596 2100 section 2. Money Transfers from to sum transfer 458473948 422087596 500 transfer 422087596 785793025 1201 section 3. Checking names and numbers account nr last name first name check name 458473948 Green John check name 422087596 Wood Anna account nr sum check balance 458473948 1000 check balance 422087596 1399 (c) 2002, QAD, LogiGear, all rights reserved 4

Test Objectives... TO-3.51 The exit date must be after the entry date... test objective TO-3.51 name entry date exit date enter employment Bill Goodfellow 2002-10-02 2002-10-01 check error message The exit date must be after the entry date. (c) 2002, QAD, LogiGear, all rights reserved 5

Navigation Scheme CONTROL Reports High-level High-level High-level Intermediate Level(s) Low Level Action Layer Intermediate Level(s) Templates Tables Scripts Application (c) 2002, QAD, LogiGear, all rights reserved 7

About QAD Producer of advanced enterprise software for manufacturers "collaborative commerce" concept, focusing on integration between the diverse business processes Manufacturers of automotive, food and beverage, consumer, electronics, industrial and medical products use QAD applications 5,200 licensed sites in more than 80 countries and in as many as 26 languages for more information, please see: www.qad.com (c) 2002, QAD, LogiGear, all rights reserved 8

About MFG/PRO Enterprise software Menu with over 2,700 screens ("programs") For the test automation project, used version with a textual interface using a terminal emulator there is also a newer, web based, version, but the text version had priority for the test automation (c) 2002, QAD, LogiGear, all rights reserved 9

Project Objectives Create an automated regression test that supports: Integration testing against MFG/PRO baseline for: "Projects staged" "Maintenance fixes staged" Service pack testing Possible localization testing Absorbs project changes Represents long term payoff for QAD (c) 2002, QAD, LogiGear, all rights reserved 10

Criteria for Automation Priorities First priorities: Modules most impacted by new projects Legacy maintenance against baseline Prioritize next modules with highest defect-density Based on previous release Remain responsive Be flexible enough to reorder priorities as needed (c) 2002, QAD, LogiGear, all rights reserved 11

Action Based Testing The navigation scheme is basically an interpreter Core is the "ABT Engine", with a library of functions that is loaded from a testing tool Reads the test lines from the Excel file Calls functions created by the user to execute the actions and checks Processes the results into easy to read reports Test step >> action line Test script >> set of action lines >> cluster MFG/PRO interface >> interface layer (c) 2002, QAD, LogiGear, all rights reserved 12

Test Automation: Development Process Test case descriptions are created and streamlined from system test scripts by a test specialist (Robin Kronk) 700 pages = 1,600 test steps On basis of descriptions ABT test clusters are made and the actions are implemented with an ABT navigation scheme plus generic Winrunner functions by two test engineers (Flavio Kubagawa and Edward Duran) 600 Action Words, 65 scripts in Winrunner Tools used to support test automation: Winrunner ABT Engine MS Excel (c) 2002, QAD, LogiGear, all rights reserved 13

Test Automation: Challenges MFG/PRO ChUI (Character User Interface) interface Created WinRunner functions to work with ChUI Maximize reusability Developed methodology to create generic action words to increase reusability Minimize need for additional scripts Created generic functions to decrease need of coding (c) 2002, QAD, LogiGear, all rights reserved 15

Test Automation: Generic Functions Layer Implements the middle layers of the ABT Navigation Scheme (without the use of templates) Virtually eliminates the need to create new WinRunner scripts/functions to implement a new action word Speeds up the process of automating test scripts. Eliminates need for large number of automation engineers (c) 2002, QAD, LogiGear, all rights reserved 16

cluster Manufacturing sheet scenarios version 1.0 author date Jun/2002... section Backlog Flavio Y. Test KubagawaCluster Enter actuals that are lower than the plan. 20.21 Backlog Plan Maintenance Site Product Line Year create backlog plan 10000 1000 D>Year+1 January February March April >>> 10 10 15 15 May June July August >>> 25 25 35 40 September October November December >>> 55 65 80 100 (c) 2002, QAD, LogiGear, all rights reserved 23

Example Interface Definition create interface mfg_awinfo AW prg menu title frames interface entity create actual pay rate rwemmt 14.13.21 Actual Pay Rate Maintenan 01,02,03 interface entity browse approval pcbr029 1.9.6.2 Approval Browse 01,Filter,02 interface entity create backlog plan plppmt05 20.21 Backlog Plan Maintenance 01,02 interface entity inquire backlog plan plppiq05 20.22 Backlog Plan Inquiry 01,PRT,SB interface entity create bom registration wlbmmt 3.22.13.4 BOM Registration Maintena01,02,03,0 interface entity build calendar cross-reference spglclmt 33.1.4 Calendar Cross-Reference B01 interface entity create commodity code ppcommt 1.4.19 Commodity Code Maintena 01,02,END interface entity inquire co/by-product work ordwojpiq 16.3.13 Co/By-Product Work Order 01,PRT,SB interface entity initialize compliance clclpm 1.22.24 Compliance Control 01,02 interface entity copy plan to simulation spfpcp01 33.17.1 Copy Plan to Simulation 01,PRT,SB interface entity copy simulation to plan spfpcp03 33.17.3 Copy Simulation to Plan 01,PRT,SB interface entity cost roll-up freeze/unfreeze bmfrzmt 13.12.1 Cost Roll-Up Freeze/Unfree01,02,03,P save interface mfg_awinfo c:\actionwords\data\mfg_awinfo.dat (c) 2002, QAD, LogiGear, all rights reserved 25

create interface plppmt05 interface entity interface entity setting title fields_plppmt05 Interface Layer 20.21 Backlog Plan Maintenance name class nbr init endkey condition value interface element frame01 frame 01 3,9 name class tab row col len frame interface element Site edit 0 3 9 8 1 interface element Product Line edit 1 3 32 4 1 interface element Year edit 2 3 71-74 4 1> name class nbr init endkey condition value interface element frame02 frame 02 7,17-23 name class tab row col len frame interface element Begin edit 0 7 17-23 9 2 interface element January edit 1 8 31-37 9 2 interface element February edit 2 9 31-37 9 2 interface element March edit 3 10 31-37 9 2 interface element April edit 4 11 31-37 9 2 interface element May edit 5 12 31-37 9 2 interface element June edit 6 13 31-37 9 2 interface element July edit 7 14 31-37 9 2 interface element August edit 8 15 31-37 9 2 interface element September edit 9 16 31-37 9 2 interface element October edit 10 17 31-37 9 2 interface element November edit 11 18 31-37 9 2 interface element December edit 12 19 31-37 9 2> save interface plppmt05 C:\ActionWords\data\fields_plppmt05.dat (c) 2002, QAD, LogiGear, all rights reserved 27

Very Good Reusability Interface layer and action words: one size fits all Once created, can be reused as often as needed. Clusters: as reusable as the related manual test. New clusters can be created reusing existing action words Different business cycles. (c) 2002, QAD, LogiGear, all rights reserved 29

Reusability, Some Numbers Top 5% most used Action Words Suite 1 = 27% of Total Steps: 7 Action Words = 86 Steps Suite 2 = 27% of Total Steps: 10 Action Words = 113 Steps Top 10% most used Action Words Suite 1 = 39% of Total Steps: 14 Action Words = 123 Steps Suite 2 = 39% of Total Steps: 19 Action Words = 168 Steps 50% of the Test Suites were completed with: Suite 1: 23 Action Words (17%) = 159 Steps (50%) Suite 2: 30 Action Words (16%) = 214 Steps (50%) (c) 2002, QAD, LogiGear, all rights reserved 30

Reusability Example Suite 1 Incorporating Project Enhancements added 2 new sections to the manual tests 22 pages 44 steps 12 new Action Words Reused 73% of existing Action Words Quickly implemented new Action Words using the layered structure (c) 2002, QAD, LogiGear, all rights reserved 31

Overall Experiences ABT process is a major improvement maintainability re-usability manageability (clear tests and test results) LogiGear's support is very effective training and coaching quick and appropriate answers to all questions QAD expects to extend the use of ABT to other systems and other tests (c) 2002, QAD, LogiGear, all rights reserved 32