ANZTB 2010 Conference. Tuesday, 2 nd March 2010 Hybrid Keyword Data Driven. Frameworks by Jonathon Wright. Introduction ANZTB

Transcription

1 Hybrid Keyword Data Driven Automation Frameworks An Insight to Approaches, Techniques & Methodologies. Jonathon Lee Wright Introduction Hybrid Keyword Data Driven Automation Frameworks What does this mean to you now? Frameworks (Slide 1) ANZTB

2 Introduction Presentation Plan Background Traditional Approach ( Frameworks) Automation: Timeline Reasons for Past Failure Where to Start Framework: Advantages Keyword Driven Automation: Failure Rate Understanding Failure GAP Analysis Which Test Tool Data Driven Break Even Point Initial Investment One Dedicated Team Modularity Driven Break Even Point Business Process Testing Summary Frameworks (Slide 2) Introduction Buzzword HYBRID Frameworks (Slide 3) ANZTB

3 Introduction Moving swiftly past the hype Historically Automation is perceived as a Silver Bullet of the Testing world. The term has been adopted into a general metaphor, where "silver bullet" refers to any straight forward solution perceived to have extreme effectiveness.. The phrase typically appears with an expectation that some new technology or practice will easily cure a major prevailing problem Frameworks (Slide 4) Background Timeline: Product Test Automation Historical trends in automation frameworks: Frameworks (Slide 5) ANZTB

4 Background Learning from past experience 92% FAIL TO MEET TARGET ROI (Estimated) Industry: Test Automation (net worth) $1 billion $6.3 billion Automation Projects (failure cost) $0.6 billion $3.8 billion Frameworks (Slide 6) Source A: ROI on Test Automation - Source B: HP pay 4.5billion to acquire mercury - Traditional Approach Reasons for past failure WHY DO AUTOMATION PROJECTS TYPICALLY FAIL? Lack of defined automation methodology Automation is not treated as a legitimate project with the necessary planning / resources Frameworks (Slide 7) ATLM Diagram: Automated Software Testing (Dustin Rashka Paul) - ISBN Source: Maximizing ROI and Avoiding the Pitfalls of Test Automation (Bill Hayduk - RTTS) - ANZTB

5 Traditional Approach Reasons for past failure Test Automation is typically performed at the end of the development cycle User Requirements System Requirements Global Design Hybrid - Test Automation Business Process Modelling (BPM) - Mapping Business Process Business Process Testing (BPT) - Reusable Modules (Jigsaw Pieces) Descriptive Programming (DP) - Enables Dynamic Object Repository Detailed Design Traditional - Test Automation Implementation Functional Test Tool Dependences: Application Under Test (AUT) Delivery - Learn Object Repository Testing Frameworks (Slide 8) Traditional Approach Reasons for past failure After the initial success the automation scripts are not maintained for future builds (contributing to a high failure rate of the remaining 32% of working automation projects) Testers are typically untrained in test tools and programming techniques No modularization (reusable functions) ) in automation scripts Automated tests cases are usually designed based on front end functionality (black box testing) Frameworks (Slide 9) ANZTB

6 Traditional Approach Putting the pieces of the puzzle together Black-Box Box Activity Diagram Registered Forgot Password Reset Password Registration Real World Example Simple Screen Register LOGIN Reset Password Frameworks (Slide 10) Traditional Approach Understanding why automation fails The Mine Sweeper Effect Automated testing repeats the same route over and over. Initially any defects that are found are whilst developing the scripts (i.e. manually) Subsequent executions will have little chance of ever finding further defects unless the flow changes Frameworks (Slide 11) ANZTB

7 Traditional Approach Record Replay is not your friend Linear process mapping (AUT) Frameworks (Slide 12) Video Source: YouTube - Bugatti Veyron Lake Crash - Traditional Approach Breaking Even Point Test Preparation (V) Execution (D) Manual Automated Manual ROI Resource (R) for (n) Automated Tests Automated Scenario % 77% 50% 32% Scenario % 77% 50% 32% Scenario % 86% 57% 37% Scenario % 86% 57% 37% Scenario % 81% 54% 35% Scenario % 81% 54% 35% Scenario % 64% 42% 27% Scenario % 42% 27% 19% Scenario % 51% 33% 22% Scenario % 73% 48% 31% Totals 5 hrs 10 hrs 140 mins 14 mins % 72% 47% 31% Rn = Aa / Am = (Va + n*da) / (Vm + n*dm) Frameworks (Slide 13) Equation Source: Automated Software Testing (Dustin Rashka Paul) - ISBN ANZTB

8 Introducing ( Frameworks) Automation: Timeline Reasons for Past Failure Where to Start Framework: Advantages Keyword Driven Automation: Failure Rate Understanding Failure GAP Analysis Which Test Tool Data Driven Break Even Point Initial Investment One Dedicated Team Modularity Driven Break Even Point Business Process Testing Summary Frameworks (Slide 14) Where to start? Start SMALL think BIG Quick wins should be avoided NEVER expect to automate 100% First find out What needs to be tested? What can be tested? What could be tested? Under promise, Over deliver? Keep it simple, wherever possible Then you can work out What needs be automated? What can be automated? What could be automated? Focus on key critical business processes Remember to think of each module as a individual Jigsaw piece but keep in mind what is needed to complete the entire puzzle Frameworks (Slide 15) ANZTB

9 GAP Analysis Understanding the full extent of Application Under Test (AUT): GAP analysis = complete application functionality actual Business Risk Reusability Complexity Usage NOTE : The GAP analysis file can be used to generate the parameter file which in turn is used to generate the dynamic object repository. Feasibility Frameworks (Slide 16) Investing the time Allocate realistic time frames & resources to achieve the initial framework: Framework 20hrs (prep) 14mins (execution) ROI = 2.8 Framework = planning + design + development + testing Traditional 10hrs (prep) 14mins (execution) ROI = 3.2 Manual 5hrs (prep) 140mins (execution) N/A leverage leverage initial investment based on sample set of 10 scenarios Frameworks (Slide 17) ANZTB

10 Breaking Even Point Test Preparation (V) Execution (D) Manual Automated Manual ROI Resource (R) for (n) Automated Tests Automated Scenario % 34% 16% 11% Scenario % 34% 16% 11% Scenario % 38% 17% 11% Scenario % 38% 17% 11% Scenario % 36% 16% 11% Scenario % 36% 16% 11% Scenario % 30% 15% 10% Scenario % 21% 12% 10% Scenario % 25% 13% 10% Scenario % 33% 16% 11% Totals 5 hrs 20 hrs 140 mins 14 mins % 33% 16% 11% Rn = Aa / Am = (Va a / n + n*da) / (Vm + n*dm) Frameworks (Slide 18) Key Advantages Maintainability significantly reduces the test maintenance effort Reusability due to modularity of test cases and library functions Manageability - effective test design, execution, and traceability Accessibility to design, develop & modify tests whilst executing Availability scheduled execution can run unattended on a 24/7 basis Reliability due to advanced error handling and scenario recovery Flexibility framework independent of AUT or environment Measurability customisable reporting of test results ensure quality Frameworks (Slide 19) ANZTB

11 Which? Which Automation Tool Suits You? Approach BPT/BPM Keyword Data Modularity Example Functional Test Tool QuickTest Professional Test Management Tool Quality Center (BPTEE, BPM) Standalone IDE Tool VBS (Windows Scripting Host) Combination of all the above Hybrid Solution* * optimised solution with preferred vendors Frameworks (Slide 20) One dedicated Team BA SA TA SME AA Business Analysts (Functional Specs) (Change Requests) System Analysts (Business Process Modelling) (Requests for Change) REQUIREMENTS TEST EXECUTION Test Analysts Subject Matter Experts - Domain Experts (BA/SA) - Developers Automation Architects Framework Automation Experts SOLUTION Test Analysts TA Subject Matter Experts - Domain Experts (BA/SA) - Developers Automation Architects Framework Automation Experts TEST PLANNING TEST REPORTING Test Managers Project Managers (Dashboard) SME AA TM PM Frameworks (Slide 21) ANZTB

12 Subject Matter Experts (SME) Business Process Testing (BPT) Framework Automation Experts Collaborative Solution Step 1 Export Core Logic Step 2 Design Test Case(s) Input Field Values Screen Format Username 64 Characters Text Password 12 Characters Hidden Button Visible Step 3 Code Generation System Requirements Specification (SRS) Functional specifications document (FSD) Software design document (SDD) Detailed design document (DDD) Frameworks (Slide 22) Subject Matter Experts (SME) design tests using documentation to map logical data flow and process modelling Framework Automation Experts transform designed tests into automation components ready for keyword stage Keyword Driven Subject Matter Experts (SME) Step 4 Create Test Case(s) Action Object Event(Type) Activate Window Set Username Editbox Set Password Editbox Press Button Subject Matter Experts (SME) create test cases using business level keywords stored within the MasterDriver file containing all the test scenarios relating to the AUT Generic Reusable Libraries Actions Object Handler MasterDriver File Event Handler Parameters File Driver & Data Repository The Hybrid Keyword Data Driven Automation Framework processes each action/object/event keyword/component in turn Example Runtime Process Log.Framework ObjectRepository.CreateObject.Window ObjectRepository.CreateObject.Button ClearAlerts Log.Events Sync.WinWaitObject.Button Action.Preamble.Check.ButtonState Event.PushDynamic.Button Action.Postamble.Check.ButtonState Action.Reaction.Check Popup.Handler Error.Recovery (retry/skip/baseline) Log.Results Log.Reports Actions Object Repository The framework calls the necessary functions to process each keyword / component and dynamically creates the object repository at run-time Frameworks (Slide 23) ANZTB

13 Data Driven Subject Matter Experts (SME) Input Data Sources SQL Query ODBC DAO/ADO LDAP/ERP(SAP) XML Spreadsheet Static/Flat File DB Query XML Spreadsheet Static/Flat File DB Query Step 5 Create Test Data DB Query Unique Random Test Object Expected Result Test Data Source Username Pass DB Query Password Pass Unique Flag Fail Negative Expected Result = FAIL False Positives Parameter Range Testing Minimum Values Maximum Values Regression Pack Change Detection Mode Comparison Subject Matter Experts (SME) create test data within the MasterDriver file from a number of possible input data sources Negative Min Max Baseline Frameworks (Slide 24) Examples of working smarter Introduction of new functionality Start Process Activate Window Set Username Editbox Set Password Editbox Press button [NEW] Terms & Condition Screen Expected Window Not Found RFC Request for Change Step 1 Export Core Logic Step 2 Design Test Case Code Step 3 Generation Step 4 Create Test Case Step 5 Create Test Data Ready [NEW] T&C Module Change to current functionality Start Process Activate Window Set Username Editbox Set Password Editbox Button Renamed to GO Unable to press button Register Reset Password Terms & Conditions CR Change Request Module Load login module make single change to Parameter File Ready Rename to Go Frameworks (Slide 25) ANZTB

14 Guidelines behind working smarter Centralized - creation of the centralized master driver file to provide stable, repeatable keyword / data driven test scenarios Dynamic - runtime generation of the object repository from the parameter file utilizing descriptive programming techniques Legacy support - for manual execution & AUT roll back Version controlled - code, driver files & object repositories Unified reporting - supporting real-time test execution & generic test results output (.xml) to analyze quality of builds Generic reusable modules / libraries - provide reliable processing of preamble and postamble actions Frameworks (Slide 26) Modularity: Generic Framework Architecture Frameworks (Slide 27) ANZTB

15 SUMMARY Hybrid utilising the best technologies and resources to do the job Keyword creating simple & robust test cases written using business level keywords combined with natural language Data effective use of business data to provide as an input source Driven processes component actions, objects & events seamlessly Automation that is collaborative, distributed and scalable Frameworks independent of application or environment under test Frameworks (Slide 28) Hybrid Keyword Data Driven Automation Frameworks What does it mean to you now? Thanks for your attention! ANZTB