Implementing an SQA Automation Strategy Using the Automature Framework, a Case Study

Size: px

Start display at page:

Download "Implementing an SQA Automation Strategy Using the Automature Framework, a Case Study"

Jesse Lindsey
5 years ago
Views:

1 Session 14E Implementing an SQA Automation Strategy Using the Automature Framework, a Case Study Steven A. Spura ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

2 The Problem Custom In-house Software Developed over decades No clear unifying architecture IT IS: hard to maintain hard to test due to poorly thought out and implemented interdependencies. 2 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

3 The Problem The Company Runs on It The company s life blood Unable to just replace it wholesale No specs Lots of customized workflows (both in code and in tribal knowledge) No clear unifying architecture TESTING (and characterizing application) was never a priority. No instrumentation in place No existing tests to leverage 3 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

4 The Solution The Goal Produce reliable and stable software product as efficiently as possible. Properly implemented automated testing improves reliability by exercising and characterizing it more extensively than the limitations of manual testing would allow. Making releases less risky and more efficient. 4 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

5 Application Software Re-build the Application using carefully crafted, modern architecture principals upgrade of current system with independent and isolated autonomous components working together Strive for Continuous Integration Create APIs in the parts that can be tested 5 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

6 Automated Testing Initiative Test instrumentation in the code Developers involved with creating and designing tests Test at several different layers of application code. Checks and double checks of design adherence Available to all engineering users 6 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

7 The Challenge of Introducing Automated Testing Top Down Sufficient external support Management support Head Count Financial Investment Strategic alignment Bottom Up Sufficient internal support Engineering Team support Test Design Patterns congruent with Core Code 7 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

8 ROI and What to Test 8 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

9 ROI and Testing A case for considering ROI before testing Much has been said about testing ROI, I m not going to rehash this except to say Automated testing is a big investment Hardware Software Training People Make sure the testing effort supports the organization s goals 9 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

10 ROI and Testing Evaluate the return to the organization Many promises, poor returns Automated testing - as carefully designed as the application Evaluate where automated testing expands testing limits AND provides real value 10 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

11 ROI and Testing Pitfall of NOT evaluating ROI Automated Testing is never seen as achieving important goals for the company 11 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

12 ROI and What to Test Levels of Testing - Value Unit Test Owned by the developers, for the benefit of the developers Quick and immediate confirmation that at this low level of testing the part works as specified 12 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

13 ROI and What to Test Levels of Testing - Value Heavy Weight Unit Test Low level tests that include out of unit dependencies Owned by the developers Benefits the engineering team Provides reasonably quick and immediate confirmation that at a fairly low level the part continues to work as specified Testing Automation suitable 13 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

14 ROI and What to Test Levels of Testing - Value Autonomous Service Tests At a higher level, encompassing many related functions that exist for a single business purpose in the architecture Only depends on Autonomous Service only and artificial constructs for external dependent actions Written by developers to confirm that the Autonomous Service s interfaces honor their contract Depend on artificial manipulation of otherwise external resources 14 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

15 ROI and What to Test Levels of Testing - Value Service Tests Similar to Autonomous Service Tests, but allows for interaction with whole system Focus is on Autonomous Service No artificial functionality needed Provides hints about integration quality Automated testing runs these on a regular basis (daily) and ad-hoc 15 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

16 ROI and What to Test Levels of Testing - Value Integration Tests Testing the APIs and interrelated functionality of the Autonomous Components Functional business process workflow tests, below the UI level Similar to Service Tests, has external dependencies Automated series of tests to validate health of major functional organizations in business terms 16 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

17 ROI and What to Test Levels of Testing - Value Acceptance Tests Similar to Integration Tests, with focus on business process Automated testing for regression and reduction in tedium Non-UI full business workflow tests Expensive and possibly fragile to automate 17 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

18 ROI and What to Test Levels of Testing - Value UI Tests This is the normal realm of manual testers Automated testing for regression and reduction in tedium Allows manual testers to concentrate on new functionality and the creative testing processes that they are best suited for, and make their task more efficient Business Trusts this level of testing the most Most expensive and fragile to automate and maintain 18 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

19 Testing Level vs Benefit Test Level Focus Value Execution time Unit Tests Discrete code functionality Product Confidence fine focus feedback on change quick Lower Heavy Weight Unit Test Discrete code functionality Code sections will work with other parts quick to moderate (minutes) Lower Autonomous Service Tests Defined, isolated business functional unit. Services will work as defined moderate to medium (minutes) Medium Service Tests APIs and interrelated functionality of Autonomous Components Services will work with other parts moderate to medium (minutes) Medium Integration Tests Application functionality below the UI Deployed code in an environment works together with other services. Medium to long High Acceptance Tests Business process testing Similar to Integration testing with business processes focus Medium to long Higher UI Tests UI driven functionality Testing the application as if you were an end user. Medium to long Highest With a nod to Adrian Smith and his talk 7 Deadly Sins of Automated Software Testing 19 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

20 Our plan, a mixture value at different levels Need a testing framework to support plan Selection of Automature for testing framework Capable of testing at all levels of the code Designed for the needs of small and medium sized shops fit our budget Best choice, even though it has many more features than we needed Low price point 20 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

21 Other testing vectors Other testing vectors not mentioned Performance Testing A/B Testing 21 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

22 Introducing Automated Testing into a Small Business or Start Up organization 22 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

23 Management Commitment and Support Involvement & support of management for Automated Testing Early involvement Highly Visible Long Term Plan Made the case for testing at different levels to maintain value of development investment and stability of deliverables Fits the corporate model to make the current development of code sustainable and flexible Regular reports to management on the progress, benefits and coverage of project 23 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

24 Testing the Framework Can Automature Framework test at multi levels? Spent the time creating tests to validate essential tests at different testing levels UI Testing Handles complex interdependencies and data driven testing Variable expansion (write the test steps, run through many times with varying input) Ability to execute at lower testing levels Handled Autonomous Component test and data gathering Can execute Heavy Weight Unit Tests and collect results for reporting 24 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

25 Testing the Framework Can Automature Framework work with all engineers? Ability to manage tests by non-developers Automature has novel spreadsheet based way to define Test Case and Test Suites. Supports reuse of useful elements Ability to Create Automated Test efficiently Simple concepts allow non-programmers to create robust tests Allows use of external applications and passing parameters We use it with Microsoft MSTest, to integrate Developer created tests into testing framework Other custom build modules allow test development from within Automature 25 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

26 Other Characteristics of Framework Record of test execution and results Notification system completion s Monitor Historic trends Resource correlations i.e. multi environment, or browser version Schedule correlations Team Sprint Build 26 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

27 More Characteristics of Framework Meaningful reports Direct Access to Framework database for customized queries Integration into the organization. The more people who can benefit from the automated tests, the more value it produces. 27 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

28 Final Tweaks Customizing the tool to work the way we do Test installation to understand all the nuances Learning the best way to use the framework Releases mapped to our Agile Teams Ignoring other features Bug tracking is done with other tools Build IDs are not relevant to us Etc. 28 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

29 Summary Automated Testing integration involves many different levels of the company. employed at different levels of the application code structure. integrated in the planning / architecture of the product. employed by all members of the engineering team maintains the features needed for the application to serve the company. 29 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

30 Questions? 30 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

31 SQA Automation References 7 Deadly Sins of Automated Software Testing By Adrian Smith The Automated Testing Handbook By Linda G. Hayes 31 ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

32 Steven A. Spura QA Manager Pro Teck Valuation Services 307 Waverley Oaks Rd. Suite 305 Waltham, MA Office: Fax: ASQ NEQC 60th Conference, Springfield, Massachusetts October 21, 2014

The Need for a Holistic Automation Solution to Overcome the Pitfalls in Test Automation

The Need for a Holistic Automation Solution to Overcome the Pitfalls in Test Automation Abstract Many organizations opt for test automation with the primary objectives of reducing the cost of quality and