Test Automation Strategies in Continuous Delivery Nandan Shinde Test Automation Architect (Tech CoE) Cognizant Technology Solutions
The world of application is going through a monumental shift.. Evolving Customer Expectations Emerging Technologies Changing Business Requirements - 2 -
An example 500 and 1000 Rupee note become obsolete on the midnight of 08/Nov 2016 Time for bankers to stop 17,000 ATMs from dispensing old notes 4 Hours - 3 -
creating the need for Continuous Delivery (CD) CD is a disciplined, integrated, and highly automated process to expedite the process of incorporating new code from initial development to production release with confidence that the new code will function as designed and improve the value of the product. - 4 -
CD lays the path for Continuous Integration.. Automated build and test methodology driven by CI tools, makes sure that changes to a software code base are successfully built, tested, reported, and are made available to all stakeholders after they are introduced. Developer Database Feedback Mechanism Developer Test Developer Commit Changes Version Control Repository Poll CI SERVER Integration Build Machine Build Server Compile Source code Integrate Database Run Tests Deploy Software Develop Scripts - 5 -
Integrate, Build,Deploy and Validate Continuous Integration and impact of Automation Identify Tools and Determine Integration Touch Points Continuously Integrate Build Deploy and Validate Health Checks Daily Critical Runs Metrics and Reporting T T Weekend Runs T T T T T Tools Identification Determine Integration Touch Points Identify Tools Stack that can support End to End CI Orchestration Integration of Build Scripts and Automated Test Scripts Build Scripts Setup Sub Version Automated Code Pulls Automated Deployments Automated Smoke Test Runs Daily - Critical Regression runs and Weekend Full Regression Analyze Test failures Automated Metrics Weekend Regression Test Runs Trends Automated Publishing of results to Project stakeholders Legend T Testing Team - 5 -
Integrating Automation in Continuous Integration.. Collaborative Development Increased collaboration between teams Continuous Testing Integration of software testing with deployment and operations Continuous Release and Deployment for increased delivery speed and frequency Continuous Monitoring Improve quality by monitoring performance - 7 -
Automation Test Strategies in Continuous Integration Behavior Driven Development Behavior Driven Development - 8 -
Automation Test Strategies in Continuous Integration Early Automation - 9 -
Automation Test Strategies in Continuous Integration Optimize Testing at Code Level - 10 -
Automation Test Strategies in Continuous Integration Minimize manual intervention - 11 -
Automation Test Strategies in Continuous Integration Achieve Continuous Regression - 12 -
Automation Test Strategies in Continuous Integration Think Before UI - 13 -
In summary Automation is imperative to induce the required agility in QA practices while enhancing the efficiency and effectiveness Business Transformation Continuous Improvement DevOps Implementation Cross Functional Service Enablement - 14 -
Bibliography References: Modernize Application Development to Succeed as a Digital Business, Gartner, 30 March 2016 (https://www.gartner.com/doc/3270018?refval=&pcp=mpe) Definitions: Continuous Integration - Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early. Continuous Delivery - Continuous delivery (CD) is a software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. It aims at building, testing, and releasing software faster and more frequently - 14 -
About the Author: Nandan Shinde is Associate Director of Projects with Cognizant s Quality Engineering and Assurance business unit and is part of the company s Technology Center of Excellence. Currently a test automation architect, he previously worked as a test automation expert, project lead, automation strategist and program manager over a period of twelve years with Cognizant s banking and finance services clients. His responsibilities pivot around building and socializing automation frameworks, as well as implementing solutions from front-end, middle-tier and data automation strategies to leading automation program for large banking customers. He handles automation consulting engagements across the vertical industries that Cognizant serves. He can be reached at Nandan.Shinde@cognizant.com. - 15 -
Thank You!!! - 16 -
Process Workflow Example Continuous Testing Dashboard / Analysis Develop Code Quality & Build Deploy Deploy Functional Test NFT SAST (DEV) (QA) Functional Test NFT Continuous Integration Continuous Testing Continuous Improvement/Feedback 18-17 -
BDD Continuous Business Involvement & Feedback Signed Off by Business Clarity in Behavior, Continuous Feedback leads to reduced defect injection Sprint 1,2, N Continuous Demo To Business User Story Automatically created by BDD Tool Automation Team//DEV Script Execution Product SME Acceptance Criteria Function Skeleton Created Script generation QA/BA/DEV Product SME/BA Step Definition generation and Feature Execution (Dry Run) Code / Refactor App App QA/BA DEV Feature Files QA/BA Write Scenarios Failed Test Case Passed Test Case Test Result Sprint 1,2, N - 17 -
BDD - Bringing in Early Testing Developer Dev. CI Delay in QA Feedback Refactor Test Fix QA Cost of Fixing a Defect Late Early Testin Testing SDLC Time Line g Need to Push Testing early in SDLC Behavior-Driven Development (BDD) brings Test Driven Development and Domain Driven Design into an integrated whole Behavior-Driven Development Test Driven Development Continuous Integration Simple Illustration of BDD & CI Product SME Product SME & BA BA & Dev Behavioral Scenarios Development Coding On Failure Product Backlog User Story BA & QA Acceptance Criteria QA Make the Behavioral Scenarios executable into test scripts Continuous Testing Automated Validation Product Demo On Success Test Automation in Development Defect Prevention Continuous Delivery and faster feedback - 18 -
Continuous Integration Implementation Benefits Automation Area Health Check Full Smoke Test Weekly Regression Run Description Focused automated test to verify deployment and associated application configurations Set of basic tests that ensures the application is ready to do the subsequent detailed testing (Functionality / Regression) by verifying key functionalities, Interfaces and Services. Full Regression suite that exercises 70% of the application functionality against the weekly build delivered for sprint testing Before Implementation After CI Implementation 12 Minutes 6 Minutes 50% 40 Minutes 20 Minutes 50% 200 Hours 80 Hours 60% Savings 50% Effort Savings 50% Effort Savings 60% Effort Savings 21-19 -
Continuous Integration - Critical Success Factors 1 Shift Left 2 QA and Dev synchronization 3 Continuous Testing Integrate, Build,Deploy and Validate 4 Service Virtualization 5 Environments on Demand - 6 -
How End to End Test Automation in CI Works Collaborative development and Continuous testing 1 2 Dev-test with lab management automation Deployment automation and Continuous Integration Automated code progression & validations Plan and build Automated Static code analysis Provision Test Environment Deploy to staging and prod Provisioning Dev Environment & SV Automated Unit test & Code Coverage Automated UI Test Integrated code coverage Monitor 3 Continuous feedback and optimization using CI - 9 -
Continuous Automation Workflow Example User Story Continuous Monitoring Continuous Delivery Requirements Modeling Dev Design Test Design Coding Automatio n Scripting Automated Build Build Verification Automate d Deploy Deploymen t Verification Continuous Automation Automated Reporting User Story creation Business Model creation Dev Technical Design QA Test case Design Test data Design Dev Code for requirements QA Automation Feasibility and Analysis Creation of Automation scripts Scheduled or Triggered Automated Build scripts using a Continuous Integration Server like Jenkins Set of automated tests run on every build to verify that the Build is testable before deploying in QA Automated code deployment on environments after the build verification tests are complete Unit\Smoke Tests to verify the high level Functional test cases Automated Execution of Functional and regression test cases Reporting the Build, Deployment and testing results in e-mail to respective teams BA DEV QA OPS - 10 -
Continuous Integration & Test Automation Requirements Modeling and Automation Tools based approach to modeling requirements as business process flows and activity diagrams Non-GUI Automation Server side Automation Message Validation Batch automation Automation for ETL layers Non-Conventional Automation Automation tools and plug-ins enabling quicker completion of Test result compilations, reporting and metrics management Integrated Automation Regression Automation + Virtualization Newer automation frameworks enabling parallel execution of automation scripts in available/virtualized environments thus accelerating cycles Early Automation Extend the tool ecosystem to Dev and Ops teams for continuous automation early in the SDLC Test Design Automation Automated generation of Test scenarios and Test cases for requirement models Generation of Risk based test cases to accelerate testing Test Data Automation Adoption of automation frameworks and tools for test data to enable data driven automation - 11 -
Key success factors for CI/CD implementation Success factors Benefits Increased automation at multiple layers Reduction in Time to Market Higher Automation Manual Regression Testing effort reduced by 60-75% Tests execute multiple times a day with every check-in, and therefore, guarantees more quality and coverage Early testing in lifecycle for early detection of defects with stringent quality gates Early Test Automation allows executing more tests in less time, while manual test efforts focus on complex use cases and edge cases that cannot be automated Enterprise Dashboard Leveraging right tools, process and Tool Integration Service Virtualization Provide visibility & feedback to all stakeholders on the health of the project at every stage starting from development through production Streamline CI/CD pipeline, automate infrastructure and tooling wherever feasible. Deliver E2E product testing capability Early Component Integration increased quality, reduction in defects - 13 -