Quality Assurance and IT Risk Management

Transcription

1 Quality Assurance and IT Risk Deutsche Bank s QA and Testing Transformation Journey Michael Venditti Head of Enterprise Testing Services, Deutsche Bank

2 IT RISK - REGULATORY GOVERNANCE Major shifts in the role and functioning of Financial Institutions over the last few years have brought about a new way to regard risk. New banking products, increased government scrutiny and intense focus on operating efficiencies bring forth greater risks and a larger set of rules and regulations. A recent banking survey by the Economist Intelligence Unit shows that [30%] of all banks answer to 10 or more regulators [75%] of all banks report to four or more regulators [350+] potential regulatory reviews a year for a global bank

3 QA & TESTING TRANSFORMATION OVERVIEW Test Transformation Enterprise Testing Services The mission of Enterprise Testing Services is to define and validate that Testing Standards, Testing Tools and Solutions are implemented in a sound and sustainable way, maximizing focus on quality, risk reduction and stability. Challenges Intense Regulatory scrutiny focused on IT Testing Processes Inconsistencies in Testing Practices Need for enhanced Oversight for Testing Need for increased Production stability Minimum Testing Standards As a first step in transformation, Minimum Testing Standards (MTS) were defined in line with recommendations from External Regulatory Bodies (such as Monetary Authority of Singapore (MAS)) and the DB Internal Audit Group Successful MTS roll-out provided a repeatable and auditable approach laid the foundation for more robust Test Standards Test Standard Framework As a next step in transformation, Test Standard Framework was rolled out to create a foundation of a common set of Testing Standards unify the practice of Testing across the bank simplify test processes by integrating standards with tools Test Standards Governance A robust governance process was implemented to track adoption or compliance to Test Standard Framework by integrating test controls into SDLC Governance enabling all applications to demonstrate compliance to controls in a centralized tool automating verification of controls

4 Minimum Testing Standards DEFINE EDUCATE GOVERN Minimum Testing Standards (MTS) were developed and published. MTS requires evidence of standard test artifacts (following good engineering practices) that can be adapted for different types of releases. Standard requirements & related artefacts are: Test Risk Assessment, Test Approach, Test Cases & and Test Execution Results, Defects, Test Completion Report MTS introduced 3 levels of maturity Level 1, 2 and 3 Standards were designed to be methodology agnostic Comprehensive MTS orientation and training plan was published and all test professionals in the bank were mandated to attend MTS framework was published in a centralized repository and is accessible by all An e-learning module was developed with integrated certification mechanism Monthly training sessions were conducted and are on-going Special focus on strategic test vendors An independent Minimum Testing Standards Compliance (CM) team was constituted All application releases in scope were monitored through the change management system MTS CM team validates the artifacts provided against MTS requirements, reports any gaps Remediation of gaps was monitored through subsequent releases A Non-Compliance management process was rolled out Applications with two Non-Compliant releases were recommended for heightened change control process

5 TEST STANDARD FRAMEWORK DEFINE Collaborated with key testing stakeholders across various Business Units, Regions, Tech Centres etc., Developed Test Standard Framework, Key Operating Procedures, Templates, Tools Standardization EDUCATE GOVERN Test Standard Framework and associated artefacts were published in the Bank s policy portal Conducted Live Orientation sessions across the globe to cover over professionals and stakeholders A global webcast was delivered to ~ professionals Detailed How-To videos were created All training media was made available in a centralized Testing Standards Portal Periodic and on-demand training sessions were conducted (on-going) Q&A feature is enabled in the Testing Standards Portal to address queries Integrated Test related controls in SDLC Governance platform to enable independent verification of controls for random selection of releases Any gaps identified are either fixed in-flight, or risk accepted by Business Automated verification of controls is implemented for various test controls such as Test Environment compliance controls, test cases, defects etc.,

6 TEST STANDARD FRAMEWORK TEST PLANNING Test Standards Framework: Test Planning o Testing Risk Assessment (TRA) o Test Strategy o Test Plan Test Design o Test Cases Test Execution o Defect o Test Execution Results Test Closure o Test Evaluation Reporting Additional Standard Requirements o Test Data o Test Environment o Performance Engineering Test Planning is required for all releases and must detail what and how Testing is to be performed. Test Planning consists of several elements that may be captured in one or several documents Testing Risk Assessment (TRA) Created to determine the risk associated with release to production, and provide the types of Testing recommendations which are required to minimize that risk Test Strategy Test Plan Created to gain general Stakeholder agreement, ensure appropriate availability of resources for What Test Types are to be performed Provides details of How" each Test Type will be performed, and is updated for each release For applications releasing very frequently a Test Plan - Multi Release may be produced, instead of a Test Plan for each release Test Plan - Multi Release Vs Test Plan Release Frequency Test Planning Artifacts <= 3 weeks Test Plan - Multi Release > 3 weeks Test Plan

7 TEST STANDARD FRAMEWORK TEST DESIGN, EXECUTION & CLOSURE Test Standards Framework: Test Planning o Testing Risk Assessment (TRA) o Test Strategy o Test Plan Test Design o Test Cases Test Execution o Defect o Test Execution Results Test Closure o Test Evaluation Reporting Additional Standard Requirements o Test Data o Test Environment o Performance Engineering Test Design Test Cases Test Cases are used to verify that the Business and Technical requirements work as expected and validate that the functionality meets specified requirements Test Execution Defect A Defect Process defines how defects are recognized and resolved in all stages of SDLC. It involves recording Defects, ensuring repeatability, classification and prioritization, and reporting the resolution progress Test Execution Results Test execution results must include planned Test Cases executed with actual Test results and stored in a standard Test Tool. All failed Test Cases must be traceable to Defects Test Closure Test Evaluation Reporting A Test Summary Report at the end of each Test Type may be required. These may be collected at the end of the Test Closure Phase of the STLC. A Test Evaluation Report must be prepared to demonstrate successful completion of all Test Types at completion of the Test Closure Phase.

8 TEST STANDARD FRAMEWORK ADDITIONAL STANDARDS Test Standards Framework: Test Planning o Testing Risk Assessment (TRA) o Test Strategy o Test Plan Test Design o Test Cases Test Execution o Defect o Test Execution Results Test Closure o Test Evaluation Reporting Additional Standard Requirements o Test Data o Test Environment o Performance Engineering Test Data Testing must be conducted on synthetic Test data that is free from Client Identifying Data. Sensitive data must be anonymized in a controlled environment before moving to a Test Environment Production-like access controls must be in place when sensitive information needs to be used without masking in a Test Environment Test Environment Applications must have Test Environments that resemble production from a hardware, software and configuration point of view QA Test Environments must be logically separated from all other non-qa Environments All Test Environment changes must be implemented with strict access controls Test Environment changes must be performed in a controlled manner Any change to a Test Environment should be accompanied with an appropriate back out or recovery mechanism and be smoke Tested to confirm viability Performance Engineering Performance Testing is required whenever - Changes are made to any hardware, configuration or software components Components are moved to different data centers Usage volume increases A dedicated Performance Test Environment must be made available that resembles production

9 GOVERNANCE OVERVIEW A robust governance model is implemented in SDLC Governance tool for governing the development and delivery of planned application releases. It is a tool of record for compliance rather than an SDLC workflow orchestrator. There are 5 key elements of the model: 1. Policy and control levels mapping elements of policies and specific audit point solutions to a set of controls of appropriate level IT & Security Policy Test Standard Framework Audit issue solutions 3. Release characteristics determining which controls need to apply depending on release characteristics Mandatory Controls for each release 2. Control origins confirmations from tools and repositories in the strategic tools stack, or performed manually SDLC Chg. Mgt. Live 4. Generic lifecycle mapping the controls to appropriate generic phases and quality gates in the SDLC, independent of development method 5. Assurance assuring that controls are appropriately met, involving project and external stakeholders where needed, plus tight integration with Change as a precursor to production deployment

10 TEST CONTROLS IN SDLC GOVERNANCE Application/ Release Level Application Release Release Release Release Release Application Application Application Testing Risk Assessment Control Summary Test Plan for release created Test Cases for release created & traceability established Test Cases for release executed Defects (where appropriate) are captured and tracked for the release Test Evaluation report approved and completed Environment management plan for application in place UAT / OAT or pre-production staging environment for application operational and distinct from production environment UAT / OAT and Development application environments are defined in the centralized tool Control Description A Testing Risk Assessment (TRA) is an Excel based tool to profile an application to minimize risks and provide recommendations of various test types required. Single or reusable Multi Release Test Plans may be produced depending on the frequency and agility of application releases. Test cases for release are created, available and linked to the requirements. Ensure that all test cases have been executed and the evidence has been generated / documented. Defects must be captured in a defined repository. Documentation of the defect must provide sufficient details to understand the impact and to reproduce it. Test Evaluation Report prepared to demonstrate successful completion of all Test Types as per the Test Plan. Where applicable, references to tools can be provided for test report data. db-tec portal enables automatic verification of these controls for all applications that are on-boarded

11 TRANSFORMATION JOURNEY Basic Inconsistent Testing processes No standard STLC process. Production instability Progressive Organizational & Governance structures Implemented Global Test Standard Framework defined Test Standards, Processes and Templates rolled out globally Test Process Governance in place 2 Basic (Level 2) 3 Pockets of Best Practices, but Inconsistent Progressive (Level 3) 4 Mature (Level 4) Institutionalized Testing Processes & Standards Policy Driven Standards & Processes 5 Industry Leader (Level 5) Leading Organization 1 Start up (Level 1) Testing is Chaotic, When Performed The Levels defined here are based on Infosys Enterprise QA Transformation Model (EQATM) which provides a framework of 4 test dimensions and 20 process areas

12 TRANSFORMATION PROGRESS SNAPSHOT Continuing to Mature Test Requirements Gathering Knowledge 5 Test Strategizing Test Case & Test Data Testing Career Path Design 4 Training & Skill Test Tools & Development Test Measurement & 3 2 Automation Test Execution & Defect Reporting Test Requirements Gathering Knowledge 5 Test Strategizing Test Case & Test Data Testing Career Path Design 4 Training & Skill Development 3 Test Tools & Automation Test Measurement & 2 Test Execution & Defect Reporting Test Organizational Structure 1 Test Environment Test Organizational Structure 1 Test Environment Test Policy Test Estimation Test Policy Test Estimation Test Process Test Planning & Monitoring Test Process Test Planning & Monitoring Test Methodology Defect Organizational Risk Test Ware Test Communication Test Methodology Defect Organizational Risk Test Ware Test Communication

13 KEY ACCOMPLISHMENTS IT Professionals offered training through Global Webcast Test Professionals Trained through Live Orientation Sessions Test Professionals attended MTS Training elearning Certificates issued Applications assessed for MTS Compliance Random Spot Checks conducted Releases monitored for Compliance Potential savings by reducing production incidents

14 THANK YOU