Approaches for Auditing Software Vendors Chris Wubbolt, QACV Consulting, LLC IVT Validation Week October 20, 2016 10/20/2016 www.qacvconsulting.com 1
Objectives Understanding impact of vendor processes on validation Review of Agile SDLC processes New approaches to auditing software vendors Understanding how SDLC and test tools are used by vendors How SaaS vendors impact your company s validation approaches and data integrity controls. 10/20/2016 www.qacvconsulting.com 2
Impact of Vendor Practices on Validation Internal Validation vs. SaaS-based Internal Validation Vendor Validation Plan User Requirements Functional Specifications Configuration Specification Installation Qualification System Testing User Acceptance Testing Traceability Matrix Validation Summary Report Standard Operating Procedures SDLC Deliverables Software 10/20/2016 www.qacvconsulting.com 3
Saas-based vs. Internal Validation SaaS Validation Vendor Validation Plan User Requirements Functional User Acceptance Specifications Testing Configuration Traceability Matrix Specification Installation Validation Summary Qualification Report System Standard Testing Operating Procedures User Quality Acceptance Agreement Testing Traceability Matrix Validation Summary Report Standard Operating Procedures Software SDLC Deliverables Functional Specifications Configuration Specification Installation Qualification System Testing Traceability Matrix SOPs Release Management 10/20/2016 www.qacvconsulting.com 4
Software Vendor Truisms Software vendors develop and maintain software. All software vendors are software developers. Quality software development is essential to the validation of a system. 21 CFR Part 11.10 (a): Validation of systems to ensure accuracy, reliability, consistent intended performance, and the ability to discern invalid or altered records. 10/20/2016 www.qacvconsulting.com 5
Software Quality Truisms Quality cannot be tested into a system. Quality must be designed into a system. 10/20/2016 www.qacvconsulting.com 6
Software Development Software Development Life Cycle (SDLC) The set of activities that constitute the processes that are mandatory for the development and maintenance of software. The management and support processes that continue throughout the entire life cycle, as well as all aspects of the software life cycle from concept exploration through retirement, are covered. Utilization of the processes and their component activities maximizes the benefits to the user when the use of this standard is initiated early in the software life cycle. (1) (1) IEEE Standard for Developing Software Life Cycle Processes, 1992 10/20/2016 www.qacvconsulting.com 7
SDLC Methodologies Waterfall Prototyping Incremental Development Spiral Rapid Application Development Agile Code and Fix (Cowboy Coding) 10/20/2016 www.qacvconsulting.com 8
Elements of an SDLC Design Testing (unit, module, system, etc.) Bug Fixes Requirements Configuration Management SQA Testing Release Management Maintenance (Customer Support) 10/20/2016 www.qacvconsulting.com 9
Vendor Quality System Elements Quality Manual Document Management Training Program Quality Assurance Supplier Management CAPAs / Investigations SDLC Procedures Customer Support 10/20/2016 www.qacvconsulting.com 10
Waterfall Methodology Requirements Analysis Requirements Analysis Design Design Implementation Implementation Verification / Verification / Testing Testing Operation / Operation / Maintenance Maintenance 10/20/2016 www.qacvconsulting.com 11
SDLC Agile Methodology 10/20/2016 www.qacvconsulting.com 12
SDLC Agile Methodology 10/20/2016 www.qacvconsulting.com 13
SDLC Agile Methodology Focus on short iterations of development Delivery of minimum viable product within short periods of time (2-3 weeks) Collaboration between end user and development team Continuous end user involvement is critical 10/20/2016 www.qacvconsulting.com 14
Agile - Scrum An iterative and incremental agile development framework. A flexible, holistic strategy where a development team works as a unit to reach a common goal. Enables teams to self-organize by encouraging physical co-location or close online collaboration and daily face-to-face communication among all team members and disciplines in the project. 10/20/2016 www.qacvconsulting.com 15
Agile - Scrum A key recognition is that during end users can change their minds about the system requirements. Scrum adopts an approach to deliver quickly and respond to emerging requirements. 10/20/2016 www.qacvconsulting.com 16
Software Vendor Truisms All software vendors are software developers. The software development life cycle methodology is arguably the most important process for a software vendor. Requirements Backlog User Stories Design/Development Unit Testing SQA Testing Release Management Code Reviews Design Documents 10/20/2016 www.qacvconsulting.com 17
Why is this important? Requirements Backlog User Stories Design/Development SQA Testing Unit Testing Code Reviews Design Documents Release Management 1. The vendors SDLC determines the quality of the software. 2. For SaaS vendors, the SDLC documentation may also be used as validation deliverables. 3. The SDLC documentation is likely to be maintained within vendor SDLC tools. 10/20/2016 www.qacvconsulting.com 18
Use of SDLC and Test Tools Requirements Backlog User Stories Design/Development SQA Testing Release Management Creation and Management of Requirements & User Stories Documentation of Unit Testing, Code Reviews & Design Documentation SQA Test Documentation Often used as validation tests. Configuration / Source Code Management Management of Bugs and Customer Support Tickets 10/20/2016 www.qacvconsulting.com 19
SDLC/Vendor Tools Requirements Management Source Code Management Configuration Management Code Review and Unit Testing Testing including automated testing Issue Management Customer Support Document Management 10/20/2016 www.qacvconsulting.com 20
SDLC/Vendor Tools - Examples Team Foundation Server (TFS) HP Quality Center HP Load Runner Altassian (Jira) Subversion Test Stuff Test Track CoSign SharePoint Wiki Pages Salesforce.com 10/20/2016 www.qacvconsulting.com 21
SDLC Tools Team Foundation Server (TFS) Requirements Management Use Cases User Stories Design Code Review Unit Testing Traceability Testing Approvals Release Management 10/20/2016 www.qacvconsulting.com 22
SDLC Tools Questions to ask What do the tools do? Do the tools impact software quality? Do the vendor s procedures reflect the use of these tools? Are the tools controlled, qualified, or validated? How are the records maintained by the tools managed and controlled? How are records approved? 10/20/2016 www.qacvconsulting.com 23
SDLC Tools What can go wrong? Issue Management Vendor used a cloud hosted version of Jira, which was used for issue management and change control. The license was not renewed and all records were lost. Electronic Approval Vendor used a local implementation of CoSign for approval of records. When license expired the electronic signatures applied previously could not be validated. 10/20/2016 www.qacvconsulting.com 24
SDLC Tools What can go wrong? Document Management Vendor used SharePoint workflow for approval of quality documents. The SharePoint configuration was setup to delete workflows after 90 days. All workflows (and subsequent document approvals) were deleted for all quality documents. Testing Test Stuff testing records could not be located for SQA testing. 10/20/2016 www.qacvconsulting.com 25
SDLC Tools What can go wrong? Automated Testing Automated test tools passed failing results. Test tools were not qualified. Tool Upgrades / Replacements Inability to migrate records from legacy tools. Records Unable to present records of SDLC activities, including test results. 10/20/2016 www.qacvconsulting.com 26
Computerized Systems GxP Electronic Recordkeeping Program Standard Operating Procedures Trained Personnel (including IT) Qualified Infrastructure Validated Applications Data Integrity Data Availability Data Retention 10/20/2016 www.qacvconsulting.com 27
The Old Days Software Applications QMS LIMS 10/20/2016 www.qacvconsulting.com 28
The Old Days Software Applications QMS LIMS 10/20/2016 www.qacvconsulting.com 29
The Old Days Pharma AData Center Inc STILL NEED GxPElectronic Recordkeeping Controls Qualified Infrastructure Standard Operating Procedures Trained Personnel (including IT) Validated Applications 10/20/2016 www.qacvconsulting.com 30
Software as a Service Saas Provider Software Applications QMS LIMS Data Center Fail Over Site 10/20/2016 www.qacvconsulting.com 31
Software Software as a Service Vendor Provider Software Vendor Quality System Quality System SDLC Processes SDLC Processes Customer Support Customer Validation Support Data Integrity Controls Hosted Environment Typically Hosted Environment not directly regulated is used for a inspected direct GxPfunction by regulatory (record agencies. keeping) Audited and is more by clients likely to for be adherence inspected to by standards. regulatory agencies. Quality Audited of by SLC clients Documentation, for adherence Testing, to standards etc. varies (GxP, considerably Part 11). for each vendor. Quality of SDLC Documentation, Testing, etc. varies considerably for Sponsor each vendor. responsible for installation, validation, and electronic recordkeeping SaaS provider responsible controls at sponsor for some location. aspects of installation, validation, and electronic recordkeeping controls. 10/20/2016 www.qacvconsulting.com 32
SaaS Vendor Responsibilities Validation (with Pharma Company) Change Control Incident Management Maintenance Security (Physical and Logical) Electronic recordkeeping Backup and Restore Disaster Recovery 10/20/2016 www.qacvconsulting.com 33
Vendor Audit Observations - Considerations Specifications Not complete Not updated periodically after changes Test Records No pre-approved Test Plans Results not reviewed by second person Integrity of test results No approved summary reports Release Management 10/20/2016 www.qacvconsulting.com 34
Vendor Audit Observations Considerations Test Record Integrity Results and signatures/initials typed into Word document or Excel spreadsheet No failures documented Test dates and times do not correlate 10/20/2016 www.qacvconsulting.com 35
Vendor Audit Observations Record Integrity Considerations Lack of records to demonstrate successful backup Failed backups Lack of documentation of disaster recovery testing 10/20/2016 www.qacvconsulting.com 36
Summary Reviewed impact of vendor processes on validation Review of Agile SDLC processes Discussed new approaches to auditing software vendors Reviewed how SDLC and test tools are used by vendors Discussed ow SaaS vendors impact your company s validation approaches and data integrity controls. 10/20/2016 www.qacvconsulting.com 37
Questions Chris Wubbolt QACV Consulting, LLC Telephone: 610-442-2250 E-mail: chris.wubbolt@qacvconsulting.com www.qacvconsulting.com 38