Approaches for Auditing Software Vendors

Transcription

1 Approaches for Auditing Software Vendors Chris Wubbolt, QACV Consulting, LLC IVT Validation Week October 20, /20/

2 Objectives Understanding impact of vendor processes on validation Review of Agile SDLC processes New approaches to auditing software vendors Understanding how SDLC and test tools are used by vendors How SaaS vendors impact your company s validation approaches and data integrity controls. 10/20/

3 Impact of Vendor Practices on Validation Internal Validation vs. SaaS-based Internal Validation Vendor Validation Plan User Requirements Functional Specifications Configuration Specification Installation Qualification System Testing User Acceptance Testing Traceability Matrix Validation Summary Report Standard Operating Procedures SDLC Deliverables Software 10/20/

4 Saas-based vs. Internal Validation SaaS Validation Vendor Validation Plan User Requirements Functional User Acceptance Specifications Testing Configuration Traceability Matrix Specification Installation Validation Summary Qualification Report System Standard Testing Operating Procedures User Quality Acceptance Agreement Testing Traceability Matrix Validation Summary Report Standard Operating Procedures Software SDLC Deliverables Functional Specifications Configuration Specification Installation Qualification System Testing Traceability Matrix SOPs Release Management 10/20/

5 Software Vendor Truisms Software vendors develop and maintain software. All software vendors are software developers. Quality software development is essential to the validation of a system. 21 CFR Part (a): Validation of systems to ensure accuracy, reliability, consistent intended performance, and the ability to discern invalid or altered records. 10/20/

6 Software Quality Truisms Quality cannot be tested into a system. Quality must be designed into a system. 10/20/

7 Software Development Software Development Life Cycle (SDLC) The set of activities that constitute the processes that are mandatory for the development and maintenance of software. The management and support processes that continue throughout the entire life cycle, as well as all aspects of the software life cycle from concept exploration through retirement, are covered. Utilization of the processes and their component activities maximizes the benefits to the user when the use of this standard is initiated early in the software life cycle. (1) (1) IEEE Standard for Developing Software Life Cycle Processes, /20/

8 SDLC Methodologies Waterfall Prototyping Incremental Development Spiral Rapid Application Development Agile Code and Fix (Cowboy Coding) 10/20/

9 Elements of an SDLC Design Testing (unit, module, system, etc.) Bug Fixes Requirements Configuration Management SQA Testing Release Management Maintenance (Customer Support) 10/20/

10 Vendor Quality System Elements Quality Manual Document Management Training Program Quality Assurance Supplier Management CAPAs / Investigations SDLC Procedures Customer Support 10/20/

11 Waterfall Methodology Requirements Analysis Requirements Analysis Design Design Implementation Implementation Verification / Verification / Testing Testing Operation / Operation / Maintenance Maintenance 10/20/

12 SDLC Agile Methodology 10/20/

13 SDLC Agile Methodology 10/20/

14 SDLC Agile Methodology Focus on short iterations of development Delivery of minimum viable product within short periods of time (2-3 weeks) Collaboration between end user and development team Continuous end user involvement is critical 10/20/

15 Agile - Scrum An iterative and incremental agile development framework. A flexible, holistic strategy where a development team works as a unit to reach a common goal. Enables teams to self-organize by encouraging physical co-location or close online collaboration and daily face-to-face communication among all team members and disciplines in the project. 10/20/

16 Agile - Scrum A key recognition is that during end users can change their minds about the system requirements. Scrum adopts an approach to deliver quickly and respond to emerging requirements. 10/20/

17 Software Vendor Truisms All software vendors are software developers. The software development life cycle methodology is arguably the most important process for a software vendor. Requirements Backlog User Stories Design/Development Unit Testing SQA Testing Release Management Code Reviews Design Documents 10/20/

18 Why is this important? Requirements Backlog User Stories Design/Development SQA Testing Unit Testing Code Reviews Design Documents Release Management 1. The vendors SDLC determines the quality of the software. 2. For SaaS vendors, the SDLC documentation may also be used as validation deliverables. 3. The SDLC documentation is likely to be maintained within vendor SDLC tools. 10/20/

19 Use of SDLC and Test Tools Requirements Backlog User Stories Design/Development SQA Testing Release Management Creation and Management of Requirements & User Stories Documentation of Unit Testing, Code Reviews & Design Documentation SQA Test Documentation Often used as validation tests. Configuration / Source Code Management Management of Bugs and Customer Support Tickets 10/20/

20 SDLC/Vendor Tools Requirements Management Source Code Management Configuration Management Code Review and Unit Testing Testing including automated testing Issue Management Customer Support Document Management 10/20/

21 SDLC/Vendor Tools - Examples Team Foundation Server (TFS) HP Quality Center HP Load Runner Altassian (Jira) Subversion Test Stuff Test Track CoSign SharePoint Wiki Pages Salesforce.com 10/20/

22 SDLC Tools Team Foundation Server (TFS) Requirements Management Use Cases User Stories Design Code Review Unit Testing Traceability Testing Approvals Release Management 10/20/

23 SDLC Tools Questions to ask What do the tools do? Do the tools impact software quality? Do the vendor s procedures reflect the use of these tools? Are the tools controlled, qualified, or validated? How are the records maintained by the tools managed and controlled? How are records approved? 10/20/

24 SDLC Tools What can go wrong? Issue Management Vendor used a cloud hosted version of Jira, which was used for issue management and change control. The license was not renewed and all records were lost. Electronic Approval Vendor used a local implementation of CoSign for approval of records. When license expired the electronic signatures applied previously could not be validated. 10/20/

25 SDLC Tools What can go wrong? Document Management Vendor used SharePoint workflow for approval of quality documents. The SharePoint configuration was setup to delete workflows after 90 days. All workflows (and subsequent document approvals) were deleted for all quality documents. Testing Test Stuff testing records could not be located for SQA testing. 10/20/

26 SDLC Tools What can go wrong? Automated Testing Automated test tools passed failing results. Test tools were not qualified. Tool Upgrades / Replacements Inability to migrate records from legacy tools. Records Unable to present records of SDLC activities, including test results. 10/20/

27 Computerized Systems GxP Electronic Recordkeeping Program Standard Operating Procedures Trained Personnel (including IT) Qualified Infrastructure Validated Applications Data Integrity Data Availability Data Retention 10/20/

28 The Old Days Software Applications QMS LIMS 10/20/

29 The Old Days Software Applications QMS LIMS 10/20/

30 The Old Days Pharma AData Center Inc STILL NEED GxPElectronic Recordkeeping Controls Qualified Infrastructure Standard Operating Procedures Trained Personnel (including IT) Validated Applications 10/20/

31 Software as a Service Saas Provider Software Applications QMS LIMS Data Center Fail Over Site 10/20/

32 Software Software as a Service Vendor Provider Software Vendor Quality System Quality System SDLC Processes SDLC Processes Customer Support Customer Validation Support Data Integrity Controls Hosted Environment Typically Hosted Environment not directly regulated is used for a inspected direct GxPfunction by regulatory (record agencies. keeping) Audited and is more by clients likely to for be adherence inspected to by standards. regulatory agencies. Quality Audited of by SLC clients Documentation, for adherence Testing, to standards etc. varies (GxP, considerably Part 11). for each vendor. Quality of SDLC Documentation, Testing, etc. varies considerably for Sponsor each vendor. responsible for installation, validation, and electronic recordkeeping SaaS provider responsible controls at sponsor for some location. aspects of installation, validation, and electronic recordkeeping controls. 10/20/

33 SaaS Vendor Responsibilities Validation (with Pharma Company) Change Control Incident Management Maintenance Security (Physical and Logical) Electronic recordkeeping Backup and Restore Disaster Recovery 10/20/

34 Vendor Audit Observations - Considerations Specifications Not complete Not updated periodically after changes Test Records No pre-approved Test Plans Results not reviewed by second person Integrity of test results No approved summary reports Release Management 10/20/

35 Vendor Audit Observations Considerations Test Record Integrity Results and signatures/initials typed into Word document or Excel spreadsheet No failures documented Test dates and times do not correlate 10/20/

36 Vendor Audit Observations Record Integrity Considerations Lack of records to demonstrate successful backup Failed backups Lack of documentation of disaster recovery testing 10/20/

37 Summary Reviewed impact of vendor processes on validation Review of Agile SDLC processes Discussed new approaches to auditing software vendors Reviewed how SDLC and test tools are used by vendors Discussed ow SaaS vendors impact your company s validation approaches and data integrity controls. 10/20/

38 Questions Chris Wubbolt QACV Consulting, LLC Telephone: