Moving from a Paper to Paperless validation effort and how to get the most efficient mix of Manual vs. Automated testing. Overview The desire to use tools to increase validation productivity with the consequent reduction in validation cycles is a main driver for using testing tools. The very time consuming nature of validation testing is an all too familiar experience for companies that are under regulatory control. The vast majority of current validation efforts at the point of test execution are currently manual. This does not mean that tools aren t deployed but it is usually in the area of validation systems management rather than the actual test execution itself. This document highlights some of the key considerations when planning to use testing tools for the complete validation lifecycle. Summary The complexity of your processes, corporate IT infrastructure and ability to control the output of any tool will have a huge impact on your final ROI from a test tool. Test tools can be used in the whole lifecycle of Requirements, Capture, Execution (manual or automated testing), Electronic Signatures, Trace Matrices and generation of the documented evidence required when validating computer systems in regulated industries. Even if validation cycles are not significantly reduced the quality, consistency and traceability of validation can be raised to make the transition to using test tools a very worthwhile exercise. Assessment Criteria Test tools for validation can be a huge time saver however it is crucial to understand exactly how easy or difficult this is going to be in your own environment before committing time and money to the effort. Some of the major factors are listed below: 1. Outputs IQ/OQ/PQ scripts need to be created, reviewed, pre-approved, executed and usually post approved as well. All of this must be produced as documented evidence and is often to be in a defined layout that is specific to your company. This is a lot of steps and while it may be unreasonable for one single tool to perform all of these actions as well as the core automation the tasks still need to get done. It is not enough for the tool to be good at creating automated scripts the output both pre and post execution must be detailed and preferably configurable to minimise any post processing efforts. In many validation scenarios the output must be at a step by step level with the ability to initial, include screenshots, etc. This means that the output capabilities of a tool are a huge part of the value equation. It is pointless having a wonderful tool that executes my scripts but only produces a pass/fail status as its output. Trace Matrices are one example of a common requirement for validated systems. Make sure your tool of choice supports this kind of systems validation specific functionality. Output capabilities are one of the weakest areas of most automated tools so is worth examining very carefully what effort it will take to get your documented evidence in the format that you need it to be in. Page 1 of 5
2. Paper vs. Paperless The systems validation industry may still be 90% manual but there is definite interest and movement towards paperless systems (and electronic signatures). Several tool vendors in the systems validation space have some or all of these capabilities. It is vital that regardless of whatever tool vendor you select there is some indication from the vendor of how to get from a totally manual wet-ink paper based system to a paperless one. This is realistically going to be a hybrid manual / automated system so look carefully to see that manual execution is explicitly covered (it often isn t). In an ideal world the output formats for manual or automated procedures would be the same allowing an easy transition between the two depending on the circumstances. It is easy to see a transition from paper to paperless and then building from manual to automated. No one wants to be revolutionary when it comes to getting your documented evidence. It is simply too important to take risks. 3. Requirements Management A management system for requirements is almost a requirement in itself for systems in the systems validation space. The management, traceability and documented evidence of how requirements have been tested are all key areas for life sciences to focus on regardless of whether automated tools are used. The techniques range from controlled documents to requirements management specific tools. It is a big advantage if your toolset understands a requirements driven model and can either manage or interface to an existing system. Generally getting requirements data in/out of a system is a basic necessity. This allows test scripts to be related directly to requirements without having to constantly refer spread sheets, master documents, other tools or manual lookup systems. Traceability matrices are a direct derivative of a requirements led system and any software help to simply and reduce what is often an unpleasant and time consuming task is also a bonus. The ideal scenario is one where any trace matrix output is either the same or very close to the trace matrices that you need to generate at end of project to ensure that everything has been validated successfully. If there is support for interim on the fly assessments to let you easily determine what is still to be done, so much the better. An automated tool that does not support requirements can still be used but remember to allow for the additional time to cross reference scripts and at the very least establish some kind of manual process to do this as scripts are created. 4. Electronic Signatures Electronic signatures are a fundamental requirement if you are going to create a paperless system but do deserve some thought in their own right. Look for tools that either support this functionality natively or have some kind of collaborative arrangement from another signature specialist. The only caveat is to make sure to build in any separate licence costs if the solution is multi-vendor as the additional components invariably are only available on a cost plus basis. If your selected tool does not support this functionality it may be worth asking your supplier for details development Roadmap. 5. Screenshots Page 2 of 5
Screenshots of processes in regulated industries are widely used to provide documented evidence of the validation. There are many formats and varying SOP s for how the documented evidence must appear between companies but the very step by step nature lends itself well to using screenshots as unarguable proof that process was followed. The drawback of this approach in the manual world is the time taken to capture, edit and insert the screenshots at the right time and in the right place in the script. Some tools are very screenshot capable and are able to capture and insert screenshots within standard customer supplied document templates. This kind of functionality oriented towards regulated companies but can make a huge difference in the amount of post processing that is required to get a script into an acceptable format for auditing. The capability to filter outputs dependent on risk or importance is also a benefit but if the screen capture technology is comprehensive enough this may not be a fundamental requirement as if the script execution can be generated easily and without further time penalty it won t hurt to have the full detail in the output. Some tools are also able to export tailored output that allows the script detail to be used in other ways (e.g. how to guides or crib sheets ) which can make validating or using a new system a lot easier for unfamiliar users. 6. Script Editing In the real testing world scripts are written or captured and then editing until they are ready to be considered for approval. Regardless of what toolsets you examine make sure that this editing process is readily understood and easily accomplished. If not, the best tool in the world will rapidly become a hindrance as scripts rarely work first time. Ease of use does have an impact here but that is covered separately in this document. The key factor is that if your scripts are difficult to edit your investment in tools will become shelf-ware very rapidly. Once scripts are edited and ready for execution in a regulated environment that means they require approval. The tools best suited or tailored to regulated environments will provide a mechanism to allow a user to review the script and crucially understand what the script actually does. It is important to ensure that clear documentation standards are followed or the reviewer will not be able to approve scripts with any high degree of confidence of what the script actually does. The script objective or description is not enough to judge on. The reviewer must have some means of determining what the script contents actually do. One final consideration is when scripts should not actually be editable. If a script has been created, reviewed and approved a facility to prevent further changes to that script is desirable. In this way scripts can be exported and distributed to other users without any fear that the original approval will be invalidated by changes. These are all facilities that make testing in regulated environments so much easier. 7. Ability to Automate There are procedural constraints such as the complexity of the process that you are trying to automate to consider. If the process or system is particularly complex, crosses system boundaries and/or is very dynamic in its execution path you might be best to leave this type of process to manual testing. The time taken to create the automated IQ/OQ/PQ script might be so long and run so rarely that it simply is not worth the effort of creation as an automated script. It really would be simpler to document and execute manually. This is often a hard point to get across when considerable sums of money have been invested in Page 3 of 5
automated tools, particularly as the ROI often depends on an artificially high percentage of the total scripts to be automated. Generally automation is easy to do and produces high returns on mostly linear processes. The ability to both run linear scripts repetitively and with varying test data is one of the big advantages of automated testing. Non-linear processes can certainly be automated but in most commercially available tools require low-level technical knowledge (often programming) that is neither cheap nor readily available. 8. Automation Ease of Use This is a hugely important issue in the validation sector and much more so than in the general software testing arena. The vast majority of testing tools currently available use a variety of programming languages as their underlying mechanism to drive automation. These are probably most familiar as the macro languages that you see in Word and Excel. In the general software testing arena a lot of the personnel engaged in the testing process are actually developers assigned to testing or have been specifically recruited because they have prior programming experience. This means that the claims of most tools that they are easy to use are true if the expected user is a programmer. In the validation world the validation tends to be done by subject matter experts who are system knowledgeable and are certainly not programmers. Make sure that your validation team actually understand what it takes to write scripts and don t see the high level abstract view that is often shown in demonstrations. If you need programmers to use your automated testing system make sure you have them available and have the funding to pay for them. If not, make sure you opt for one of the tools that will not require this additional resource. Non-programmers and subject matter experts find programmer tools unusable so your investment will have been wasted. 9. Technical Tool Issues Technical issues are always brought into play when considering automation of any sort and particularly with CSV due to the nature of bespoke software and hardware system. A wise evaluator will make sure that they have a qualified tool that can actually interact or drive your underlying application. This is particularly noticeable in ERP applications like Microsoft AX or SAP where the screen rendering of the main applications makes it very difficult for traditional automated testing tools to operate successfully. It is always worth making sure that the solution or tool of choice actually supports your IT infrastructure and that it does so without the need for expensive interface add-ons or support kits which devalue the financials that you are basing your ROI calculations upon. It is pointless having all the bells and whistles if at the coalface the underlying technology does not support your application. Also check that from a validation perspective you getting outputs that correspond to the requirement for documented evidence. It may not be appropriate for a single solution if your hardware infrastructure is particularly heterogeneous so be prepared to have a healthy mix to ensure coverage of all your company systems. 10. Risk Risk led validation techniques are a key means of managing the validation process where resources are typically short and in high demand. Sensitivity to risk and the ability to Page 4 of 5
categorise scripts based on this risk is important to ensure that resources know what is important and requires the most attention. There are a number of Risk management systems that allow the identification and classification of risk and once assessed, this risk needs to be traceable within the testing tools so that the scope and level of testing can be appropriately scaled to this risk. The functional risk assessment approach may not work for some systems or for companies with a low risk tolerance. Ensure that the selected tool either provides risk assessment internally, can be mapped to your own internal risk assessment strategy or you must allow for the time to do this mapping manually. Bio This white paper was created by Mark Murray of Raltus Software. Mark has over 25 years experience in the software testing industry, including the development of a variety of industry leading testing tools. Mark can be reached at m.murray@raltus.com. Raltus Software helps establish documented evidence which provides a high degree of assurance that a Computer System will fulfil its intended purpose in a regulated environment. For further details on the Raltus toolset see www.raltus.com. Product demos are available at www.raltus.com/demos. Evaluation copies are also available. Page 5 of 5