B/17/73. PACS Update for the NSS Executive Board. NSS Formal Board Meeting 29 June 2017

Transcription

1 B/17/73 NSS Formal Board Meeting 29 June 2017 PACS Update for the NSS Executive Board Purpose The Board is asked to review and consider the PACS Update report in terms of the current PACS issues and the NSS IT action plan and timescales. Recommendation The Board is asked to note the key issues, the NSS IT action plan and timescales. The action plan contains immediate short term activities to help address/alleviate the areas of risk identified by Audit Scotland and also a detailed options appraisal which will be presented to the relevant PACS, ehealth and Scottish Government governance groups to determine the best way forward for PACS in terms of Contract Re-Provisioning. The Board is asked to make comment on any additional actions that NSS IT should be taking. Timing The detailed options paper will be presented to the PACS Programme Board on 27 July and the ehealth Leads Group on 7 August with a view to creating an Outline Business Case and submitting for approval in September Irrespective of the Contract Reprovisioning option chosen, there has to be a technology refresh of the central hardware and an upgrade of the central software from v to v which could take upto12 months. The central hardware goes out of support variously between April 2018 and September Background During 2016 the PACS Programme Team/Programme Board embarked on the initial planning activities relating to a Re-Provisioning of the PACS IT Services used in NHSScotland because the Carestream contract is due to expire on 30 Jan In November 2016, operational responsibility for PACS was moved to CVSMT within NSS IT and the NSS IT Operations Director (J.Hall) initiated a technical assurance review of PACS. A number of technical, service delivery/performance, contractual and financial concerns relating to the existing PACS Service were identified. The High Level Options were presented to the ehealth Leads Group on 5th June and agreement was given to proceed to a Detailed Options Appraisal. Additionally, Audit Scotland outlined a number of PACS findings at the May NSS Audit and Risk Committee Meeting and they have incorporated a PACS Risk into their Draft Annual Audit Report to NSS. A Management Response has been provided which reflects the actions contained in PACS Update Report. Engagement Briefing provided to the NSS Chief Executive following the May NSS ARC Meeting. The Clinical Governance Committee (12 June) engaged in terms of Corporate Risk IT 4303 which relates to the possible unavailability of some PACS images in a disaster situation and possible permanent loss of images. Name of the Author: Andy Robertson Designation: Director of IT Tel: andy.robertson@nhs.net

2 PACS Update for NSS Executive Board June 29 th 2017 Version 1.0 (Issued) 1. Purpose The purpose of this brief paper is to update the NSS Executive Board on the current Picture Archiving and Communications System (PACS) issues, outline the NSS IT Action Plan and provide an update on the PACS Options Appraisal work. 2. Situation During 2016 the PACS Programme Team/Programme Board embarked on the initial activities relating to a Re- Provisioning of the PACS IT Services used in NHSScotland. A PIN, Procurement Strategy and Draft Initial Agreement were all produced based on the best information available at the time. Work currently on Hold. In November 2016, operational responsibility for PACS was moved to CVSMT within NSS IT with the purpose of bringing contract and service management activities under one organisational team to share best practice. Additionally, the NSS IT Operations Director (J.Hall) initiated a technical assurance review of PACS. This review and CVSMT due diligence activity identified a number of technical, service delivery/performance, contractual and financial concerns relating to the existing PACS Service that need to be taken into account when deciding the best way forward for PACS. At the 24 th April 2017 ehealth Leads Planning Session, J.Hall explained the need to carry out an urgent Options Appraisal which would take into account the potential solutions to address some of the short term issues as well as the Re-Provisioning considerations. The High Level Options were presented to the ehealth Leads Group on 5 th June (See Appendix 1) and there was agreement to proceed to a Detailed Options Appraisal to be completed by the end of July Audit Scotland outlined a number of PACS findings at the May NSS Audit and Risk Committee Meeting and they have incorporated a PACS Risk into their Draft Annual Audit Report to NSS. A Management Response has been provided which reflects the actions contained in this document. 3. Background 3.1 Contractual arrangements There are two contracts that relate to the PACS Service: Carestream provide the Managed Technical Service (MTS) for the application used at both national archive and local implementation levels the contract end date for the entire contract is January Atos as an additional Managed Technical Service (MTS) under the National IT Services Contract, host and manage the hardware, storage and backup for the National PACS archive the hardware for the Atos MTS goes out of support variously between April 2018 and September The Atos Contract itself ends on 31 March Current Solution Architecture: There are several components to the overall solution for acute hospitals, namely those components that are located within local Health Boards (31 local implementations) and the Central Archive component set (See Figure 1 on next page) It should be noted that all of the hardware (servers and disk arrays) for both the Local Health Board implementations and the Central Archive components was purchased in Local Health Board Components To operate effectively in real time a local PACS system, incorporating local storage, is required for each Health Board location where images are collected or reported. PACS Update for NSS Executive Board June J.Hall, G.Sommerville, K.Triner

3 SCOTTISH WIDE AREA NETWORK Figure 1 Current PACS Storage Model 2017 PULSANT Backup Site Data Centre 2 (31 SITES) Local Pacs sites have 24 months storage. 1 or 2 sites have less than 12 months due to local practice. 1 copy of Image/site Second Site (Backup) Partial DR site of 6 months rolling image data. Full copy of Global Worklist/Index which is synchronised with the Primary Site. Everything is copied to tape. No compression at the backup site. Data Centre 1 Note: Reviewed and agreed with radiology clinical lead, Clinical Advisory Group and CCLG Primary Site Full copy of all data captured at the Local Sites and is a DR site for all 31 local implementations. Global Worklist/Index keeps a record of the originating storage devices inthe hospitals. Data is compressed. Local Health Board Components (continued): Within Health Boards there are 31 local implementations of the PACS system. In each local implementation there are 2 sets of servers and RAID disk arrays. This means that there is one copy of each image within each Health Board implementation. The local disk arrays (in the 31 x implementations) theoretically should store 2 years worth of images for each site but in practice they store (on average across Scotland) about months of images. Local copying and viewing behaviour can vary this storage due to custom and practice in each Health Board between radiologists. The Health Board implementations are located in local Data Centres to various standards on Health Service property. Hardware is fully managed by Dacoll (on behalf of Carestream) with the application layers managed by Carestream. Additionally, there are about 350 physical reporting devices in Radiology across Scotland, on which the images taken in the X-Ray departments and other locations are reported. Carestream currently provide hardware support for all of the 350 devices (via Dacoll) but some Health Boards are now seeking to provide support for these local devices themselves (to save money). The Carestream PACS client reaches a further estimated 15,000 devices in wards and clinics which are used mainly for viewing and some of these will be used for diagnosis and reporting (eg.ultrasound and orthopaedic work). These are supported wholly by Health Board IT and are not under the Carestream contract Central Archive Components: The Primary site of the Central Archive is a Disaster Recovery site for all of the 31 x local implementations. The Primary site is located in the Atos Data Centre at Appleton Parkway, Livingston and the Secondary site is in an Atos controlled part of the Pulsant Data Centre at the Gyle. In the Primary Site, the data is largely in synch with the local sites but can be several minutes out of date for reasons of transmission between the local implementations and the Central Archive. There is also a Global Work List (Meta Data) which keeps a record of the originating storage PACS Update for NSS Executive Board June J.Hall, G.Sommerville, K.Triner

4 device in the hospitals. If the Primary site was ever to go down, then a Local Site can access the copy of the Global Work List from the secondary site, find the originating location for an image and retrieve it from the Local site storage for viewing in another Health Board site. This function is limited to the number of months images stored at a Local Site. Current data volume of images is nearly 1 Petabyte, this being all of the data transferred from V10 of the software from 2006 onto V11 and the data accumulated on V11 since July It should be noted that each generation of MIR and other imaging devices take more dices and slices of patients and the use of these devices, along with the single image devices is growing significantly on an annual basis. The 1 Petabyte of data is only going to increase in the years to come as devices are run 24 x 7 and imaging becomes more sophisticated. The Secondary Central Archive site for V11 was built as a partial replica of the Primary Central Archive site. It holds 6 months of pixel (image) data copied across from the Primary site plus the data that has not yet been copied to tape. In addition there is a Global Work List (Meta Data) which is synchronised with the GWL held in the Primary Central Archive. The Secondary Central Archive is both a limited pixel DR site for the Primary site and a staging area for the transfer of the Central Archive data onto tape. This decision was taken to save money. The previous version of the system implementation (V10) used the Secondary site as a full replica of the Primary site. In practice, however, the Secondary site does not hold the most recent six months worth of pixel data. The transfer of data from this site to tape has been affected by various technical factors since its inception in July 2015 and these are attributable to both suppliers. 4. Summary of Key Issues/Risks 4.1. Contract and Service Management Carestream Service Management support has been mixed over the years and this has recently resulted in a number of incidents that have affected both patient care and the effectiveness of the overall PACS service. In addition, the Contract with Carestream does not contain significant provisions around performance management. There is a need to review the approaches to contract and service management with Carestream, clarify all contractor responsibilities and incentivise performance where possible Historical decisions about different aspects of the PACS service (contract extension vs hardware purchased date/supportability duration) have been made on a compartmentalised basis without fully understanding the knock on impact to other areas of the service; For example, the contract was extended from Jan 17 to Jan 2020 without taking cognisance of H/W coming out of support in At the time of writing this document it is not yet clear how the budget for the work outlined below will be financed. However, avenues for funding and profiling are being explored National Archive Storage There is no provision for full replication of images held on the PACS Central Archive (the normally preferred method for assuring data availability for critical clinical systems under the National IT Services Contract) the Central Archive is, in essence, a single national disk based data store for all patient images with a combination of disk and tape at the second site. Given the fact that the tape solution is way behind where it should have been in terms of backup then this represents an effective single point of failure if the Primary Site was lost then went off-line there would be no immediate access to images that are only stored on the national archive and if it failed then there would be no practical way to build the Primary Site from an incomplete tape library. Even if this tape library was up-to-date it would take a prohibitive amount of time to restore all of the images from tape therefore the whole system would have failed There is no contractual provision with Atos for periodic technical refresh of the Central Archive hardware (either Servers or Disk Arrays) and all of this hardware is nearing the end of its practical support life. The hardware goes out of support variously between April 2018 and September PACS Update for NSS Executive Board June J.Hall, G.Sommerville, K.Triner

5 The tape backup solution has not operated correctly since its inception in July 2015 and there is backlog of images which have not been copied to tape. Indeed, it is understood the tape backup arrangements for V10 did not operate properly either The version of the PACS software currently used throughout Scotland (V on the Central Archive and at the Health Boards) is reaching the end of its support life. Additionally the V software on the Central Archive is not stable and requires periodic software resets. Carestream only support one other Central Archive site on earth at V and this is currently being moved to V12 of their software (reported expected completion date December 2017). Consequently, Carestream recommend that NHS Scotland move onto V for the Central Archive (64 bit version of their software that can use more powerful hardware). This is a non trivial task and would take a minimum of 9 months to upgrade the Central Archive. In addition, this would require the local sites to transfer to V to enable them to be used in the same configuration. This latter change is a relatively easy software upgrade that could be undertaken in a few hours per site Local Site Storage A small number of the Local Site Stores at territorial boards no longer have the capacity to meet the requirement of storing months of local activity. As a minimum, each local cache should be able to store all images taken, or referred to, during the last 12 months. A varying number (between 1 and 2) of these 31 local stores do not meet this minimum requirement (some of these can only store 9 to 12 months worth of local activity which increases the reliance upon, and use of, the national archive for access to current patient activity).local copying and viewing behaviours can vary this storage and this is due to custom and practice. Growing workloads and advances in technology in image capture since the contract was set in 2005 have also exacerbated this situation. This is a Health Board risk There is currently no planned technical refresh of the local systems, which also continue to operate using V11 of the PACS software (local PACS systems must operate on a compatible version to the Central Archive). This is a Health Board risk which has been mitigated in part by extending the support SLA with Carestream to January The intention was to sweat the assets towards the end of the contract in January If the Carestream contract is extended further then this risk will need to be revisited by the Boards and technical refresh will need to be re-considered. 5. Action Plan A key timeline driver is the need to have technology refresh and support arrangements agreed before the current hardware support arrangements for the national archive end (variously between April 2018 September 2018) or as soon after these dates as practically possible. NSS IT has created the following action plan which will be progressed over the next three months Agree a joint plan (NSS/Carestream/Atos) to address the tape backup issue End of June KPMG audit of PACS IT Resilience and PACS IT Disaster Recovery June Identify the end-to-end storage position (Central Archive and Local Implementations) and highlight any risks to PACS Governance Groups/Health Boards End of July Review the approach to contract and service management with Carestream with a view to clarifying all contractor responsibilities and incentivising performance where possible. This work will include a Scott Moncrieff audit of Carestream Change and Release Management September 2017 PACS Update for NSS Executive Board June J.Hall, G.Sommerville, K.Triner

6 5.5. Produce a Detailed Options Appraisal to determine the way forward for PACS End of July 2017 followed by an Outline Business Case September 2017 (See Appendix 1 for a summary of the High Level Options). Appendix 1 PACS Options Appraisal Update A high level options appraisal relating to the PACS Way Forward and PACS Re-Provisioning was completed at the beginning of June and the recommended next steps agreed by the ehealth Leads Group on 5 th June Key Messages Existing contractual arrangements expire in January 2020 (Carestream) and March 2022 (MTS for the National Archive through Atos extendable to 2026) The NSS IT technical assurance review of PACS (Nov-March) identified a number of technical, service delivery, contractual and financial concerns/issues Eg. Support for the central hardware purchased in 2012 (servers and 1 PB storage)expires between April & September 2018; Historical decisions about different aspects of the PACS service (contract extension vs hardware purchased date/supportability duration) have been made on a compartmentalised basis without fully understanding the knock on impact to other areas of the service; There are two important activities that have to take place no matter which Re-Provisioning Option is chosen. These are: I. Carry out a technical refresh of the National Archive equipment II. Improve the Contract and Service Management arrangements. There are 3 main procurement options, each with 3 different service delivery sub-options. 1)Single Supplier Extension of Carestream Contract for 1 to 3 years and then Re-Procure a Full PACS Service (OJEU) 2)Re-Procure Full PACS Service )OJEU) 3)Include PACS Service into National IT Services Contract with Atos plus A)Existing MTS Service Delivery Model o PACS Service Provider at Local Sites o Atos Data Centre for both Central Sites B)Change the MTS Service Delivery Model o PACS Service Provider at Local Sites o Atos Data Centre for 1 Central Site o PACS Service Provider for 1 Central Site C)Change the MTS Service Delivery Model o PACS Service Provider at Local Sites o PACS Service Provider for both Central Sites Next Steps and Current Timetable The following next steps have been agreed by the ehealth Leads Group. 1 Carry out a Full Options Appraisal Target Date: End of July 2017 a. Review with PACS Programme Board 27 July b. Review with ehealth Leads 7 th August 2 Produce an Outline Business Case Target Date: September 2017 a. Review with PACS Programme Board b. Review with ehealth Leads 4 th September c. Agree with ehealth Delivery Board (replacement for ehealth Strategy Board) in September. PACS Update for NSS Executive Board June J.Hall, G.Sommerville, K.Triner

7 NB. Irrespective of which re-procurement option is chosen the technical refresh of the central hardware and the upgrade from v to v of the central software is required. PACS Update for NSS Executive Board June J.Hall, G.Sommerville, K.Triner

8 Picture Archiving and Communications System (PACS) Update Andy Robertson NSS IT Director NSS Executive Board 29 June 2017 Version 1.0 (Issued)

9 PACS overview Ongoing business requirements/ current issues, risks, activities Business Strategy Local PACS BAU service management Farr Institute Image repository National archive resilience/ DR Cross border reporting capability Pathology Images system PACS contract reprocurement National Archive BAU service management Adverse event lessons learned Technical refresh Radiology BI/ analytics Extended clinical use other specialties RIS contracts renewal Digital mammography Supplier practices, controls, etc Multiple supplier/ no lead supplier Enhanced Voice reporting capability Data migration challenges Cloud storage/ VNA Technical Strategy Options Assessment criteria Governance Structure and Decisions Contractual mechanism Funding allocation Roadmap Delivery

10 SCOTTISH WIDE AREA NETWORK Current PACS Storage Model 2017 PULSANT Backup Site Data Centre 2 (31 SITES) Local Pacs sites have 24 months storage. 1 or 2 sites have less than 12 months due to local practice. 1 copy of Image/site Second Site (Backup) Partial DR site of 6 months rolling image data. Full copy of Global Worklist/Index which is synchronised with the Primary Site. Everything is copied to tape. No compression at the backup site. Data Centre 1 Note: Reviewed and agreed with radiology clinical lead, Clinical Advisory Group and CCLG Primary Site Full copy of all data captured at the Local Sites and is a DR site for all 31 local implementations. Global Worklist/Index keeps a record of the originating storage devices in the hospitals. Data is compressed.

11 Key Issues & Risks No provision for full replication of images held on the PACS Central Archive No contractual provision for periodic technical refresh of the PACS National Archive storage and the hardware is nearing the end of its practical support life The PACS National Archive software has reached the end of its support lifecycle and a significant upgrade has been recommended In the absence of full replication of images there is a solution in place to incrementally archive images to tape but this is not currently operating properly A small number of the Local Site Stores at territorial boards no longer have the capacity to meet the requirement of storing months of local activity Carestream Service Management support has been mixed over the years and recently resulted in a number of incidents that affected patient care. In addition, the Contract does not contain significant provisions around performance management. Full budget provision to over the anticipated cost of addressing these issues has still to be agreed. Audit Scotland Risk (Extract from Appendix 1, Draft Annual Report) If the PACS national archive fails, patient image records may be lost. Audit Scotland Recommendation (Extract from Appendix 1, Draft Annual Report) The Board should ensure that robust arrangements are in place to ensure business continuity and system resilience for PACS.

12 Action Plan 1. Agree a joint plan (NSS/Carestream/Atos) to address tape backup issue June KPMG audit of PACS IT Resilience and PACS IT Disaster Recovery June Identify the end-to-end storage position (Central Archive and Local Implementations)and highlight any risks to PACS Governance Groups/Health Boards July Review the approach to contract and service management with Carestream. Work to include a Scott Moncrieff audit of Carestream Change and Release Management September Produce a Detailed Options Appraisal to determine the way forward for PACS End of July 2017 followed by an Outline Business Case September 2017.

13 Background Slides

14 PACS Background Carestream PACS contract initially 10 years from 31/1/07 and recently extended to 30 January Access Agreements entered into between all Territorial Health Boards, the Golden Jubilee National Hospital and Carestream. All end simultaneously at the end of the PACS Contract on 30 January ATOS provide storage, infrastructure and hardware in the data centres. This contract has been extended to 2022 RIS is provided by either Carestream, HSS or TRAKCARE NHSS has 15 RIS systems across Scotland and the contracts are at Health Board level. PACS is a local install on 31 hospital sites and 2 central data centres (1 primary the other partial backup and tape), PACS/RIS integrate on 350 diagnostic workstations around the country and on around 15,000 PCs and tablets across wards in all acute hospitals, PACS currently has an archive of 28m studies nationally and 3.2m studies are added every year, During the working week 15k studies are produced and ingested per day, There are 23k registered clinical users, NHSS recently finished a full refresh and version upgrade to PACS V11 this took 2.5 years.

15 PACS Governance NHS Board Internal Goverance Chief Executive Group Clinical Change Leadership Group Scottish Government Health Directorate Board ehealth Oversight Group ehealth Leads ( NHS Board ) ehealth Delivery Board Board and Consortium Programme and Project Boards Portfolio Management Groups Existing Systems NO LONGER EXISTS National Systems Programme and Project Boards PACS Programme Board (Chair Dr Hamish McRitchie) Focus on Strategy and Direction of travel since March 2016 SLA/Performance Management PACS User Group PACS Clinical Advisory Group RIS User Group Group

16 PACS NSS IT Corporate Risk IT 4303 Risk Description: There is a risk that due to PACS Backups not working at sufficient speed - backlog approximately 12 months and growing. Some PACS images >12 months old will soon exist only at ATOS Livingston, after deletion at local health board. For around 1m studies there is only one copy. The risk detailed above could result in possible unavailability of some PACS images in a disaster situation, possible permanent loss of images. Current Status: Original risk raised in quarter /17 as a red risk and categorised as Clinical (secondary-business). Lorna Ramsay (NSS IT Medical Director) presented an SBAR relating to the above risk to the NSS Clinical Governance Committee on 12th June 2017 The recommendation was to change the categorisation and reduce the risk scoring as NSS does not itself use PACS to deliver its services. The clinical risk resides with the Boards. Primary category changed to Business, Secondary category to Reputational and scoring reduced to amber (12). CVSMT were asked to liaise with CLO to obtain clarification on NSS s position/liability should images become unavailable or permanently lost. The advice is summarised below: Reputation Issue if any central archive images become unavailable or permanently lost Data Protection Issue if a loss of data causes a breach of data protection which could expose NSS to a fine imposed by the Information Commissioner (currently limited to 500k but could increase). NSS may be in a position to back off any financial liability it may incur as a result of loss of data. NSS could have a potential claim against Carestream and/or Atos if NSS could establish that the relevant party was in breach of an obligation under its contract and it was this failure which resulted in loss/liability being incurred by NSS. Medical Negligence risk where loss of data is a contributing factor in a medical negligence claim. This is an NHS risk rather than a specific NSS risk.