National Video Conferencing Project Final Report. Version 1.0 December 2013

Transcription

1 National Video Conferencing Project Final Report Version 1.0 December

2 1 Executive Summary IP based video conferencing is now well established across NHS Scotland with 95% of calls made using IP. Around 900 devices are registered with the central management platform, providing call logging, management and directory services. The National Video Conferencing Service has been established, with staff based in Inverness, Aberdeen, Edinburgh and Glasgow. A single point of contact service desk has been set up using contact centre technology and call logging. The service is currently handling around 160 phone contacts and 150 contacts per week. This has generated approximately 290 service desk tickets per week. A shared bridging service has been established to manage and support multisite meetings. The service is operational in 20/22 health boards. Work is ongoing with the wider public sector, in particular with Criminal Justice, to develop common standards with a view to providing shared services. Agreement has been reached with both the NHS VC Steering Group and the Criminal Justice VC Strategic Board to develop a proposal to provide VC services to Justice Partners from 2014/15. Call volumes currently stand at around 60,000 video calls / quarter. The use of SIP video conferencing (primarily the Cisco Jabber desktop client) has increased significantly) and now accounts for approximately 35% of the total call volume. Customer satisfaction remains high, with 97% or respondents rating the overall service as good or very good. 2 Background In October 2009, the Regional Planning Chief Executives sub group tasked NSS with improving video conferencing. Working in conjunction with the Scottish Centre for Telehealth & Telecare (SCTT) and the North of Scotland Planning Group, the National Video Conferencing Project was established. Leading a consortium of the 6 North of Scotland NHS Boards the project aimed to pilot a pan-scotland approach to Video Conferencing Services running over the N3 network. Following discussion with ehealth leads it was clear that NHS Boards that had already invested in video infrastructure had little appetite to embark on a pilot involving a rip and replace of existing infrastructure. It was therefore agreed that an organic development of existing infrastructure was the preferred route. This was perceived to provide the most cost effective and sustainable solution. 2.1 Project Objectives The original objectives of the project were to undertake a Technology Pilot, serving 50 video conferencing endpoints, across the 6 NoS NHS Boards and to develop an Outline Business Case (OBC) for full rollout. The Technology Pilot was to be trialled by the Northern Boards, the North of Scotland Planning group and key clinical users and would include: 2

3 Standard Definition Video capabilities over N3, with appropriate firewall traversal; Devolved User Provisioning for booking video conferences; Remote management of video devices; Aggregated Directory of video devices; Visibility of MCU capability. Depending on the outcome of the pilot, an OBC would be developed for full implementation. 3 Development of Standards When planning the initial technology rollout, it became clear that agreement between boards was required in a number of areas. These included: Naming conventions Dial plan Firewall traversal IP addressing Rather than limiting discussion with the NoS Boards, a wider consultation including key stakeholders and ehealth infrastructure leads from across Scotland was undertaken. A standards based approach was agreed that allowed Boards to build on previous investment and these standards were formally adopted by NHS Scotland in September 2011 and have been published, as follows: In developing the standards, it was accepted that there was a need for local flexibility of infrastructure. This standards based approach has allowed boards to implement the standards on existing infrastructure hence maintaining existing service provision and keeping costs low. This fits with ehealth policy of growing services organically and with the recommendation from McClelland 1 to first re-use, then buy and build only as a last resort. 3.1 Collaboration with JANET (Joint Academic Network) JANET provides video conferencing services to higher and further education, research bodies and schools. As part of the wider global video network, JANET links to the Global Dialling Scheme (GDS). Rather than develop a dial plan restricted to NHS Scotland, it was agreed that the scheme developed should link with JANET and the global network beyond. This provides each video endpoint registered within NHS Scotland a unique numerical ID. Security and policy decisions permitting, this can provide the basis for developing video conferencing links across the public sector in Scotland as well and providing onward links across the world. 1 Report by John McClelland C.B.E. on his Review of ICT Infrastructure in the Public Sector in Scotland 3

4 4 Technology Rollout 4.1 Technology Pilot During the early stages of engagement with local NHS board video conferencing and network leads it became apparent that attempting to run a pilot limited to 50 systems would cause severe operational difficulties as it would result in an incoherent numbering scheme, a multitude of management systems and a requirement to provide duplicate infrastructure for the two distinct video networks. Importantly, it was recognised that the level of service provided to video conferencing users would be unacceptable and undermine the project objectives. Having completed the tendering process for the procurement and agreeing the basic technology design, discussion with the manufacturer indicated that they would be willing to transfer existing software licenses to the new setup. The participating NoS Boards and the Project Board agreed that the pilot should proceed on that basis, with a full rollout across the North, increasing the project scope from 50 to approximately 400 endpoints. Although this had a major impact on the project schedule, costs were not increased as additional work was undertaken by the participating boards in light of the perceived benefits. 4.2 The Initial Rollout The initial rollout was made up of a number of tasks: Firewall and local network configuration to support the IP addressing scheme; Renumbering of video conferencing endpoints to comply with the agreed dial plan; Reconfiguration of endpoints to comply with the naming convention; Reconfiguration of video conferencing gatekeepers to establish the hierarchical network; Migration of existing management systems to the central platform. While much of this work proceeded as planned, there was a delay in the rollout out of the management system, possibly due to corruption in the underlying database imported. As a result, the management system was re-installed in June Since this time the database has been performing well and system migration in the north has been completed. 4.3 Increase of Scope Following agreement form the Regional Planning Chief Executives Group, it was agreed that the scope of the project should be expanded to include a pilot of staff video links from home and that other NHS boards should be allowed to access the service. As a result, further funded of 15k was obtained from Scottish Government to provide the necessary hardware. This was procured and installed in early With the increasing utilisation of the central service, the project board agreed that additional funding of 72k should be sought to provide additional capacity and fault tolerance for the central gatekeeper. This was provided by NHS 24, through the Scottish Centre for Telehealth and Telecare. A fault tolerant cluster, spread over the Atos and ScoLocate data centres was installed and became operational in Engagement with other NHS Boards Throughout the project, great weight has been placed on ensuring that Health Boards outwith the pilot area were fully engaged in the process and were able to comment and influence the 4

5 standards developed. This was achieved through regular updates and consultation with both the ehealth leads and ehealth infrastructure leads groups. In January 2011, a stakeholder event was organised to both discuss and review the proposals for the video service. This event was well attended by representative from almost all NHS boards throughout Scotland and there was unanimous agreement on the proposed development of the service. 4.5 National Rollout Following the installation of the gatekeeper cluster and the provision of additional call volume licences in 2011, the remaining health boards in Scotland were invited to migrate to the N3 video conferencing network. Figure 1 Uptake of Management System Licences shows the migration of systems to the national network and subsequent registration on the central TMS management system. Figure 1 Uptake of Management System Licences As of mid 2013, all Boards have adopted the VC standards and have devices migrated to the N3 VC network. A summary of uptake is outlined below. Table 1 VC Service Migration Status September 2013 Board Ayrshire and Arran Borders Dumfries and Galloway Fife Forth Valley Devices Comments on TMS 36 Complete 5 Complete 30 Complete 13 Complete 4 Most devices are connected to a local management system. A project to migrate the systems to the 5

6 national TMS is underway. All devices are accessible via N3 using the national dial plan. Golden Jubilee 9 Complete Grampian 144 Complete Greater Glasgow and 45 Complete Clyde Health Scotland 2 Complete Highland 230 Complete HIS 15 Complete Lanarkshire 9 Complete Lothian 69 Complete National Services 50 Complete NES 32 A significant number of NES systems are still to be migrated to the VLAN. This is dependent on local network implementations. All devices are accessible via the national dial plan via the Grampian VCSX border controller. NHS 24 9 Approx 10 systems are still to be migrated to N3. These are currently only accessible via ISDN Orkney 21 Some GP sites on ISDN due to limited bandwidth available. Scottish Ambulance 7 Complete Service Shetland 15 Some GP sites on ISDN due to limited bandwidth available. State Hospital 6 Complete Tayside 83 Complete Western Isles 3 Most systems are still to be migrated to N3. These are currently only accessible via ISDN. Total 837 As of 15/11/13 there are 873 devices registered. These are categorised as follows. Table 2 Registered devices by Category Unknown 1 Endpoint 836 Gatekeeper 12 Gateway 7 MCU 7 Room (endpoints not recognised by TMS) 10 Grand Total 873 6

7 4.6 Current Video Conferencing Capabilities All boards are able to make and receive IP video calls from H323 devices on the national VC service. H323 devices are also able to receive calls from SIP users, with conversion undertaken by the national infrastructure. This is transparent to the end user. Desktop video conferencing is available through the nationally administered Cisco Jabber service. (Support for Polycom desktop clients in also available.) Staff members are able to video conference outside the office using secure internet connectivity. Outbound dialling has been established to allow NHS VC units to dial out to any routable address. A national management system is in place allowing remote management of the majority of NHS VC endpoints. This supports the provision of real time VC directories both on the web and directly to endpoints. 4.7 Next Steps Under the direction of the National VC Steering Group and Technical Advisory Group, a number to technical developments are underway or anticipated (subject to approval). Online Booking Service work is currently underway to pilot a token based authentication service that would support the development of a web based booking service. This is being led by NHS Tayside. WebRTC Technology consideration is currently being given to implementing a browser based VC service. This would be particular useful for communicating with patients in their homes as it would allow a one time use web link to be to an external user to allow them to participate in a meeting using a web browser. Integration with Unified Communications consideration is also being given to upgrading central infrastructure to better support links to unified communication systems such as Lync 2013 and Cisco Jabber IM and Presence. 5 National Video Conferencing Service As part of the original project brief to develop an OBC for full implementation the project team proceeded to work with colleagues in EHealth to develop the case for establishing a National Video Conferencing Service that would provide a central support service and multi-site bridging service. 5.1 Aggregation of MCU Resources A paper on Options for Improving Video Conferencing Bridging Infrastructure was presented to EHealth Infrastructure Leads and EHealth leads in June This set out the cost savings associated with aggregating MCU (bridging) capacity rather than adopting a board or regional approach. Based on the paper, agreement was reached to procure a 30 port MCU to provide appropriate capacity (along with existing MCUs already in service) to support a national bridging service. 7

8 5.2 Option Appraisal Also presented to the June Infrastructure Leads and EHealth Leads meetings was an Options Appraisal on the provision of video conferencing services. This compared the cost and benefits of doing nothing, developing an in-house national service and outsourcing over a 3 year period. Based on the benefits of a national service and the significant cost savings over an outsourcing option, the project team were tasked with developing a full specification for the in-house national service. 5.3 Service Specification The service specification was completed and a fully costed model presented to the EHealth leads in December This was accepted and full funding for 2013/14 awarded. Based on ITIL principles, the key elements of the service include: A service desk providing real time support for video meetings; Remote support of video conferencing hardware; Support for video conferencing software such as Cisco Jabber Video; National bridging services to support multi-site meetings. 5.4 Rollout of the National VC Service Funding for the national service was available from April Unfortunately this coincided with a number of staff resignations and absences resulting in a delay of the formal service launch. Following recruitment to key posts, the resolution of HR issues, IT installation and reaching agreement on key operational processes the service was formally launched in May 2013 covering Grampian and Highland. The rollout was extended to the remaining boards throughout the summer. Utilising contact centre technology and the LANDesk help desk software available over N3, the service operates from 4 locations in Inverness, Aberdeen, Edinburgh and Glasgow and is available from Monday Friday 8am 6pm. Table 3 Service Desk Rollout Status (November 2013) Board Grampian NHS Education for Scotland Highland Fife Orkney Shetland NSS Healthcare Improvement Go Live Progress / Comments Date 27/5/13 Service operational 27/5/13 Service operational (NES calls going to service desk due to preexisting NES / Grampian contract.) 12/6/13 Service operational 5/8/13 Service operational 5/8/13 Service operational 19/8/13 Service operational 19/8/13 Service operational 19/8/13 Service operational 8

9 Scotland Golden Jubilee Nat Hosp Dumfries & Galloway Greater Glasgow & Clyde (Excl Beatson) Borders Tayside NHS24 Ayr & Arran Lothian Health Scotland Lanarkshire State Hospital Forth Valley GGC Beatson Scottish Ambulance Service Western Isles 26/8/13 Service operational 27/8/13 Service operational 30/8/13 Service operational 1/10/13 Service operational 1/10/13 Service operational 10/10/13 Service operational on registered systems. Waiting on NHS 24 to migrate remaining systems to N3. 14/10/13 Service operational 14/10/13 (limited) Support staff in Lothian working as part of national team. Due to on-going issues regarding a major upgrade of NHS Lothian infrastructure, it has not been possible to provide a fully integrated service From 14/10/13, calls have been logged on the service desk system and cross training is underway. 21/10/13 Service operational 28/10/13 Service operational 18/11/13 Service operational 25/11/13 Go live date set for Monday 25 th November. Most units on local rather than national TMS. Project to be set up to migrate units to vlan and national TMS. TBA Agreement reached on transfer of conferences to national bridge. Project being established to ensure orderly migration. TBA Meeting held, agreement in principle reached but to be discusses at SAS management team. Awaiting confirmation of start date. Hold Initial planning meeting held. Jon Harris advised unable to provide support till equipment migrated to IP network. New device registered to TMS 8/8/13 Five units currently on TMS. 9

10 6 Next Steps 6.1 Collaboration Across the Public Sector With support from the Scottish Centre for Telehealth and Telecare, significant progress has been made towards improved VC communication across the public sector. Working with the Criminal Justice VC Project, draft standards for video conferencing (in line with current NHS Standards) have been agreed by the project group (Scottish Prison Service, Police Scotland, Scottish Courts Service, Crown Office and Procurator Fiscals, Scottish Legal Aid Board and the Mental Health Tribunals Service). It is hoped that this can be developed under the SWAN umbrella, to establish VC communication standards across the whole public sector, however this is dependent on the setup of an appropriate governance group to complete and ratify the standards. The introduction of common standards will also allow for the development of shared services such as the provision of as Service Desk and bridging. Agreement has been reached with both the NHS VC Steering Group and the Criminal Justice VC Strategic Board to develop a proposal to provide VC services to Justice Partners from 2014/15. 10

11 7 Appendix 1 Review of VC Call Data This section contains a review of key statistics generated from the TMS Analytics application. Figure 2 Total Number of Calls, shows the increase in calls since the national TMS service became operational in The steep incline in 2011 is largely a result of units being added to the service, however the increase seen in 2013 represents an increase in call volumes. It should be noted that a call is defined as a single point to point link. A multisite conference with 3 participants would therefore be recorded as 3 calls with each participating unit calling the MCU. Total number of calls therefore represents the overall activity. Figure 2 Total Number of Calls 11

12 Figure 3 shows the total duration of calls by quarter. This mirrors the call count statistics as the average duration of calls is relatively constant over time at around 25 minutes. Figure 3 Total Call Duration 12

13 Figure 4 Call Count by Protocol, provides a breakdown of the total call count by H323, SIP and ISDN. It is noticeable that there has been a significant increase in the volume of SIP traffic. This is due to an increase in the use of the Jabber desktop client. Figure 4 Call Count by Protocol Table 4 Call Count by Board, provided a breakdown of the calls make from each board by quarter. It should be noted that this information collated by the TMS Analytics tool records data collected from Cisco gatekeepers registered to TMS. Devices registered on other gatekeepers have only been counted if the call traverses the central gatekeeper cluster. As a result call statistics are under reported in these areas. 13

14 Table 4 Call Count by Board (* Known under reporting) Grand Total Not in Folder (ie Jabber) Golden Jubilee Nat Hospital Healthcare Improvement Scotland National Bridging Service National Infrastructure NHS Ayrshire and Arran* NHS Borders* NHS Dumfries & Galloway* NHS Education Scotland NHS Fife* NHS Forth Valley* NHS Grampian NHS Greater Glas & Clyde NHS Health Scotland NHS Highland NHS Lanarkshire NHS Lothian* NHS NSS NHS Orkney NHS Shetland NHS State Hospital NHS Tayside* NHS Western Isles NHS Scottish Ambulance Service Grand Total

15 Table 5 provides information on calls to and from SIP devices by board. Calls made using SIP are predominantly using the Cisco Jabber desktop client. Table 5 Total Call Count for SIP provisioned Systems (Cisco Jabber and SIP based E20s) Non Provisioned Systems National Jabber Service Grand Total Non Provisioned Systems* National Jabber Service NHS NHS Western Isles Scottish Ambulance Service NHS Lothian NHS Shetland NHS Tayside NHS NSS NHS Healthcare Improvement Scotland NHS Highland NHS Lanarkshire NHS GGC NHS Grampian NHS Fife NHS Health Scotland NES: NHS Education Scotland NHS Forth Valley NHS GJNH NHS Borders NHS Orkney NHS Dumfries and Galloway National Jabber Service Grand Total * Non provisioned systems are all systems not set up using SIP provisioning, ie standard endpoints. 15

16 8 Appendix 2 Review of National VC Service Desk Performance Table 6 Contact Report provides summary information from the contact centre system. Table 6 Contact Report Inbound Calls Calls Answered Calls Abandoned (Post Welcome Msg) Avg Answer Time Longest Wait s Contacts 3rd June s 40s 10th June s 1m 10s 17th June s 8m 30s 24th June s 2m 30s 1st July s 55s 8th July s 36s 15th July s 40s 22nd July s 38s 29th July s 2m 14s 5th August s 1m22s 12th August s 37s 19th August S 1m 5s 26th August s 42s 2nd September s 40s 99 9th September s 23s th September s 4m 48s rd September s 33s th September s 4m 41s 124 7th October s 35s th October s 2m 19s st October s 43s th October s 58s 203 During September and October, call volumes averaged 160 phone contacts and 150 contacts per week. Holiday periods are noticeably quieter. 16

17 Table 7 Service Desk Report LANDesk Calls Logged (Total) LANDesk Calls Breached SLA % LANDesk Calls Breached SLA 2nd September % 9th September % 16th September % 23rd September % 30th September % 7th October % 14th October % 21st October % 28th October % During September and October, the average number of calls logged was 288 per week. 17

18 9 Appendix 3 Customer Satisfaction Survey An on-going survey of service desk users was set up in October Data analysed here represents responses collected up to the 12 th of November Basic Information A total of 104 responses were received, 40% from NHS Highland and 20% from NHS Grampian. The remaining responses were spread over a further 14 health boards. 54% or respondents classified themselves as working in administration, but only 16% of meetings were classed as administrative,. This echos the perception that a significant proportion of meetings are arranged by administrative staff for others. Table 8 Meeting Type (n=61) Admin Meeting 16% Senior Management Meeting 43% Education / Training 16% Case Conference 2% Clinical Networks 7% Multidisciplinary Team Meeting 13% Clinical Consultation involving the patient / carer Clinical Consultation using video conferencing to undertake diagnosis 3% 0% 9.2 Contacts The main method of contacting the service desk is via with 65% of respondents using this method with 32% using the phone and only 3% via the web site. It is anticipated that as on-line services are developed, these proportions will change. Table 9 Quality of Service Received via Phone N=34 Very Good Good Neither Good nor Bad Bad Very Bad Time taken to answer your call Helpfulness of call handler 44% 52% 3% 0% 0% 61% 36% 3% 0% 0% 18

19 Ability of call handler to resolve your issue 48% 46% 3% 3% 0% Speed of resolution 48% 42% 3% 6% 0% Overall opinion of the service 54% 39% 3% 3% 0% Table 10 Quality of Service Received via N=62 Very Good Good Neither Good nor Bad Bad Very Bad Time taken to answer your Suitability of the response Usefulness in resolving your issue 74% 23% 3% 0% 0% 79% 20% 2% 0% 0% 73% 23% 5% 0% 0% Speed of resolution 71% 24% 5% 0% 0% Overall opinion of the service 79% 19% 0% 2% 0% 9.3 Purpose of Call The purpose of the contacts to the service desk were as follows: Table 11 Purpose of Call to the Service Desk (N=86) To report a problem with a live video conference 36% To report a VC problem not currently affecting a live conference 7% To request or alter a bridge booking 54% To request a service (other than bridge bookings) 4% 9.4 Technical Problems 29 survey respondents indicated that they had a technical problem during their meeting. Faults were described as follows: Table 12 Fault Categorisation (N=29) 19

20 Difficulty starting the equipment 0% Problems connecting to the remote site 34% Unknown dial in details 0% Missing participants 17% Sound problems 21% Video quality problems 21% User error 3% Other 28% Of these, 76% of issues were fixed on the phone by the service desk, 10% fixed by on-site support staff and 3% fixed it without support. In only 10% of cases was it not possible to resolve the issue. This is reflected in the meeting outcomes: Table 13 Meeting Outcome (N=28) Meeting abandoned 11% Meeting significantly disrupted 28% Meeting proceeded with minor disruption 61% 9.5 Free Text Comments or Suggestions I am a PA and book many video conference bridges. In the past the organiser of the bridge was sent a copy of the vc booking form, this is no longer sent (just an automated to the organiser of the bridge). Would it be possible to reinstate this as it's helpful to participants to know who else is vcing into the meeting and where they will be. Thanks If there have ever been any problems or issues these have been sorted out very quickly and I find the service very useful and helpful. Thanks Very polite and helpful staff. Staff were very helpful, although the problem could not be rectified quickly enough for our meeting meaning we had to miss out on one of our participants it was addressed swiftly Would be helpful on s confirming the VC bridge that actual VC units that are booked are also identified on the confirmation . I VC quite often and find the picture and sound can be very hit and miss, i.e. unreliable which is frustrating as important information can be missed. Therefore tendency is to try and travel if I can. Not a good idea in the winter months 20

21 Very useful service which has always worked efficiently The service was very good and the operators were extremely helpful. Local staff know me and know the specifics of my project so I find them more efficient as I don't have to explain what I am trying to achieve. All staff lately have been most helpful. Very pleased with the service. Quick, efficient, supportive. Would not hesitate to approach the team for assistance. It would be helpful for information showing the sites being connected to the v/c bridge could be returned with confirmation as this is very helpful for those attending the v/c meeting. The current reply does not show all sites which means more work for the person arranging the meeting. The previous system we had with the booking form being returned with the TMS reference number was good. However, I have to say that the current service is excellent if only the above suggestion could be incorporated in some way to make it even better, in my opinion. The service I received was fantastic - extremely quick and efficient. When confirming bridge bookings it would be really helpful to include the original bridge request form to allow other users to check that they have been included and that their details are correct; a double check for peace of mind. Sometimes as a PA although you have confirmed VC details, these can change and you are then unsure when the confirmation comes through whether the change has been made. I received a response very quickly and the staff were very helpful I think problems arise when people add VC sites at the last moment. I have not had problems with one to one. I've used this service a few times and its really excellent. Normally, the VC service is superb, adaptable, courteous and accommodating. Instructions could maybe be clearer onsite Finding the support for VC connections has improved and everyone is very helpful and understanding when we unfortunately need bridge facilities at last minute --- Thank you! I think its a really brilliant service and I've always had helpful, friendly support when contacting the team. I have already written to the helpdesk around the issues and received an appropriate response which I believe will eliminate the problem in future. The meeting was hardly disrupted but the people on VC missed two chunks of the meeting. Excellent service in my experience. Staff efficient and helpful. No adverse comments at all. The confirmation of booking does not list all the participants on the bridge - this would be useful for double checking the booking and for reference in the absence of the person who made the booking. Otherwise service to date has been very positive and staff very helpful. Thank you. Although our meeting started 40 minutes late because of the difficulties connecting to the VC, we were still able to accomplish what we set out to achieve. Thanks to Lorraine in Grampian for her helpful manner on the phone, and for her colleague (sorry, I didn't catch his name) who managed to get us connected in the end. 21