Disclaimer Ground Rules

Transcription

1 Critical Success Factors in a Complex BCP Development Environment Don Groth Disclaimer Ground Rules The views and opinions expressed today do not necessarily reflect the position of Froedtert and Community Health or Jefferson Wells International, Inc. Any copyrights/trademarks belong to owners... Comments about vendor products or services are intended for illustrative purposes only. BCP Critical Success Factors Case Study What was done How was it done Lessons Learned Success Factors 1

2 BCP Critical Success Factors Success Ability to survive an actual incident Ability to survive a review Internal Audit Regulators Management Case Study The success was due to: Hard work and dedication of hospital staff Simple key factors The factors may be called simple because they are easy to describe; putting these factors in practice may not necessarily be simple. Easy to say tough to do? Business Continuity Planning at Froedtert & Community Health Froedtert & Community Health Milwaukee-based regional hospital system Combination Froedtert Hospital Milwaukee Community Memorial Hospital Menomonee Falls 2

3 Case Study Continuity plan development for Froedtert & Community Health Development of continuity plans for over 60 key clinical, facility, support, and business departments throughout the two hospitals. Linkages to existing Emergency Management / Incident Command Structure and to an IT Disaster Recovery plan. Froedtert Hospital 414 Bed Academic Medical Center Staffed by Medical College of Wisconsin Physicians > 4,500 staff and > 10,000 people on campus The Only Adult Level I Trauma Center in Eastern Wisconsin 3

4 Froedtert Campus Partners Joint Ventures Staffing Medical College Staff Departments Off-campus facilities Community Memorial Hospital 205 bed community hospital >2,000 Staff Staffed by independent physicians Community Memorial Two large clinics Cooperative Ventures Free Standing Ambulatory Surgery Center Independent community physicians Off-campus facilities 4

5 F&CH Environment Recently combined hospitals as F&CH Healthcare considerations Joint Commission on Accreditation of Healthcare Organizations JCAHO Health Insurance Portability and Accountability Act (HIPAA) Existing Memorandum of Understanding HEICS / HICS Environment - continued Emergency management structures Regional disaster drills Downtime procedures IT Environment Many initiatives including data center relocation A number of high availability systems A number of systems managed by clinical and support departments Hot site plan in development Downtime procedures 5

6 Overall Business Environment Everybody is Busy! Why BCP at Froedtert & Community Health? Why? August 2003: Power Failure 6

7 Source - U.S./Canada Power Outage Task Force report F&CH Business Continuity Project Began early 2004 Board directive It s not a question of if we do this. The only question is how should we do it. Mandate - Not just an IT plan F&CH Business Continuity Project Coordinate Business Continuity Planning Emergency Management IT Disaster Recovery Consider Prior Threat Assessments, Risk Assessments, and Hazard Analyses 7

8 Project Organization Chart Timeline Au Oc De Ma Au Oc Ma Ju J Se No De Ja Fe Ma Ap Ju J Se No Ja Fe INITIAL PROJECT SCOPING BIA THREAT ASSESSMENT RECOVERY STRATEGIES BUSINESS CONTINUITY PLAN DEVELOPMENT MAINTENANCE AND EXERCISING Scoping Set project scope Determined departments to include / exclude Grouped departments Selected department staff 8

9 Scoping Lessons Departments Combining departments About staff BIA Facilitated working sessions Groups of departments Identified critical IT systems Resources Tangible Intangible assessment Detailed calculations RTO & RPO BIA Multiple steps Made detailed estimates only for systems and resources with High / Medium Impacts Consolidation 9

10 Step 1 Identification of Key Systems / Resources Step 2 Detailed Estimates Scales 10

11 BIA Consolidation To provide management with estimated impacts To provide IT with system RTOs To identify the most significant resources Avoid double counting Recognize significance of department / process impacted Judgment required BIA Lessons Grouping of departments People will respond differently Time Intangible vs. Tangible Impact Surprises about systems Work sessions Threat Assessment Timing Critical Global Resources Electric Power (utility & emergency), Fire Detection Systems, Medical Gas, Natural Gas, Steam, Telecom, Water Supply, Waste Water Identified Threats Impacts, Probability, Vulnerability 11

12 Threat Assessment Lessons Actual incidents Water Assumptions Strategy Selection Mitigation Strategies Global resources with vulnerabilities identified in Threat Assessment Hardening Strategies Work Around strategies Other Resources (Not Global) In-place Strategies Published Recommended Strategies Budget process Strategy Selection Lessons Ownership of solutions Tie strategies to budget process Executive Support And then 12

13 Reality Check # 1: July 2005 Brief power failure at Community Memorial Renewed enthusiasm Department Plan Development Working sessions to create department plans Remember that everybody is busy allow plenty of time Department Plan Development Two sets of working sessions Provided sample plan and templates Contact information BIA Strategies Vendors Forms Recovery procedures, and then 13

14 Reality Check # 2: December hour power failure at Froedtert Renewed enthusiasm Department Plan Development Lessons Time Stories Use what you have but it is probably not enough Executive support And then Reality Check # 3: Thursday March 9, ,000,000 gallons of water flood the power plant and steam tunnels at the Milwaukee County Grounds. It appears to have been a pretty catastrophic blowout, says George Torres, County Public Works Director. 14

15 Reality Check # 3: Initial report The basement of the electric power plant that supplies power and steam to the hospital and clinics is filling with water The walls may buckle There is a substantial leak in the water main Water pressure is declining affecting: drinking water sanitation water central vacuum pressure steam Reality Check # 3: Continued Maintenance crews have not been able to isolate the leak May have to take the plant out of service for an extended period of time Could have to evacuate approximately 400 inpatients hundreds of outpatients all staff It could be weeks for the problem to be identified and repaired. 15

16 Lessons Learned Threat Assessment conclusions - reinforced Water is a critical resource Other organizations were eager to assist Hospitals Ambulance companies Lessons Learned Communications Staff used extensively and was effective, however Media Incident Command Center ICC established quickly Department Command Centers Plan Exercises Individual departments Tabletop exercise / plan review Participants Department staff (1 10) Safety Facilitators Scenario Action Plans 16

17 Lessons Learned Be flexible Training opportunity Challenge / validate Raise the bar Stories Have participants tell their stories Use the stories with others Current State Transitioning from project to program Incident command integration Strategies for critical resources Program expansion Transition From Project to Program BCP Plan Development BCP Program Business Impact Analysis Threat Assessment Plan Development Light the Fire Integrate with Incident Command Exercise Maintain Plans Ongoing funding and resources Keep it Fueled 17

18 Success Factors Executive Mandate and Executive Support Existing Emergency Management Experience Steering Committee Culture Patient Care Terminology Leveraging NIMS Alert - Compliance and day-to-day operations NIMS Alert - Compliance and dayto-day operations From the August 17, 2005, NIMS Alert The requirement to adopt and implement NIMS and ICS means NIMS and ICS for incident management every day. Those who don t are not NIMS compliant. Success Factors Pilot Real incidents Leverage actual Incidents Lessons Learned Sessions for staff Reinforce the need to plan They will tell us what to do We will do whatever it takes 18

19 Success Factors - Linkages Emergency Management Plans IT Disaster Recovery Regional Partners BCP Success Factors GETS Government Emergency Telecommunications Service (GETS) Participant profile / skills Familiar with department processes Department decision maker And computer skills Success Factors Persistence We can do anything we want as long as we stick to it long enough. - Helen Keller Even if you are on the right track, you will be run over if you just sit there. - Will Rodgers 19

20 Success Factors Always Serve Good Food! Final Thoughts Just because you re paranoid, it doesn t mean that people aren t out to get you. - Unknown The reason for time is to avoid doing everything at once. - Albert Einstein Questions Jefferson Wells or don.groth@jeffersonwells.com 20

21 Jefferson Wells Headquartered in Milwaukee, WI Founded in 1995 More than 45 offices Over 2,500 employees Subsidiary of Manpower Inc. Provides services in the areas of: Internal Audit and Controls Technology Risk Management Finance and Accounting Tax Don Groth Jefferson Wells - Technology Risk Management Services CBCP, CISA, CIA Member of BRPASW, IIA, ISACA, Infragard or Don.Groth@jefffersonwells.com 21