NFPA members can obtain staff interpretations of NFPA standards at Although the speaker is Chairman of the NFPA.

Transcription

1 2014 TAHFM Interlink Conference Managing Electrical Systems for Reliability David Stymiest, PE, CHFM, CHSP, FASHE cell Copyright 2014, Smith Seckman Reid, Inc., All rights NFPA Disclaimer Although the speaker is Chairman of the NFPA Technical Committee on Emergency Power Supplies, which is responsible for NFPA 110 and 111, the views and opinions expressed in this presentation are purely those of the speaker and shall not be considered the official position of NFPA or any of its Technical Committees and shall not be considered to be, nor be relied upon as, a Formal Interpretation. Readers are encouraged to refer to the entire texts of all referenced documents. NFPA members can obtain staff interpretations of NFPA standards at DStymiest@ssr-inc.com 1

2 Main topics Finding & assess hidden vulnerabilities Typical common-mode failures Power system reliability, availability & dependability Assessing O&M, testing, electrical safety & contingency plans Other lessons learned NOTE: HANDOUT DOES NOT INCLUDE PHOTOS OR COPYRIGHTED IMAGES USED IN PRESENTATION It s not just about the weather not for a long time 1999 State of Wash. gasoline pipeline rupture ~ 2009 DC Metro train crash -P.E. 5/10 April 8, 2009 USA Today DStymiest@ssr-inc.com 2

3 3 Hours of Electrical Utility Outage Number of Facilities (138 had utility outages) (25 < 4 hrs) ASHE survey: Utility outage duration ASHE super storm Sandy survey had 390 responses and what about this bit of oldnews? 2014 TAHFM Interlink: Managing Electrical Systems

4 ASHE survey: EES unanticipated outage? YES YES 9% 9% NO 91% NO 91% YES = 24 NO = 246 ASHE survey: cause of EES failure Causes of EES Failure Fuel System Other Cooling System Batteries Flooding/Rain Other: 7 Other includes 1 each: Breaker Failure, Electrical Fire, Oil Leak, Water Pump, Fan Bearing, Generator, and Required emergency items on NP DStymiest@ssr-inc.com 4

5 Emergency Power Lessons RE-learned Reliability Availability Dependability Lessons RE-learned Things break Ask critical questions Pay attention to the details Analyze impact of what if scenarios Importance of testing & maintenance Commonalities & history Common-mode failures Comprehensive vulnerability analyses 5

6 Other lessons RE-learned Basic E.M. concepts Ran out of fuel oil, no replacements Lack of cell service to get assistance Generator service could not get there Cascading system failures Staff training for portable connections FEMA realities during an emergency More lessons RE-learned Ongoing testing & maintenance are crucial Generation, Switching, Distribution Don t forget other utilities You can t control what you can t control So plan for it Without information you have only opinions The details will get you sweat the small stuff DStymiest@ssr-inc.com 6

7 Sweat the small stuff Details are important Things break Communication is vital Image courtesy of Mario Sanchez, AP Managing Vulnerabilities Finding Prioritizing Assessing Reporting Mitigating Verifying 7

8 Simplified hospital power system Common-mode failures Failures of two or more components or systems due to a single event or cause A safety engineering concept: once a failure mode is identified, it usually can be mitigated by adding extra or redundant equipment to the system The existence of an uncorrected common mode failure potentially removes the advantage of other redundancies. You cannot correct what you have not yet identified. DStymiest@ssr-inc.com 8

9 Look beneath the surface Other types of common-mode failures Normal and emergency power equipment on same level Fuel oil storage tank subject to flooding Common fuel oil transfer pumps, controls, power circuits Feeders for elevated equipment located in flooded levels Other types of damage also Contaminated fuel oil system 9

10 Causes of fuel oil contamination Natural fuel degradation from aging Day tank corrosion Excessive fuel oil filter replacement interval Workmanship during FO system renovation Fuel oil truck operator error Day tank microorganism contamination Inconsistent fuel oil quality from the supplier Incorrect biocide usage Inadequate sampling techniques Clogged or fouled fuel oil filter 2013 NFPA 110 Fuel Oil Recommendations 10

11 Service contractor observations Monthly testing work-arounds (VFD issues) came back to haunt facilities Single spin-on fuel filters clogged when Extended electrical utility blackout Dirty fuel was delivered Or clean fuel delivered to nearly empty tank stirred up bottom sediment Filters had extremely small micron level Put in multiple filter assemblies with isolating valves and bypass valves Common locations Paralleled generator sets can mitigate the impact of a generator failure, but also can be subject to common mode failures due to sharedlocation, sharedfuel or shared cooling systems. DStymiest@ssr-inc.com 11

12 Major Changes in NFPA to 2010 Category / Topic Protection from hazards 1999 Edition (Ref. by 2000 LSC) Rooms, shelters, or separate buildings located to minimize the possibility of damage from flooding (fire fighting, sewer, similar) 2010 Edition (Ref. by 2012 LSC) Rooms, shelters, or separate buildings designed and located to minimize the damage from flooding (fire fighting, sewer, similar) 2013 Edition Rooms, enclosures, or separate buildings designed and located to minimize the damage from flooding (fire fighting, sewer, other) TJC Clarifications & Expectations : Jan 14 Ref: NFPA shall be designed and located. shall be a design consideration. Risk assessment before designing new EPSS Battery powered lights at ATSs (NFPA ) DStymiest@ssr-inc.com 12

13 Paralleling Switchgear All generator outputs connected together Potential common-mode failure Control power failure Internal short circuit (no GF protection) Low probability but very high impact May become apparent when EPSS is energized next. Other types of common-mode failures Co-located equipment and systems One sump pump Multiple sump pumps on same branch Transfer switch failure 13

14 Maintenance improves dependability Bypass isolation transfer switches can be maintained without turning off their loads, improving operational dependability. Other common issues Communications with caregivers Some clinical personnel believe EP is or should be uninterruptible, should never fail. Misunderstandings: unrealistic expectations Medical journal article: usually less than 1- second duration upon loss of commercial power Different types of failures Different responses for each Updated failure procedures More pervasive, more complex systems now 14

15 Types of failures Normal down with emergency power working 1 emergency power branch down, normal working, other branches working 1 CB down with other CB still working Total electrical failure Simultaneously Cascading events speaker to request teaching slides Explaining Normal vs. Emergency Power Normal Outlet Red (Emerg.) Outlet Generator is usually off Emergency Generator DStymiest@ssr-inc.com 15

16 Communicating NP switchboard outage What will happen? Power that will notbe available (utility power fed through that switchboard) Selected normal lighting Selected NP receptacles (white & brown face) Selected equipment served from normal power Power that will be available (on generator) Power fed through other normal power switchboards Emergency lighting (includes egress lighting) Emergency receptacles (red face) Equipment served from emergency power Explaining Normal vs. Emergency Power Simplified Emergency Power Supply System Generator on Red (Emerg.) Outlet Emergency Generator 16

17 Communicating EP branch outage What will happen? Power that will be available (utility power) Normal lighting Normal receptacles (white & brown face) Equipment served from normal power Power that will notbe available (load side of selected transfer switch[es]) Emergency lighting (includes egress lighting) Emergency receptacles (red face) Equipment served from emergency power Contingency plans: stay cool under pressure What can go wrong? Then what happens? What response to use? Decisions Flexibility vs. details 17

18 Most common generator failures Starting system problems Fuel oil system problems Cooling system problems Installation error / lack of acceptance testing Inadequate maintenance Overloads generators, breakers, fuses Load shed malfunctions multiple generators fail Other causes of generator failures Lightning power surge damages generator controls Generator auxiliaries on normal power (fans, fuel transfer pumps) Failures during routine testing (thrown engine rod, fuel hose rupture) Other mechanical or electrical failures Generator breaker trips lack of protective coordination DStymiest@ssr-inc.com 18

19 Planning for internal electrical failures Contingency planning: details important Doesn t need to be long Just needs to be correct for templates Simple contingency plan EMERGENCY CONDITIONS and BASIC STAFF RESPONSE UTILITY FAILURE BUILDING UTILITY FAILURES WHAT TO EXPECT WHAT TO DO EMERGENCY CLINICAL INTERVENTIONS Normal Electrical Power Failure Power only to emergency lights and RED plug outlets. Open Disaster Bin for flashlight, extension cords, batteries, etc. Know areas on emergency power. Ensure that Life Support Systems are attached to RED plugs; be prepared to handventilate. List clinical interventions Emergency Electrical Power Failure (only) Power only to normal lighting, and gray or white plug outlets Open Disaster Bin for flashlight, extension cords, batteries, etc. Check all patient care equipment and patient task lighting. Ensure that Life Support Systems are attached to gray/white plugs or to BACKUP red plugs if available; be prepared to hand-ventilate. List clinical interventions DStymiest@ssr-inc.com 19

20 Some examples - generators Some examples - switchboards DStymiest@ssr-inc.com 20

21 Planning for Internal Failures Must consider different failure points, not just at the mains. The responses will be different for each type of failure. It is TOO LATE to formulate a response after the failure has occurred. Revisit SEA-37 Vulnerability Analyses Preventing adverse events caused by emergency electrical power system failures published by TJC 9/6/2006 Also in TJC s 9/2007 EC News Recent events: Should we address the vulnerability analysis againperhaps more comprehensively this time? DStymiest@ssr-inc.com 21

22 So what do we want from EP? Our emergency power systems need to power What they must When they must For as long as they must And we need to be able to roll with the punches when things go wrong A new paradigm Reliability Probability that system operates and gives the same result on successive trials Availability Probability that system will function at any instant required, includingthenext instant, and for as long as required from that point Dependability Measures availability, reliability & maintenance support DStymiest@ssr-inc.com 22

23 Availability Consider this If no facilities system can guarantee 100% reliability, can any facilities system assure 100% percent availability? Common metric for large data centers 4 nines facility availability % How does your power system compare with data center power system design? Recommended approach to vulnerabilities 1. Consider each component that must operate; 2. Determine what scenarios will cause it to fail, including all What if? scenarios that could damage the power sources or feeders that keep it running; 3. Compare those scenarios with others that may take out other redundant components, redundant power sources or redundant feeders; 4. Investigate all the possible causes of those scenarios, including commonalities in power sources, feeders or controls; 5. Address all resulting vulnerabilities that have been identified. DStymiest@ssr-inc.com 23

24 Preparedness for power failures Things break Sweat the small stuff Small issues can take out systems Different failures When to plan? Vulnerability Analysis: Infrastructure For infrastructure, look at features, components, condition, locations, operating flexibility, spares, maintenance histories, vulnerabilities for Electrical service and NP distribution to ATS s EPSS and its auxiliary subsystems (FO +) ATS s, feeders, branches EP System documentation, labeling, failure procedures, test results, training DStymiest@ssr-inc.com 24

25 Vulnerability Analysis: Power Sources NP system: major distribution EP system, Gen, ATS, ATS sources Sortinfrastructure systems, facility areas and facility services by each power train By each main switchboard, generator, ATS Determine where single equipment failures or wiring/feeder failures can take out redundant mechanical systems, areas or functions. Common mode failure vulnerability analysis. Vulnerability Analysis: Areas For all functional areas, look at: Higher vulnerability from infrastructure analysis (such as with less reliable equipment; poorer documentation, lack of power failure procedures, training, etc.) Higher vulnerability from common mode failure vulnerability analysis All other vulnerability assessment tools 25

26 EP Vulnerability Analysis: Clinical Prep. Modified from ASHE Management Monograph, 1/ 2007: Performing an Emergency Power Systems Hazard Vulnerability Analysis, Timothy Adams System vulnerabilities: examples Common-mode failure potential ATS's not maintained regularly Because not bypass-isolation type ATS s not transferred every month Lack of switchboard / breaker maintenance Lack of EES branch maintenance Life Safety Branch Critical Branch Equipment Branch DStymiest@ssr-inc.com 26

27 Vulnerability analysis results Preparedness Activities Additional capacity Emergency equipment Identify additional resources More training / testing Contingency planning Mitigation Activities Policies & procedures Change in process Maintenance program (Plan electrical shutdowns) Infrastructure repair/upgrade Tighten rooms Leak detection Gap Analysis for VULNERABILITIES Gap Analysis can also address results of Vulnerability Analysis How vulnerable is EP System to failures? How vulnerable is NP System to failures? Where are the common-mode failure vulnerabilities? What can we do to mitigate these vulnerabilities: short term; long term? DStymiest@ssr-inc.com 27

28 Gap analysis for SUPPLIED SERVICES Examples: fuel oil supplier, generator or ATS service company, spare parts supplier Understand any vulnerability or overcommitment with the service provider. Systematically identify gaps between where the supplier s crisis management capabilities end and your contingency plans begin. (EM ) Fix them. Proactive power system maintenance Predictive Maintenance (PdM) Condition-based Preventive Maintenance (PM) Calendar-based Reliability-Centered Maintenance (RCM) Based on system analyses, logic, statistical input, and criticality of equipment to be maintained Optimum mix of reactive, time-interval-based, condition-based, and proactive maintenance practices DStymiest@ssr-inc.com 28

29 Shutdown was not permitted Inadequate arc flash hazard labeling 29

30 PdM Examples: Infrared Thermography Many facilities already scan NP equipment Include generator panel, paralleling switchgear, and transfer switches in IR scanning scope of work. Make sure to scan equipment paralleling switchgear when it is energized. Thermal cycling works electrical lugs loose. Scan EPS (and EP lugs in ATS s) during monthly EPSS tests. Designing for infrared thermography Pinhole lens technology can see through ½ viewport 30

31 Some other PdM examples Diesel generator fluid testing Fuel oil Lubricating oil, Cooling water Rotating equipment vibration analysis Ultrasonic analysis Electrical room maintenance Learn from weekly inspections Cleaning rooms: minimize contaminants finding their way inside the electrical equipment Change the filters Inspect for evidence of water Water and electricity are a mixture that no facility engineer wants DStymiest@ssr-inc.com 31

32 Leak detection in electrical rooms Warns of water-based vulnerabilities when relocation is not practical Mech/Elect co-locations Elevation issues External water Internal piping Broken sumps Some risk assessment considerations Transfer switches maintenance history Normal power operational history Normal power maintenance shutdowns User and facility management action plans EP system maintenance shutdowns Thermographic scanning results Infrastructure conditions 32

33 Competency training for maintainers Responses to various internal failures Responses to simultaneous multiple utility failures Operation of different equipment, not just the same equipment every month Understand and look for second order consequences Shutdown lessons learned Where were extension cords required? What special precautions did Users take? What surprised you? What surprised the Units? Update contingency plans Utility Management Plan; P&P s EOP, Action Plans Input to next capital budget cycle DStymiest@ssr-inc.com 33

34 Tracers on Preparing for Power Failures Test your own readiness Clinical equipment & personnel responses Reliance on UPS s Power shutdowns Maintenance EP loading Equipment failures Documentation ASHE paper with power failure tracers available upon request Thank you. Questions? David Stymiest, CHFM, CHSP, FASHE (Registered P.E. in LA, MS, MA) Cell DStymiest@ssr-inc.com DStymiest@ssr-inc.com 34