Power System Vulnerabilities : AHCA 2016 Copyright (c) 2016 SSR, Inc. All rights reserved

Transcription

1 Training presentation for: AHCA Florida 2016 Annual Conference Finding & Managing Power System Vulnerabilities David Stymiest, CHFM, CHSP, FASHE, (PEin LA MS MA) Copyright 2016, Smith Seckman Reid, Inc.; All rights reserved Major topics Finding common mode failure potential Improving communication Planning for different types of failures Assessing reliability, availability and dependability Importance of inspection, maintenance & testing Lessons learned from natural disasters and other events Disclaimer Content Disclaimer: These slides are only meant to be cue points, which were expounded upon verbally by the original presenter and are not meant to be comprehensive statements of requirements or best practices, nor represent all the content of the presentation. Thus, care should be exercised in interpreting content based solely on the content of these slides. David Stymiest, PE CHFM CHSP FASHE. 1

2 NFPA Disclaimer Although the speaker is on the NFPA Technical Committee on Emergency Power Supplies, which is responsible for NFPA 110 and 111, the views and opinions expressed in this presentation are purely those of the speaker and shall not be considered the official position of NFPA or any of its Technical Committees and shall not be considered to be, nor be relied upon as, a Formal Interpretation. Readers are encouraged to refer to the entire texts of all referenced documents. NFPA members can obtain staff interpretations of NFPA standards at A new paradigm Reliability Probability that system operates and gives the same result on successive trials Availability Probability that system will function at any instant required, including the next instant, and for as long as required from that point Dependability Measures availability, reliability & maintenance support Lessons from real failures Lessons from real failures Things break Ask critical questions Pay attention to the details Commonalities & history Common-mode failures Analyze impact of what if scenarios Importance of testing & maintenance Comprehensive vulnerability analyses David Stymiest, PE CHFM CHSP FASHE. 2

3 MORE lessons from real failures Lessons from real failures Generators run out of fuel No cell service to call for assistance Poor or impossible travel conditions Cascading system failures Staff training FEMA realities during an emergency Ongoing ITM is very important You can t control what you can t control The details will get you Simplified power system Potential vulnerabilities to consider David Stymiest, PE CHFM CHSP FASHE. 3

4 Managing vulnerabilities Finding Prioritizing Assessing Reporting Mitigating Verifying Common-mode failures Failures of two or more components or systems due to a single event or cause Identified failure mode can usually be mitigated by adding extra or redundant equipment Cannot correct unidentified failure modes Uncorrected common mode failure removes advantage of other redundancies Common-mode failures Normal and emergency power equipment on same level Fuel oil transfer pump set subject to flooding Common fuel oil transfer pumps, controls, power circuits Feeders for elevated equipment located in flooded levels Contaminated fuel oil system Co-location of equipment & systems One sump pump or multiple sump pumps on same panel Transfer switch failure David Stymiest, PE CHFM CHSP FASHE. 4

5 Common location / subsystems Paralleled generator sets can mitigate the impact of one generator failure. Can also be subject to common- mode failures: Sharedlocation Shared fuel system Shared cooling system Common distribution Paralleling switchgear Control power failure Internal short circuit (fault) Low probability but very high impact May become apparent when EPSS is energized next. Mitigating vulnerabilities All power systems and equipment Understand Plan Inspect Test Maintain Remove David Stymiest, PE CHFM CHSP FASHE. 5

6 Explaining Normal vs. Emergency Power Generator is usually off Red (Emerg.) Outlet Emergency Generator Explaining Normal vs. Emergency Power Simplified Emergency Power Supply System Generator on Red (Emerg.) Outlet Emergency Generator Types of failures Normal down with emergency power working 1 emergency power branch down, normal working, other branches working Total electrical failure Simultaneously Cascading events speaker to request teaching slides David Stymiest, PE CHFM CHSP FASHE. 6

7 Most common generator failures Starting system problems Fuel oil system problems Cooling system problems Installation error / lack of acceptance testing Inadequate maintenance Overloads generators, breakers, fuses Load shed malfunctions multiple generators fail Other causes of generator failures Lightning power surge damages generator controls Generator auxiliaries on normal power (fans, fuel transfer pumps) Failures during routine testing (thrown engine rod, fuel hose rupture) Other mechanical or electrical failures Generator breaker trips lack of protective coordination Planning for Internal Failures Contingency planning: details important Doesn t need to be long Just needs to be correct for templates David Stymiest, PE CHFM CHSP FASHE. 7

8 Planning for Internal Failures Consider different failure points Different responses Don t wait until failure occurs Approach to vulnerabilities 1. Consider each component 2. What if? scenarios, including feeders 3. Common-mode failure potential 4. All possible causes of those scenarios 5. Address all resulting vulnerabilities Preparedness for power failures Things break Sweat small stuff Small issues can take out systems Different failures When to plan? David Stymiest, PE CHFM CHSP FASHE. 8

9 Vulnerability Analysis: Infrastructure Infrastructure design, features, components, condition, locations, operating flexibility, spares, maintenance histories, vulnerabilities Electrical service and NP distribution EPSS and its auxiliary subsystems (FO +) ATSs, feeders, branches Power system documentation, labeling, failure procedures, test results, training Vulnerability Analysis: Power Sources NP system: major distribution EP system, Gen, ATS, ATS sources Sortinfrastructure systems, facility areas and facility services by each power train By each main switchboard, generator, ATS Equipment or wiring failures take out redundant systems / areas / functions? Common mode failure vulnerability analysis Vulnerability Analysis: Areas For important functional areas, look at: Higher vulnerability from infrastructure analysis Less dependable equipment Poorer documentation Inadequate power failure procedures Inadequate training Higher vulnerability from common mode failure vulnerability analysis All other vulnerability assessment tools David Stymiest, PE CHFM CHSP FASHE. 9

10 System vulnerabilities: examples Inadequate acceptance testing Common-mode failure potential ATSs not maintained regularly Because not bypass-isolation type ATSs not transferred every month Lack of switchboard / breaker maintenance Lack of EES maintenance Vulnerability analysis results Preparedness Activities Additional capacity Emergency equipment Identify additional resources More training / testing Contingency planning Mitigation Activities Policies & procedures Change in process Maintenance program (Plan electrical shutdowns) Infrastructure repair/upgrade Tighten rooms Leak detection Gap analysis for VULNERABILITIES Gap Analysis can also address results of Vulnerability Analysis How vulnerable is EP System to failures? How vulnerable is NP System to failures? Where are the common-mode failure vulnerabilities? How to mitigate these vulnerabilities Short term Long term David Stymiest, PE CHFM CHSP FASHE. 10

11 Gap analysis for SUPPLIED SERVICES Examples: fuel oil supplier, generator or ATS service company, spare parts supplier Service provider vulnerability or over-commitment Systematically identify gaps between where the supplier s crisis management capabilities end and your contingency plans begin. Fix them. PdM Example: Infrared Thermography Predictive maintenance (PdM) vs. calendar-based maintenance (PM) Many facilities already scan NP equipment Include generator panel, paralleling switchgear, and transfer switches in IR scanning scope of work. Scan paralleling switchgear when it is energized. Some other PdM examples Diesel generator fluid testing Fuel oil testing is required annually BEST PRACTICE also testing lubricating oil & cooling water Rotating equipment vibration analysis Ultrasonic analysis David Stymiest, PE CHFM CHSP FASHE. 11

12 Electrical room maintenance Learn from required weekly EPSS inspections Cleaning rooms: minimize contaminants finding their way inside the electrical equipment Change the filters Inspect for evidence of water Leak detection in electrical rooms Warns of water-based vulnerabilities when relocation is not practical Mech/Elect co-locations Elevation issues External water Internal piping leaks Broken sumps Risk assessment considerations Transfer switches maintenance history Normal power operational history Normal power maintenance history User and facility management action plans EP system maintenance shutdowns Thermographic scanning results Infrastructure conditions David Stymiest, PE CHFM CHSP FASHE. 12

13 Competency training for maintainers Responses to various internal failures Responses to simultaneous multiple utility failures Operation of different equipment, not just the same equipment every month Understand and look for second order consequences Thank You! David Stymiest, P.E., CHFM, CHSP, FASHE (P.E. in LA, MS, MA) Senior Consultant Smith Seckman Reid, Inc. Compliance News archives and sign-up are available at: David Stymiest, PE CHFM CHSP FASHE. 13