INTELLIGENT ELECTRONIC DEVICE DATA EXTRACTION AT HYDRO ONE. Sheng Hsu Senior Protection & Control Engineer Hydro One

Transcription

1 INTELLIGENT ELECTRONIC DEVICE DATA EXTRACTION AT HYDRO ONE Sheng Hsu Senior Protection & Control Engineer Hydro One Adam Gauci Field Applications Engineer Cooper Power Systems INTRODUCTION Hydro One is the transmission and distribution utility for the province of Ontario, Canada. It is one of the largest government owned utilities in North America with 28,600 km of transmission lines and 122,800 km of distribution lines connecting 350 transmission and 1,035 distribution stations. Within the next few years a large percentage of Hydro One s skilled workforce will be retiring. This has lead to a rethinking of how soft resources are allocated as they begin the process of acquiring and training new talent. In Ontario s northern regions the travel distances between substations can be tremendous. It can sometimes take more than a day s drive through treacherous winter conditions to retrieve one protection event file. Add to this the amount of time to locate a line fault in heavily forested areas, and the amount of time and man power required for a fault can be quite extensive. Over the last two decades Hydro One has been replacing electromechanical relays with Intelligent Electronic Device (IED) protections. These microprocessor devices contain advanced communication features which were, until recently, never fully utilized at Hydro One. Hydro One primarily uses General Electric (GE) Multilin Universal Relays for their A protection schemes and Schweitzer Engineering Laboratories (SEL) relays for their B protection schemes (but are not necessarily limited to these). These relays are part of their engineering standard and are replacing end of life electromechanical and microprocessor based protection devices. Improving the ability of their staff to access and perform quick analysis of event data has allowed a faster response time, decreased the time of the decision making process and has made finding the location and origin of a fault much simpler. Secure networks were then designed at the substation level. All IEDs were connected through an intelligent substation gateway. In some newer installations SCADA data is collected directly from the protection IEDs via DNP3. This eliminates the need for redundant cabling to a substation Remote Terminal Unit (RTU). NON-OPERATIONAL DATA Each of the IEDs that Hydro One uses in their protection schemes contains different types of data. In this section it will be explained what the different data types are and how they are collected from the two different types of relays. GE Multilin Universal Relay Data Types The GE Multilin relay has three types of events. These types of events are called: A fault report (FAULT), an event record (EVENT) and an oscillography file (OSC). Fault Report: This report can be triggered by any operand within the IED. Generally the virtual output used for tripping the circuit breaker is selected as the trigger. Information on the protection state immediately before and after the fault report is triggered is recorded into the report. Event Record: Any operation within the relay can be selected to be reported as an event. When an event is triggered the event identifier, cause and timestamp of the event are recorded into the record. Oscillography File: This file is triggered by a logic operand. It contains waveforms of analog data at a configurable sampling rate. SEL Relay Data Types The SEL relay has two types of events. The two types of events are called: The standard event report (EVE) and the sequential event report (SER). A GE EVENT file is similar to a SEL SER file, and a GE OSC file is similar to a SEL EVE file.

2 Standard Event Report: This is triggered when an output or a trip is executed. This report gives information on time, currents, voltages, frequency, etc. The data can be shown by sending an EVE command to the relay. Sequential Event Report: Relay elements that trigger a SER event are selected by the user. The data contained in a SER event is a timestamp, relay element or condition and element state. Event Data Collection Protocols The event files are collected and pushed to a configurable location by a Cooper Power Systems Cybectec SMP 16 Substation Gateway. The gateway uses two protocols with two completely different methods for collecting the events. GE Multilin UR Events Master Protocol: The GE relays contains a MODBUS register for each type of event, when a new event is available the register is incremented. The gateway checks the register every minute to see if there are any new event files and if a file is available, it will send it to the location configured in the gateway. SEL Events Master Protocol: For standard reports the gateway periodically issues a HIST command. It then identifies if any new events have occurred. If a new event is identified the gateway will run the EVE command and copy the output to a file. For the sequential reports, the gateway periodically issues the SER command and records any changes to a file. Once a new file is created it is then sent to the location that is configured within the gateway. Event Data Locations and User Access Substation Human Machine Interface (HMI): Some substations contain an HMI for local control of the station. Hydro One uses Cooper Power Systems Cybectec Visual Substation software package running on a Windows XP Professional substation hardened computer. This software package maintains a local database of event files and provides tools for viewing and analyzing event data. These tools compliment the protection commissioning process by allowing protection engineers immediate access to data generated by the events. The substation gateway can be configured to copy the event data files that are extracted from the relays to the HMI computer. The file is then copied to three places. The first place the file is copied to is a folder where Visual Substation processes it into the local database. A second backup copy of the file is stored on the hard drive of the HMI computer. A third copy of the event file is copied up to an enterprise level server via Hydro One s Wide Area Network (WAN).

3 Enterprise Interface Server: The event data files are sent to the server either by the substation HMI computer or directly from the substation gateway. To manage event files at the enterprise level, Hydro One uses the Cooper Cybectec Event Manager Server software. Event files that are received by the Event Manager are processed and stored in a Microsoft SQL server database. Certain events can be configured to automatically send an or a SMS message. A web-based interface allows users to access the event data from anywhere in the company over Hydro One s corporate network. The Event Manager allows for the implementation of a data security policy. A user database can be created or the package can be tied into the corporate Active Directory structure. User rights are assigned determining what data the user can access, down to a per IED level. From the web interface users can query the event file database and see what event files have been collected from each individual IED. Text based event files are viewable directly in the web browser. Oscillography files are converted to the COMTRADE format and can be downloaded and viewed in any standard COMTRADE viewer. All event files can be downloaded in their original format and viewed in their native vendor applications. The Event Manager is also able to directly retrieve events on a periodic interval. The events are checked by the Event Manager once every 24 hours to ensure that no events have been missed during the push-up process. A user control is available in the web interface that forces a check of the relay by the Event Manager on demand. OPERATIONAL DATA Traditionally all control and operational data collected in a substation was done through a central RTU device connected to the SCADA system. These devices have a finite capacity in terms of inputs and outputs. Once this limit is reached expansion of the RTU can be very costly. The solution for Hydro One was to use the communications ability of each relay via DNP3 to read telemetry, protection related alarms and some basic control. Gathering telemetry from the protection relays eliminated the need for extra current transformers (CT) and potential transformers (PT) to be wired to the RTU. Instead of requiring direct wiring of inputs/outputs for protection alarms, communications cabling is replacing enormous amounts of copper wire. Some basic control such as protection scheme selection and re-closure control are also performed directly from the relays. Important Trip/Close controls are still performed from RTUs, this allows for an extra backup to trip a breaker in case the protections are not functioning correctly. By using a substation gateway as a data concentrator, the substation appears as a single device to the SCADA master. By having the SCADA master poll one device instead of many it greatly simplifies the configuration at the control center. NETWORK ARCHITECTURE Substation Local Area Network (LAN) The IED Data Extraction Project at Hydro One was tasked with the creation of substation LANs at selected stations and making connections to protection relays with the intention of retrieving event data. The preference for the physical connection medium was fibre optic Ethernet for its inherent resistance to electromagnetic noise. Due to the harsh substation environment and the requirement for many different connection types, hardened switches and routers were selected. If possible, each substation was fitted with fibre optic cable between buildings. Existing outdoor cable pans were used and different building entrance/exits were required. The switches were connected with 1000 GB fibre optic links in a redundant ring structure. If it was not possible to run fibre optic cable due to site conditions an b network was utilized. A substation gateway was placed in the control house building and connected via fibre optic patch cable to the switch. Once the gateway had been configured it was immediately able to retrieve events from any IEDs on the network. Some older relays such as the SEL 200 and 300 series relays do not have network connections. Serial cables were directly connected between these types of relays and the gateway allowing it to retrieve events via RS-232. When operational data is involved at the substation level, redundant network architecture is required. Each switch and router is doubled up in all buildings. Switches are simply added to the same ring structure, but are usually placed in physically different locations and powered from a separate source. Two substation gateways are used in a redundant configuration. One gateway is always active and the other is in a hot-standby state. When a failure condition occurs the standby gateway becomes active without any interruption to the operational data that is being sent to SCADA. In this type of situation the gateway would be connected via LAN or RS- 232/485/422 to protection relays and RTUs.

4 Some of the protocols in use at Hydro One are DNP3, MODBUS, MDAC and Tejas. Substation Wide Area Network (WAN) Some substations are able to connect to Hydro One s synchronous optical networking (SONET) network, which have direct point-to-point access to the datacenter at T1 speeds. In stations that do not have access to the SONET network, the Bell Canada Frame Relay network is used at speeds of 56k, 128k or T1. In northern areas of the province where WAN communication mediums are limited, satellite communications are used. Issues Encountered Cyber Security: After the events of September 2001 and the North American northeast blackout of August 2003, the North American Electricity Reliability Corporation (NERC) was tasked with creating the Critical Infrastructure Protection (CIP) standard. Part of this standard includes cyber security. To remain a part of the interconnected North American grid, all utilities must strive to meet these standards. Each IED that is accessible through a routable protocol must be enclosed within an electronic security perimeter. Any communications paths that cross the boundaries of these perimeters must be documented, managed and secured. The substation gateway gives the ability of having all data pass through a single device, which makes documenting communication outside of the electronic security perimeter much easier. Hydro One is still in the process of developing and implementing a comprehensive solution to meet all of the NERC CIP standards. The implementation of this network within substations has also created a requirement for a physical security perimeter as per the NERC CIP. This requirement has given Hydro One another item to evaluate as they judge the benefits versus the costs of implementing such solutions. Network Bandwidth: With traditional DNP3 SCADA traffic a 56k network link was sufficient. Now with file transfers, online diagnostic and configuration tools a 56k link can rapidly reach its capacity. A single substation fault can result in

5 thousands of events files from multiple IEDs, which has resulted in an overloading of the link. In the long term, substations will be integrated into the SONET network which should resolve most bandwidth issues. In the short term, the solutions are to increase the bandwidth of the link if technically feasible, or to add a second link and separate operational and non-operational data. Network Boundaries: Determining the boundaries between networks for the enterprise solution is a challenging task. The enterpriselevel Event Manager must have connection between two varying networks with different levels and policies for security, and with completely different purposes. Currently at Hydro One the Event Manager is positioned in a DMZ between the two networks. The best location for the server is currently being studied in an architecture review for the NERC CIP Cyber Security project. Understanding the behaviour of relays: Fault, Events, Oscillography are all outputs given by the relays. Before the automated data extraction, staff would drive to the station and retrieve the information they were told to extract from the relays. They were interested in the actual fault and would feed the fault information manually into a program and an estimated fault location calculated. With automation, it was realized that many of the relays were not fully configured or were not configured to provide the correct data, allowing for a lot of data to be sent out through the network or erroneous data to be sent out. For example, certain relays will create event files if there is a 2% fluctuation of the analog values. This will allow for tremendous amounts of information that are not relevant unless one is looking for power quality reports. The actual fault location can be extracted from the relays within a minute of occurrence with this technology. However if the impedance of the line was not properly calculated and configured into the relay, the information coming out to dispatch a crew to investigate will be incorrect, causing further delays to restore the power. These are just a couple of issues from a list of facts that came to our realization during and after the commissioning of this project. Implementing IED Data Extraction gave Hydro One a better understanding of how they were calibrating their relays. In many instances when faults occurred in long high voltage lines, adjacent relays also detected the fault and captured oscillography traces. Relay events for the neighbouring lines show that they almost tripped as well. This allowed Hydro One to better refine their relay settings and protection design so that faults in one line do not inadvertently cause faults on other neighbouring lines. This is not possible with the older method of retrieving data where only one relay had information gathered. CONCLUSION Hydro One has found efficiencies by extending network architecture to its substations. Enabling extended functionalities of protection relays with cost effective management tools has allowed for non-operational data to be used for quickly making decisions when a fault event occurs. Responsible staff spends less time in transit to diagnose a problem and is instead sent directly to solve the problem. This is particularly important in remote northern regions where inclement weather conditions can impact travel times and a lot of stations would require the staff to travel the whole day just to retrieve one event. Engineering and operations staffs now have easy access to current event data for analysis, helping them improve grid management and predict future trends. Quicker response to restore the power can be seen with this service to the operators. Significant savings have been recognized by the reduction of direct I/O wiring with copper cables. New operational data systems are easily expandable and do not carry the same capacity limitations as previous RTU systems. Other benefits from the new architecture at Hydro One include remote troubleshooting and configuration of protection and control devices. Troubleshooting tools include a network statistics tool and a real-time protocol analyzer which allows technical staff in the control room the ability to try to diagnose and correct problems in the field before field staff need to be contacted. Configuration of protection, control and telecom devices can be performed remotely by head office engineering staff. Overall Hydro One has benefited from this initiative by understanding their system better through increased substation monitoring; this has achieved the goal of reducing the need for having personnel on site in substations. REFERENCES Matt Efremov, IED Data Extraction Data Types Hydro One. Jean-Louis Paquet, Anthony Wright, Strategies for Automating the Retrieval and the Management of Non-Operational Substation Data Western Power Delivery Automation Conference, April 2004.