Technical Spotlight Demo4-5

Transcription

1 Technical Spotlight Demo4-5 Spotlight on the new communication network devices implemented in DEMO4 This project has received funding from the European Union s Seventh Framework Programme for research, technological development and demonstration under grant agreement n

2 Table of content TABLE OF CONTENT... 2 LIST OF FIGURES & TABLES INTRODUCTION AND SCOPE OF THE DOCUMENT OBJECTIVE AND TECHNICAL REQUIREMENTS Context & Objective Requirements DEVELOPMENT AND IMPLEMENTATION Architecture and technical characteristics Lab tests Field implementation TECHNICAL RESULTS avril /27

3 List of figures & tables Figure 1: Demo4 Complete Network Architecture... 6 Figure 2: protocol stack... 8 Figure 3 Typical Smart Grid Communication requirements values (refer to CIRED Paper 0107)... 9 Figure 4: CGR Figure 5 CGS Figure 6: Cgr 1120 back and front view Figure 7: Industrial Ethernet 3000 Catalyst Switch Figure 8: Ethernet frame with vlan tag Figure 9: goose ethertype value Figure 10 : Complete scenario of Communication network and detail flows and functions Figure 11: L2tpv3 transport and replicate Gosse packet over the Wan Figure 12: Details about Secondary substation architecture to connect Active Customer (PV generation) Figure 13: Scenario to transport Scada to Figure 14: Simple Lab Scenario to tested in Selta for interoperability with DPLC Modem Figure 15: A couple of Selta DPLC connected to different router CGR Figure 16: First Cgr1120 router installation Figure 17: First IP Ping latency measurement avril /27

4 1 Introduction and scope of the document Type of solution Equipment / Hardware / Firmware Information system Process Manufacturer(s) implied (for equipment or hardware) Cisco Work Stream considered Active Demand DER integration Location / Topology (with regards to distribution grid) Storage Islanding MV Innovation LV Innovation HV/MV Substation MV MV/LV SS LV DER Meter Downstream meter Other Centralized system (calculations, information system) Other Decentralized Other : Thematic(s) system Grid Monitoring / state estimation Active demand / DSM DER Integration / increased grid capacity Islanding Anti Islanding protection Automatic Failure Detection 1 Remote Grid Operations Automatic Failure Management / Grid Automatic Grid topology reconfiguration Other : recovery 1 Use Case(s) DEMO 1 Failure Management in MV networks Decentralized grid operation in MV Networks DEMO 2 Outage detection in the LV Network DEMO 3 DEMO 4 DEMO 5 Automatic Grid Recovery (AGR) Voltage control on MV grids (with high DER penetration) MV grid automation of failure management DEMO 6 Islanding Key figures Reduction of power demand Automatic Outage Detection (AOD) Anti-islanding protection on MV grids Secondary Substation Node (SSN) Customer Engagement MV Measurement acquisition LV grid automation of failure management Manage maximised PV production on LV network regarding constraints and flexibility programs Demand response for MV Customers Management of islanding operations Encourage resident to adopt smarter habits according to network state 1 Enabled, but not implemented in the Demo 19 avril /27

5 Through out Grid4EU Demo#4, Cisco implemented a Field Area Network (FAN) Architecture through the use of dedicated HW and SW features. By fact Grid4EU Demo#4 requirement was to develop a dedicated Secondary Substation interface to realize backhauling link using different telecommunication transportation technologies (4G-LTE, DPLC). A dedicated modular HW device, CGR 1120 (Cisco Connected Grid Router), has been prototyped and then industrialised through the project. Other Connected Grid devices (CGR1240, CGR 2010, CGS 2520) and products (IE 3000) have been characterized, industrialised and deployed to realize a complex Telecommunication Infrastracture. Likewise, dedicated SW has been developed by adapting IEC protocol over IP. Moreover the following SW features have been implemented on field: SCADA Gateway ( gateway - translator ) GOOSE protocol transportation on the WAN As final result, areas involved into Grid4EU Demo#4 Telecommunication Architecture consists of: 2 Primary Stations, and FAN (93 MV/LV Secondary Stations, 5 DERs and 1 Storage), a LAN to connect IED devices using Ethernet protocol. 19 avril /27

6 2 Objective and technical requirements 2.1 Context & Objective The Distribution System Operator is the actor responsible for the operation, maintenance and development of the distribution network in a given area. The increased penetration of DER in the distribution network combined with the objective of more reliable and cost-efficient network operation, have opened up new opportunities for meeting system obligations and providing grid services (reactive power, voltage support, congestion management) at distribution level. In Demo 4 new Smart Grid solutions have been designed and developed for the remote control and automation of DSO electric grid. Figure 1: Demo4 Complete Network Architecture One of the main targets of the new developments is the management of Distributed Energy Resources (DER) in the Medium Voltage (MV) Grid; in order to enhance power quality, increase grid hosting capacity with reference to DER, avoid undesired islanding operations. All the new automation and control functions, designed according to the new scenarios in the MV grid, are totally based on IEC service suite. In Demo 4 it was agreed to adopt the IEC standard for any new development of IEDs and Smart Grid functions. The main functions are: 1. voltage regulation, by the remote control of DER and Storage. 2. MV grid advanced monitoring 3. a real time DER shedding application (as anti-islanding approach) 19 avril /27

7 2.1 Requirements Grid4EU Demo#4 staple requirement was to develop a dedicated Secondary Substation interface to realize back-hauling link using heterogeneous telecommunication transportation technologies (originally WiMax, DPLC, 3G and 4G-LTE). Based on above, the following high level architecture electrical requirements have been faced and considered into Internet-working Architecture: 1. control the value of the voltage on the nodes of MV network, being used the possibility of modulation of reactive or active power/of the generators, with the objective to increase the DG penetration, to improve the quality of the voltage and the efficiency of the distribution MV network; 2. perform the automatic disconnection of DER, to avoid potential unwanted islanding operation of Distribution Grid s portions, in case of lack of the of main feeding (Loss of Main) (detach production); In order to meet above requirements, Cisco have defined new detailed functional specification and started to build new dedicated hardware (Connected Grid Router and Industrial Ethernet Lan Switches). Moreover new software features have been developed, using the architectural and communication model based on TCP\IP protocol and supporting other industrial standard for interoperability to respect EC mandate M490. The integration of small renewable generators into power networks will offer a cheaper supply option to consumers and help supply isolated consumers, hence increasing the reliability of the power grid and reducing its operation costs. However, it is clear that managing this new system will also require a new control system to effectively manage it. This new system will require utilities to move away from the traditional centralized approach and adopt a more distributed, collaborative, and dynamic paradigm. The introduction of DER into the power grid will create new issues for the distribution network operators (DNOs).While in traditional power grids the electricity usually flows from the central power stations to the consumers, in a smart power grid incremented with DER, the electricity now flows in two directions, either from the station to the consumer or from the consumer DER back into the grid. To introduce DER, DNOs will therefore be faced with the challenge of making their power distribution networks more flexible and dynamic. Active control is predicated on continuous real time monitoring and management of the power network. Therefore, sensors need to be deployed in far larger numbers than are currently in order to efficiently monitor the power network conditions. These measurements will need to be taken across the entire distribution network. Moreover, many works in the area of active control are advocating the move from the current centralized active control model towards a more autonomous active control paradigm. More recently, the International Electro technical Commission (IEC) proposed a new standard that targets interoperability and fast communication between Intelligent Electrical Devices (IEDs) called IEC The superiority of IEC61850 over other communication standards in substations is: 19 avril /27

8 the functions, the services, and the communication protocols are not mixed together but are defined separately the standard contains the data models of all possible functions in a substation, and standardizes the names of these functions and their data the standard specifies a set of generic abstract services which cover all the data transfers required within a substation and maps these abstract services and the standardized data onto real communication protocols which include Ethernet, TCP/IP and Manufacturing Message Specification (MMS). Figure 2: protocol stack The implementation of active control for distribution and generation necessitates the introduction of changes in the communication infrastructure that supports control operations in the power grid. This communication infrastructure cannot currently support the control traffic generated by thousands of extra sensors and control devices that will be added to control the power grid. Therefore, the existing communication infrastructure first needs to be urgently upgraded in terms of transmission capacity to support this extra control traffic and new internetworking devices routers and Lan switches with industrial grade and new communication links will need to be added in order to connect intelligent controllers and support the coordination of traffic. With a preliminary activity to Demo4 developed during the last part of the year 2010 we have prepared with the Demo4 Team a detailed list of requirements for hardware and software that telecommunications equipment had to be compliant. Much attention was all hardware aspect of products which until then did not take into account industry standards required to operate in environments very often industrial harsh environment. Automation and protection features, historically separated mainly for security reasons, should start interoperating, at least in a limited way, in order to ensure an overall system properly working. 19 avril /27

9 Figure 3 Typical Smart Grid Communication requirements values (refer to CIRED Paper 0107) 2 At the end of the First year of the Demo4, Team produce some technical specification documents which were used to build any prototypes. Now these prototypes have become products that did not exist prior to the project In particular version indoor of Secondary Substation router was designed completely to support Demo 4 Project. 3 Development and implementation 3.1 Architecture and technical characteristics The Cisco Connected Grid portfolio of solutions is designed specifically for the harsh, rugged environments often found in the energy and utility industries. These solutions include the Cisco 2010 Connected Grid Router (CGR 2010) and the Cisco 2520 Connected Grid Switch (CGS 2520), which have been designed to support the communications infrastructure needs of the energy delivery infrastructure across the generation, transmission, and distribution sectors. This infrastructure includes utility and customer owned energy infrastructure such as substation applications supporting electrical transmission and distribution, renewable generation.. Designed for highly secure, reliable, and scalable infrastructure, the CGR 2010 and CGS 2520 are an ideal platform to support the Smart Grid and other energy delivery infrastructure needs of customers. These ruggedized products have been extensively tested and are KEMA certified to meet challenging substation compliance standards, including IEEE 1613 and IEC The Cisco CGR 2010 is a rugged router optimized for use in the multitude of different communication networks found in the energy and utility industries (Figure 3). One example application for the Cisco CGR 2010 is for substation networks in harsh environments common in utility transmission and distribution substations. The CGR 2010 provides operators with the benefits of improved security, manageability, and network reliability. The CGR 2010 uses Cisco IOS Software, which is the operating system powering millions of Cisco routers deployed 2 Cired Paper LTE AND PLC TECHNOLOGIES FOR MV NETWORK SUPERVISION AND AUTOMATION : Tullio Zannoni, Fabio Campagna, Matteo Quarantelli 19 avril /27

10 worldwide. Cisco IOS Software delivers the benefits of integrated security for North American Electric Reliability Corporation/Critical Infrastructure Protection (NERC/CIP) compliance, quality of service, and network management to help ensure integrity and priority of operational data communications. Primary Cisco CGR 2010 features: Rugged industrial design, featuring no fans or moving parts, and an extended operational temperature range Substation compliance with IEC and IEEE 1613 for utility substation environments Integrated security to help address compliance with critical infrastructure protection mandates High availability design for maximum network uptime and redundancy Network and device management tools for deployments, upgrades, and remote monitoring Advanced quality of service (QoS) capabilities to support mission-critical communications such as substation communications such as SCADA (Supervisory Control and Data Acquisition) Comprehensive network security features based on open standards Figure 4: CGR 2010 The primary target of this new platform will be deployment in the transmission and distribution primary substations, in conjunction with the Cisco 2520 Ethernet Switch platform. The following is a deployment scenario envisioned for the Cisco 2010 Utility router within the Smart Grid power Substation Control Center and the Data Center. Substation automation promises to bring more automation and intelligence to the power grid network to address a myriad of utility challenges. Utilities are focused on how to improve grid reliability, enhance network security to meet regulatory requirements, and reduce operational expenses. The Cisco Connected Grid Router and Switch offer utilities a rugged networking solution to enable reliable and secure two-way communication for substation automation. Figure 3 shows a converged end-to-end IP network from the data center to the home. The CGR 2010 and the CGS 2520 are deployed in both transmission and distribution substations. Networking these points of presence provides network operators with greater visibility into grid assets and helps identify, isolate, and restore outages more efficiently 19 avril /27

11 Figure 5 CGS 2520 Cisco CGR 1120 is a ruggedized router designed for deployments in secondary substations for the power utility market. The CGR 1120 will extend Cisco s presence into the power distribution grid with the goal of creating an end-to-end IP communication network. It will serve as a platform for enabling new applications to drive greater power efficiency and reliability, lower energy consumption, and reduce greenhouse gas emissions. The CGR 1120 will provide distributed intelligence to create a Smart Grid that will support diverse applications on a converged network. Some examples are: Demand Response (DR); Distribution Automation (DA); Integration of Distributed Energy Resources (DER) (a.k.a., Renewable Energy Sources (RES) and Distributed Generation (DG)); The platform is designed to be din-rail mounted with estimated dimensions as laid out above. The integrated power supply supports 3 phase Alternate Current or Direct Current input and supports failover to DC if AC Power Supply fails (and vice versa). As you can see, it has 2 GE ports for WAN use either SFP based for Fiber connectivity or Cu ports. The six Fast Ethernet ports are to connect local devices in the secondary substation. The Serial ports supporting RS-232 and RS-485 can be used to connect legacy RTUs that using some legacy protocol. There is one console port for access to device configuration and one alarm port for digital input/digital outputs. The back side shows the 2 module slots along with antenna connectors for GPS and Wi-Fi. 19 avril /27

12 Figure 6: Cgr 1120 back and front view The Field Area Network (FAN) (a.k.a. Neighborhood Area Network (NAN)) is a critical component of the Smart Grid because it links together power generation and transmission to end consumption. Government agencies are creating various incentives for Distribution System Operators (DSOs) to improve power reliability and lower operational costs. In order to meet those goals, DSOs will need more comprehensive access to information within the distribution grid to be able to monitor power flows and react to changing conditions. A standards-based TCP/ IP communication network for the FAN will allow DSOs easier access to that information, integration with back end systems, and improved monitoring and control of their power assets within the distribution grid. Using IP as a common communication protocol will break apart the Smart Grid value chain between the applications and services, the communication network, and the power assets within the electrical grid thereby increasing application flexibility and network effect of new vendors for Smart Grid. The Cisco Industrial Ethernet 3000 Series (IE 3000 Series) is a family of Layer 2 and Layer 3 switches that bring Cisco's leadership in switching to Industrial Ethernet applications with innovative features, robust security, and superior ease of use. The Cisco IE 3000 series features: Industrial design and compliance Tools for easy deployment, management, and replacement Network security based on open standards Integration of IT and industrial automation networks The Cisco IE 3000 series is an ideal product for Industrial Ethernet applications, including factory automation, energy and process control, and intelligent transportation systems. The Cisco IE 3000 offers: 19 avril /27

13 Design for Industrial environments, including operation in extended temperatures, certification for elevated shock and vibration, and elevated surge protection A complete set of power input options, convection cooling (no fans), and DIN-rail or 19" rack mounting A modular design which will support 150 different hardware configurations Software design built on the catalyst Internetwork Operating System architecture. The Layer 2 switch has a specialized 2960 LAN Base image with a Layer 2+ feature set. The Layer 3 has a specialized 3750 IP Services image with a Layer 3+ feature set Support for leading industrial protocols, including Common Industrial Protocol (CIP) and PROFINET, including PROFINET v2 certification, with PROFINET conformance class B compliance Easy setup and management using the Cisco Device Manager Web interface and supporting tools, including Cisco Network Assistant. Easy switch replacement using removable memory, allowing the user to replace a switch without having to reconfigure - "Zero-Config" Replacement Support for IEEE1588v2, a precision timing protocol with nanosecond-level precision for high-performance motion applications Compliance to a wide range of Industrial Ethernet specifications covering industrial automation, ITS, power utilities, railway, and other markets Industrial PoE Figure 7: Industrial Ethernet 3000 Catalyst Switch 19 avril /27

14 The evolution proposed for Demo 4 consists in a new network automation system furnished with a suitable channel of communication between the DSO Primary electrical substation protections and the units of distributed generation that pertain to it with a view to resolving a series of problems relating to flow inversion and in particular to the current interface protection systems of active users and thereby allowing for an immediate increase in distributed generation in view of the future development of smart grids. To realize this infrastructure Cisco propose to use Cisco CGR2010 and CGS2520 in Primary Substation and CGR 1120 and IE3000 in Secondary Substation and in Customer Control System Cisco has implemented different technologies that are used to connect different Substations networks to transport data over a WAN: a) Tunneling GRE Tunnel. Tunneling the Ethernet frames on top of an IP GRE tunnel Layer 2 Tunneling Protocol. b) Encapsulations Ethernet over Multi Protocol Label Switching (MPLS); L2TPV3 PseudoWires (vlan ethernet pseudowire) 3 ; Virtual Private LAN Service (VPLS). Transport ethernet over TDM circuit on SONET/SDH IEC has a profile for securely sending both GOOSE and SV messaging over wide area networks IP Multicast/UDP All the new automation and control functions, designed according to the new scenarios in the MV grid, are totally based on IEC service suite. In the Demo4 DSO decided to adopt the IEC standard for any new development of IEDs and Smart Grid functions. All these functions introduce new telecommunication requirements which overcome the current communication solution in MV grid, based on point-to-point (SCADA-RTUs) and 4G LTE always-on connections. 3GPP Long Term Evolution (LTE), the latest mobile communication network, is a promising option for a Smartgrid. LTE was developed to fulfill mobile users demands for higher data rates and more stable service performance. Some White paper about 3GPP and LTE service operators announced that the main advantages 3 The pseudowire (or pseudo-wire) is an emulation of a point-to-point connection over a packetswitching network. The pseudowire emulates the operation of a "transparent wire" carrying the service, but it is realized that this emulation will rarely be perfect. The service being carried over the "wire" may be Asynchronous Transfer Mode (ATM), Frame Relay, Ethernet or time-division multiplexing (TDM) while the packet network may be Multi-protocol Label Switching (MPLS), Internet Protocol (IPv4 or IPv6), or Layer 2 Tunneling Protocol Version 3 (L2TPv3) 4 IEC is targeted at end-devices (IEDs) to provide efficient P2P and P2MP L3 transport of GOOSE and SV over the WAN with the capability to support authentication and integrity using HMAC. IED vendors however, will take time to adopt 90-5 into their products, and existing IEDs may not be upgradable. 19 avril /27

15 with LTE are to provide theoretical 5 high throughput (up to 300 Mbps in downlink, and 75 Mbps in uplink), low latency (less than 50 ms for control plane latency, and less than 5 ms for user plane latency), plug and play, Frequency (FDD) and Time Division Duplex (TDD) in the same platform, an improved end-user experience and a simple architecture resulting in low operating costs. However, LTE standard has not make plan for Smart Grid applications. The Smart grid communication network, latency is one critical technical requirement. New communication solutions have to enable: new logical data communication paths among substations real-time, and therefore always-on connections, everywhere. The first strategic constraint (no proprietary communication infrastructures), in Demo4 we adds the compliance with IEC standard, not just inside a single substation but for each cluster of substations. We have to remark that IEC standard was designed at the beginning for the automation inside a substation. It means that several services and applications, used in this IEC suite use layer 2 protocols (ISO/OSI stack), in order to match realtime requirements. By default GOOSE messages are stopped at a router interface. For GOOSE messages to be transported over the WAN a layer 2 transport technologies must be employed. These technologies either bridge or tunnel the layer 2 GOOSE frames over the WAN. Bridging and tunneling are techniques by which routers wrap and unwrap GOOSE packets at each end of the WAN link. Each of the following standards-defined techniques for transporting GOOSE over a layer 3 WAN has pros and cons in terms of latency, scalability, security, and quality of service (QoS) management for the GOOSE transport mechanism: 1. EoMPLS (Ethernet over MPLS) as defined in Standard RFC 4448 of the Internet Engineering Task Force has the least amount of overhead and is typically faster to transport as compared to the layer 2 tunneling protocols, which apply layer 3 headers. This transport method does not apply a layer 3 header so it is both difficult to encrypt and more challenging to apply quality of service policies. 2. L2TP (layer 2 tunneling protocol version 2) and L2TPv3 (layer 2 tunneling protocol version 3) as defined in Standards Request For Comment 2661 and Request For Comment 3931 of the Internet Engineering Task Force, respectively, are comparable. Although they have slightly more overhead than EoMPLS, they have much simpler configuration. They are also easier for the router to encrypt and are more manageable in terms of quality of service policies. 3. VPLS (virtual private LAN services) is not supported on any hardened equipment for substation use at the time of this writing, but it could be a powerful transport mechanism if ever integrated. These are defined industry-standard services which must be provided by the router suppliers as configuration settings in routers selected for the application at both ends of the WAN links. EoMPLS and L2TPv3 are point to point tunneling solutions, meaning they can connect a substation 5 LTE capacity depends on the following values: Channel bandwidth, Network loading: number of subscribers in a cell which impacts the overhead, configuration and capability of the system: whether it s 2 2 MIMO, SISO, and the MCS scheme. 19 avril /27

16 to a control center, but not connect two substations and a control center over the same tunnel. VPLS, by contrast, allows for multipoint connections, but this technology is typically only available on high end routers and has yet to be offered in a hardened substation router platform. Until the IEC communications standard is modified to include methods to make GOOSE messages that are addressable over a WAN, there are two mechanisms to bring GOOSE messages from one substation to the other substation. Large ethernet bridge domain can cause instability. Ethernet tunneling or encapsulation using a pseudowire emulation is a way to establish a secure channel through a public WAN. A virtual private network (VPN) is an example of such a secured channel. The process essentially establishes the behavior of IEDs in a local substation interacting with IEDs in a remote substation as if they were on the same LAN. A layer 2 GOOSE packet contains a standard Ethernet field defined in the IEEE 802.1q part of the full Ethernet standards. This field, sometimes called the 802.1q tag, contains a VLAN identifier (one of up to 4096 VLANs) and an 8-level priority tag. Digital Relays and controllers have the ability to communicate using IEEE 802.1q tagging. Figure 8: Ethernet frame with vlan tag The Ethernet switch functions (switches, or switch functions within switch/router combined units) can sort messages to eliminate the unnecessary traffic flows GOOSE is based on a layer 2 multicast frame, which is intended for use only on a local area network (LAN). This multicast frame floods to every live network port within the LAN environment. GOOSE messages are best effort publisher-subscriber transmissions, meaning that there is no confirmation that a published frame has been received by intended subscribers. To help ensure delivery of a frame, publishers send multiple copies of a critical message reporting a change at a decaying rate just in case one is lost (. as an example: 4ms, 8ms, 16ms, 50 ms, etc). GOOSE traffic needs to be forwarded by a router. The most logical way of doing this is to use VLAN technique. The GOOSE message to be sent to the outer world from the teleprotection device adds a VLAN tag that is recognized by the router. The router converts the GOOSE into a routable IP packet and transports it to its destination address Goose 6 is one of IEC protocol based on MULTICAST message, mapped on the layer 2 used in DEMO4 WAN (not only in substation LAN) to implement several Smart Grid functions. 6 GOOSE implement a Publisher/Subscriber model and the multicast transmission of frames is key to this. Each subscriber (an IED in this application) receives a copy of messages that it is interested in, and the publishers do not distinguish between the various subscribers. IEDs (various types) publish GOOSE messages containing digital or transduced analogue information. IEDs are the subscribers of GOOSE messages. This model is connectionless and therefore the publisher transmits multicast messages without expecting any acknowledgment. This is efficient and enables high levels of real-time traffic to be transmitted by Ethernet. 19 avril /27

17 Figure 9: goose ethertype value Different Industrial Cisco routers and switches are installed in different part of Electrical Grid. In the next picture are identified all specific location and are specified overall functions. 19 avril /27

18 Figure 10 : Complete scenario of Communication network and detail flows and functions The architecture solution has not just to offer real-time performances to the new Smart Grid functions, but it has to enable new communication flows between substation and an always-on communication, overcoming the current Wireless Wan communication between SCADA system and RTUs in the Medium Voltage (MV) Grid. Furthermore, the new network architecture has to face MPLS Service Provider network, who has in charge 4G coverage of Enel plants, satisfying scalability requirements. A Hub and spoke design 19 avril /27

19 was adopted, where the HUB node, based in Enel MPLS corporate network is essential for the aggregation, commutation traffic and security. Thanks to the flexibility of the operating system installed on the Head End router that performs the function of head end was implement a point to multipoint logical connection through a workaround designed, tested and implemented in the field.it was possible to overcome this limitation and transform our implementation in a multi-point connection allowing to replicate the message goose multicast messages to other router interfaces that performs the functions of Head End. GOOSE traffic to be collected through L2TPv3 tunnels and mixed/replicated in the Head End router, just terminate P2P L2TPv3 on ASR1000 sub interfaces, tag the sub-interfaces with 802.1Q and bridge the relative ethernet service-instances. Some details are illustrated in the next figure below. Figure 11: L2tpv3 transport and replicate Gosse packet over the Wan The Field Area Network is represented from 93 Cisco CGR 1120 installed in different Secondary substations. The CGR is a indoor router with some industrial characteristics used to aggregate FAN devices and provide a backhaul mechanism to a utility s data and/or operations center. To support Grid4Eu requirements, the architecture will support both IPv4 and IPv6 for future usage. Backhaul communication will be over secure communication tunnels using L2tpV3 over IPSec. The 4G Wireless WAN backhaul is provided by unique service provider and will be leveraged for the communication between the FAN CGRs and the edge access routers in HEAD END router. To 19 avril /27

20 ensure security the communication over service provider networks will use IPSEC tunnels for communications. Figure 12: Details about Secondary substation architecture to connect Active Customer (PV generation) Digital communications are paramount to control systems in the current power grid and will become more so in the smart grid. The power grid is typically monitored by Supervisory Control And Data Acquisition (SCADA) systems that collect data from control devices located at strategic points within the power network. However, the legacy and proprietary nature of these control systems is reflected on the underlying communication networks, being typically low bandwidth. Currently, power grids are monitored by legacy Supervisory Control And Data Acquisition (SCADA) systems, which exchange data and send commands to the field devices, through remote terminal units (RTUs). The RTU represents a contact point with field sensors and actuators through a field bus.. the SCADA system comprises a central Master Unit computer.. The role of the central host computer is primarily the processing of data collected from different field-based control devices and presenting them in a readable format to the operator. The communication network is central to this process as it allows data transfer between the control center and the field-based devices, and it comprises IT equipment such as routers, switches, modems, etc. The new Industrial equipment device (i.e. Cisco s indoor router CGR1120) installed inside to in MV/LV substation has four main scopes: 19 avril /27

21 Realizing Ethernet 7 switched Lan Infrastructure that provides and replace hardwired link services to all IED connection Enabling an IP infrastructure, allowing the routing, switching and tunneling (IPSec or L2TPv3). Translate the protocol of local RTUs (IEC ) to a correspondent full IP protocol (IEC ). Transport using an overlay protocol L2tpv3 IEC Goose protocol for new automation features. The telecommunication infrastructure was designed to enable the traffic of IEC GOOSE between substations. The function of protocol translation has been subject to the most significant functionality tests because of its innovative features CGR1120 (for indoor) implement a new dedicated hardware to be used in Secondary Substation and a new SCADA gateway software function to translate packet that arrive from RTU (UP) to ethernet Local Area Network Master Controller Scada (STM) Protocol translation functions from IEC to IEC performed by CGR1120 are: Receive data from RTUs (T101) and relay configuration commands from the SCADA Master (T104) to RTUs. Receive configuration commands from the SCADA Master and relay RTU data to the SCADA Master Since the CGR1120 router acts as a proxy master, the T101 represents always a master and T104 is configured to be a slave on the CGR router. The next figure below illustrate a similar scenario in Grid4Eu Demo: Figure 13: Scenario to transport Scada to 104 For this principal feature of translation, other three important enhancements linked to T104/T101 principal features of translation have been added: Exception handling (interlock feature) The CGR1000 can restart T104/T101 communication in the event of a connection failure. 7 There are three types of Ethernet message. The most common are unicast messages (received by a single device) and broadcast messages (received by all devices on the same LAN segment). Multicast messages are received by multiple devices that each share a common need for the message. 19 avril /27

22 Clock pass-through scheme The CGR 1000 can pass its clock setting at regular intervals to the downstream T101 RTU. Clock synchronization is denied on T104 side. Non-volatile event storage/recovery The CGR 1000 saves a copy of a T101 change event to non-volatile memory until the T104 side confirms delivery. 3.2 Lab tests Many Lab tests were done both in the laboratory and in the field. were made in different workshops with the involvement between the partners themselves. Several long and intensive sessions were conducted to certify the implementation of SCADA Gateway to support different RTU models and the Master Terminal Unit interoperability. During these sessions, it was also necessary to revise and extend some requirement that were added to existing features 8. Other important session test is conducted with other Demo 4 partner Selta to certify interoperability with DPLC prototype modem. Figure 14: Simple Lab Scenario to tested in Selta for interoperability with DPLC Modem 8 Exception handling (interlock feature), Clock pass-through scheme, Non-volatile event storage/recovery. 19 avril /27

23 Figure 15: A couple of Selta DPLC connected to different router CGR avril /27

24 3.3 Field implementation The five routers on a series of Secondary Substations are installed from last week of June These router are installed has been installed in the city and rural area of Cesena, in presence of Enel and Cisco people, the manufacturer of the Secondary Substation router. Some preliminary tests have been conducted. Secondary substation routers are integrated into the system of ancillary power supply of secondary substation by verifying that the sources of AC power and DC power would ensure full redundancy. Through some simple commands from the console of the router is to verify the accessibility and check in a first phase to measure the maximum IP latency between the IP address configured on the router and the IP address of the router that performs the ASR1000 aggregate functions to this network. Figure 16: First Cgr1120 router installation 19 avril /27

25 Figure 17: First IP Ping latency measurement 19 avril /27

26 19 avril /27

27 4 Technical results Throughout this project, the team leveraged SGAM Architecture development methodology, a rigorous program developed to extract and define business requirements, map these to specific communications needs, and deliver an optimal architectural design. We worked closely with Enel to capture and help to develop use cases across the business, aggregate relevant metrics, and create a suite of deliverables, including strategies for testing, production support, network integration, and holistic implementation. Based on these detailed findings, Cisco helped to develop a proposal for a scalable, highly modular, flexible architecture supporting not only a single standard (i.e., IEC 61850). This modular approach supports further cost savings, enabling larger implementation for major substations and smaller deployments for remote sites supplying fewer customers. The design also allows for a phased implementation for new Active secondary substations scenario and a retrofit to Passive secondary substation for existing sites. The Cisco team supported these design concepts with a set of technical and business requirements, as well as providing input for a cost-benefits analysis. Cisco was asked for communications equipment tough enough for very hot and rural extreme conditions in substations as part of Enel s transforming the electricity network into a smart grid. Cisco Connected Grid routers and switches create a reliable multi-service network in the substation. These industrial-grade routers and switches are designed for the substation harsh environment, and are compliant to industry standards including IEC and IEEE We offer a first result about Network Availability KPI 910 to apply to Secondary Substation and Hub router (asr1k). We offer you the evaluation of key performance indicator for the end-to-end communication. System is divided logically into different functional blocks. First and last block of contributions were defined on the basis of our internal tool others intermediate blocks (such as the exact value of the LTE Radio Access Network, the exact value of the Service provider's Core Network) were provided as a contribution from a survey in collaboration with the Service Provider. 9 See to general delivery dd4.4 Demonstration activity results 10 The final value of System Availability End-to-End = 99,99*99,999*99,9999*99,999*99,989= 99,9769% 19 avril /27