THETARAY ANOMALY DETECTION ALGORITHMS ARE A GAME CHANGER

Utility Network UPGRADES MADE SIMPLE MATHEMATICAL EVOLUTIONS FOR RISK MANAGEMENT: THETARAY ANOMALY DETECTION ALGORITHMS ARE A GAME CHANGER WHITEPAPER 1

THE UTILITY TELECOMMUNICATIONS INFRASTRUCTURE The communications networks deployed by utility companies (power, water, transportation, etc.) have unique requirements to ensure continuous and safe operation of the services that they provide. Along with traditional data traffic, utility networks must also reliably transmit tele-protection (C37.94), Supervisory Control and Data Acquisition () traffic, serial management signals, and others. These lower speed signals have strict requirements for latency, jitter, round-trip delay, symmetry, and always-on connectivity, and in the past were typically carried on relatively expensive service-specific equipment with SONET high speed interfaces. POWER TRANSPORTATION WATER OIL & GAS Often utility networks are managed as two separate subnetworks one operations network carrying delay-sensitive signals such as and C37.94 and one IT network carrying data traffic for communications, database backup, and other applications. The physical fiber infrastructure between sites must be optimized for both networks simultaneously, and combining networks on common fibers is usually desired. To combine networks, equipment can be deployed that either pools signals at the optical layer or multiplexes sub-rates at the electrical layer, but often multiple fibers are used to avoid complexity. The use of multiple fibers adds to the operational costs of the network due to leasing costs and physical inventory management, but in the past those costs have been justified due to the lack of cost-effective alternatives. Until recently, specialized equipment for multiplexing utilityspecific signals was based on SONET TDM standards for transmission. As network demands have increased, especially on the data traffic part of the network, it is increasingly necessary to upgrade all equipment that is in the data path. However, SONET as a transmission protocol has effectively been depreciated and new SONET equipment is not being developed or supported, increasing the operating costs of existing networks and limiting upgrade paths. This leaves utilities with limited options for modernizing and upgrading their communications networks. ADVANCED TECHNOLOGY ALTERNATIVES SONET as a technology was designed to carry TDM traffic based on signal speeds that are a multiple of voice signals over traditional telephone service provider networks. When originally designed, SONET was engineered with characteristics that are particularly well-suited to services like C37.94,, and. For example, SONET has very low latency, especially when compared to packet-based services such as IP/Ethernet. SONET does a good job of combining lower speed signals together into a higher speed signal for efficiency. SONET is a timed protocol, so timing can be maintained through the network. SONET has built-in resiliency protocols that automatically recover from signal failures in the network. SONET was designed with a 50ms maximum recovery time for protected services. And SONET has built-in communications channels to allow remote management and visibility of equipment in other sites. 2

SONET s failures were not in the technology design, but in the evolution of modern networks. Network services are no longer based on voice, but on data and data protocols operating at different speeds than voice (e.g. 10Mbps, 100Mbps). The disparity between voice and data speeds results in inefficient SONET networks, wasted bandwidth, and underused investment. Additionally, SONET is overengineered for modern networks with overhead signaling bytes reserved for functions that are no longer required. This wasted overhead bandwidth results in even more inefficient networks, increasing network costs. For modern networks, new protocols have been designed that are more appropriate for data-centric networks. One successor to SONET is the OTN (Optical Transport Network) protocol, which is also a circuit-based protocol like SONET with many of the same features as SONET, but is based on more modern network requirements. For many telecommunications providers with SONET networks, a transition to OTN makes sense. However, for a utility network with an ongoing need to carry very low speed signals such as C37.94,, and, OTN can be overkill as the lowest defined payload (ODU0) is designed to carry Gigabit Ethernet signals. A more granular technology is needed. MPLS-TP is an extension to standard MPLS that is designed to emulate many of the beneficial features of SONET in a packet environment. Standard routed MPLS, usually referred to as IP/MPLS, is a mainstay of core packet networks and is ideal for the data traffic side of the utility network. MPLS-TP, which is fully interoperable with IP/MPLS, is a connectionoriented extension that enables low latency, deterministic traffic routing, timing transport, and simple provisioning. And because MPLS-TP is a packet protocol, it can be scaled to whatever size service needs to be transported, including low speed signals such as those required by utility networks. All that is required is equipment, such as ECI s Neptune product line, with the ability to convert the low-speed signals into MPLS-TP packets via a process known as circuit emulation. MPLS-TP KEY OBJECTIVES ARE: To enable MPLS to be deployed in a transport network and operated in a similar manner to existing transport technologies (SDH/SONET/OTN). To enable MPLS to support packet transport services with a similar degree of predictability, reliability, and OAM to that found in existing transport networks. 3

ADVANCED TECHNOLOGY ALTERNATIVES MPLS-TP is designed to remove all of the complexity inherent in MPLS and add back some of the capabilities that made SONET ideal for delaycritical, connection-oriented services. Just as with SONET, a circuit in MPLS-TP is deterministic - defined over a specific route so that the links being used can always be determined. This means that the latency and other properties of the service will not vary over time, as could happen in a routed IP/MPLS network where the path is indeterminate and can change. Redundancy in MPLS-TP can likewise be defined on a determinate path to ensure that switching is very fast and always fails over in predicable ways (sub-50ms recovery times). MPLS-TP circuits can be defined to be symmetric so that the properties are the same in both directions. Just as with SONET, the control and communications information is carried on the same path as the traffic so failures in other parts of the network do not affect operation. The stripped-down nature of MPLS-TP ensures extremely low latency through a network, on the order of SONET circuit latency. Combining MPLS-TP with packetbased synchronization protocols such as SyncE and IEEE 1588v2 further enhance the capabilities of MPLS-TP to replace SONET 1. DISCARDED OVERLAP ADDED Once the low speed signals have been converted to MPLS-TP, they can be carried over a dedicated MPLS-TP backbone or over an IP/MPLS data core. The advantage of this flexibility is that both sides of the utility network requirements operations and IT can now be efficiently combined onto a common core network without compromising the capabilities of either network. Dual stack equipment supporting both protocols, like ECI s Neptune product line, can optimize for both protocols simultaneously. The network can then be upgraded, expanded, and enhanced much more easily as network demands increase and new services are added. Setting up the routes used by MPLS-TP and integrating with optical and IP/MPLS networks has been made extremely simple by modern management systems. Creating an MPLS-TP route through a network is a simple a point-and-click operation when using management systems like ECI s LightSoft. Network operators who are familiar with setting up circuits through SONET networks will find the process of setting up services over MPLS-TP networks to be at easy or even easier than their current experience. ECI s Muse modular software suite can further increase productivity and insight into the network beyond what can be achieved with a legacy SONET network. Maintain predictable and deterministic performance over a common packet network infrastructure.! CRITICAL NEEDS? Low latency, low jitter, and accurate timing Defined, very fast resiliency Symmetric operation Risk free transition to packet HOW Strictly connection oriented Transport like protection Transport like OAM Transport like operation 1 For more detailed information about MPLS-TP, please see the ECI Application Note MPLS-TP for Mission-Critical Networks, available at www.ecitele.com. 4

Another advantage of moving to a modern technology like MPLS-TP is that the evolution to an updated protocol enables an evolution to updated equipment such as packet-optical transport platform (POTP) equipment. POTP equipment combines the capabilities of MPLS with the capabilities of a modern optical transport network. Advantages of POTPs include the ability to transport signals over much longer distances (80km to several thousand km) and the ability to use wavelength division multiplexing (WDM) to increase the amount of bandwidth carried on each fiber. The costs associated with modern WDM networks above those of non-wdm networks are minimal, and WDM is a now a very attractive and cost effective way to increase distance or fiber capacity in a utility network. Updated equipment also has more advanced integrated security options, allowing utility operators to comply with the latest security regulations. For example, ECI s Neptune and Apollo product lines include in-line encryption of the optical and packet layers of the network for enhanced security. The Neptune product line can also be equipped with security functionality based ECI s Mercury NFV platform. This industry standard Network Function Virtualization (NFV) platform provides additional security functionality such as protection against network-based attacks. ECI s holistic Muse cyber security suite can provide end-to-end security for utility networks by incorporating and coordinating security solutions at all layers of the network. NETWORK EXAMPLE A simplified utility network is shown in the diagram below. Edge locations collect management information via to two redundant centralized management locations. Tele-protect C37.94 traffic is routed among the edge locations. circuits for voice, timing, and other traffic is collected from specific sites and routed to the public switched telephone network (PSTN) via redundant connections at a centralized location. IT Ethernet () data connections are routed from remote locations as well as around the central IT network. Some of the data connections are routed between sites, and some are handed off to the wide area network (WAN), including remote backup locations and the internet. C37.94 C37.94 Figure 1: Simplified Utility Network. C37.94 PSTN C37.94 WAN C37.94 5

This network is served by two separate technologies, SONET and Ethernet, as shown in the diagram below. The SONET network carries the TDM traffic, including,, and C37.94. This traffic is carried on a set of interconnected OC-3 and OC-12 rings using specialized SONET equipment designed to carry traffic. At interconnection points, ports on the SONET equipment must be reserved to route traffic between rings. Ethernet traffic is carried on a set of Ethernet switches located in strategic locations with a hub-and-spoke architecture that relies on routing protocols for protection. These separate networks are managed independently, use independent resources such as fibers and real estate, and require different operational skills. However, due to the different requirements on each traffic type (e.g. latency, protection, scalability), these technologies were the best choice when initially installed. Ethernet Network SONET Network C37.94 C37.94 Figure 2: Legacy SONET (TDM) and Ethernet (packet) network connectivity. OC-3 C37.94 OC-3 PSTN C37.94 OC-12 OC-3 WAN OC-12 C37.94 6

In the updated, modern evolution of this network shown below, both SONET and Ethernet technologies have been migrated to a combined IP/MPLS and MPLS-TP network running over an infrastructure based on low-cost Gigabit Ethernet. The result is a network based on a common technology that enables combining the TDM (, ) traffic and IT (Ethernet) traffic on the same hardware and management system. At the edge nodes, the MPLS equipment uses circuit emulation to carry the, C37.94, and RS-232 traffic. This is only possible if the equipment is designed to meet the strict protection and latency requirements in tele-protect and TDM traffic. MPLS-TP, as a deterministic protocol, can be designed to meet those requirements and ECI s implementation on the Neptune packet/ optical platform has been demonstrated to meet or exceed the tele-protect failover timing requirements. For protection, the TDM traffic is protected with pre-determined MPLS-TP routes ensuring that the failover routes will still meet the requirements of the services. The IT traffic can also be protected this way or via Layer 3 routing protocols, depending on the needs of the services and the expertise of the network managers. C37.94 C37.94 Figure 3: Modern dual-stack MPLS network connectivity. C37.94 nx PSTN C37.94 nx WAN C37.94 Building the updated network is simple, as MPLS-TP circuits can be established via simple point-and-click connections just as easily if not easier than SONET circuits were established in previous TDM networks. IP connections can be set up either using the techniques generally used in IT departments or via advanced intelligent network management systems like ECI s LightSOFT. Managing alarms and events on the network is also simplified, as only one common technology infrastructure must be managed. This allows much easier correlation of events for fault isolation and preventive maintenance. On most links, at least initially, a single Gigabit Ethernet circuit is sufficient for all traffic in both working and protection/ failover conditions. However, in the core of the network more than one Gigabit Ethernet circuit may be required between main locations. One of two options are available on those routes. The network can be equipped with 10Gigabit Ethernet links, which allows significant room for growth but has a higher initial cost and will require an additional upgrade if the 10Gbps limit is reached. Alternatively, wavelength division multiplexing (WDM) can be used to combine multiple 1Gbps links onto different colors (wavelengths) of light on a single fiber, allowing up to 40, 80, or more 1Gbps circuits to share the same fiber infrastructure. WDM provides the ability to start with a few Gbps and expand to a much higher capacity long term with a similar up-front cost and a few extra pieces of equipment to provide the wavelength multiplexing. WDM also has the benefit of allowing optical signals to be amplified for transmission over much longer distances (80km to over 1000km). Modern packet-based equipment such as ECI s Neptune product line include extensive security measures to ensure compliance with federal regulations on network security. Additionally, modern WDM equipment such as ECI s Apollo product line include optical-layer encryption for an additional level of security against intrusion. By deploying modern equipment, security capabilities of the utility network both the TDM network and the IT network - can be expanded even further with new innovations in network technologies that are currently being developed and deployed. 7

FUTURE EXPANSION Once a common infrastructure is in place, upgrading the capabilities and capacities of the network is greatly simplified. The common MPLS backbone can easily be expanded to new locations, regardless of the services to be offered in those locations. As the move to smart grid, internet of things (IoT) capability, and state-of-the-art customer platforms happens over the next few years, data center integration with the utility network will become critical, further increasing demands on the network. Capacities on individual links can be increased by upgrading the speeds of the links (from 1Gbps to 10Gbps, for example) or by adding wavelengths to the WDM network. By moving to a modern network design, innovations such as software defined networking (SDN) and network function virtualization (SDN) can be considered in future plans. Ethernet speeds up to 100Gbps are now defined and readily available, but are generally overkill for most utility networks. 10Gbps Ethernet, however, has come down greatly in price and is a legitimate upgrade strategy for all segments of the utility network. Combining 10Gbps Ethernet with WDM can provide up to 800Gbps of bandwidth on any given link, which is more than enough for the foreseeable future. However, since WDM bandwidth is only deployed when needed one wavelength at a time it is truly a pay-as-you-grow technology that does not burden the network operator with onerous upfront costs. An MPLS-based common infrastructure enhanced with WDM can, therefore, be used to design a combined utility network that is affordable today with enough upgrade potential for any foreseeable future. This nearly unlimited bandwidth potential has been employed by some utilities to enhance their evolution into utelcos, or utilities that offer telecommunications services either as a wholesale service (to other carriers) or retail service (to end users). 8

SDN & NFV FUTURE SDN capabilities include functions such as bandwidth on demand, network automation, service on demand, and network segmentation. For a generally static utility network, there may not be much benefit to SDN, although functions like remote data backup could certainly be made less operations intensive. However, by deploying a modern network infrastructure the possibility of network enhancement via SDN is left open for the future. Should the utelco path be chosen, SDN can make the transition to and maintenance of a more dynamic network simple to deploy and manage. ECI s Muse software suite is designed to provide these and other SDN capabilities today and a migration to SDN capabilities in the future. NFV capabilities in a utility network do have some usefulness today, especially in the security arena. By deploying hardware like the ECI Neptune platform with Mercury NFV capabilities and Muse holistic cyber, security functions can instantly and flexibly be deployed at any point on the network based on proactive and reactive security decisions. Additionally, NFV functions like de-duplication and deep packet inspection can help keep unwanted traffic out of the core network, extending the time between required network upgrades. 9

ECI S ELASTIGRID UTILITY SOLUTIONS ECI s ElastiGRID contains all the necessary components for successful evolution from a legacy utility network to a modern utility network to a utelco. ECI s pay-as-you-grow architecture and extensive portfolio breadth ensure that investments are linked to actual revenues. Converged L1 to L3 from metro access to the core ensures support of any service and scale to support any traffic volume. ElastiGRID s extensive security schemes, incorporating holistic approach to protect both IT and OT networks, ensure that the initiative for new revenues will not jeopardize the utility s core business. With a proven global track record for serving both utility internal communication and utelco business, ECI is well-positioned to help you take your business to the next level. Contact us today to learn more about ECI s ElastiGRID Utility Solutions 10 ABOUT ECI ECI is a global provider of ELASTIC network solutions to CSPs, utilities as well as data center operators. Along with its long-standing, industry-proven packet-optical transport, ECI offers a variety of SDN/NFV applications, end-to-end network management, a comprehensive cyber security solution, and a range of professional services. ECI's ELASTIC solutions ensure open, future-proof, and secure communications. With ECI, customers have the luxury of choosing a network that can be tailor-made to their needs today while being flexible enough to evolve with the changing needs of tomorrow. For more information, visit us at www.ecitele.com