Whitepaper. Replacing Carrier TDM Services. Connecting Substations to the Network by Satellite

Transcription

1 Whitepaper Replacing Carrier TDM Services Connecting Substations to the Network by Satellite

2 Contents Introduction... 1 Critical Network Infrastructure... 1 Physical Network Connectivity... 2 Solving the Conundrum with Satellite... 3 Service Provider vs Private Satellite... 4 Reliability... 5 Security... 6 Latency... 6 Bandwidth... 6 Financial Considerations... 7 Eliminating VPN and Teleport Charges... 7 Minimizing Transponder Charges... 7 Reducing Installation Costs... 8 Conclusion All Purpose Transport Network Replacing Carrier TDM Services - Whitepaper

3 Introduction Utilities and pipeline industries have relied on 4 wire analog circuits leased from the carriers to transport Supervisory, Control, and Data Acquisition (SCADA) information to and from remote terminals/substations (RTU) and provide the infrastructure for Pilot Wire protection for decades. These services offered an effective low cost, low bandwidth transport medium to enable basic monitoring services. These Time Division Multiplexor (TDM) based networks, commonly used good old metallic cabling and have long provided the bread and butter connection infrastructure for both business and consumer services. However, over the last several years the carriers have viewed this infrastructure as a dinosaur, a relic of an age when voice was king and data was a word. These services are no longer profitable for the carriers and they do not have engineers who understand this technology and they have no interest in continuing to support them. Tier one carriers, such as AT&T, Verizon, and CenturyLink, among others have announced plans to discontinue all TDM based services. All services based on 2/4 wire lease lines have now been declared End of Life and in most cases no requests for new services are being accepted, however none of the carriers have been definitive with the dates they plan stop supporting them. Most major customers have been given end of service letters indicating services will be terminated between now and 2020 (under current FCC rules they are required to give 180 days notice) Critical Network Infrastructure For the vast majority of consumers and enterprise businesses this removal of services will not have a major impact. The number of homes who rely on a wireless phone is increasing every month and carriers are offering enterprise customers alternative services based on Ethernet and MPLS, leading to a steady decline in copper based services for the carriers. For most enterprise customers they have no alternative but to adapt to the new offerings as they are solely dependent on the carriers and also in most situations their applications can be fully supported on a packet based infrastructure. Also the remote locations can be easily and cost effectively connected to the network. Replacing Carrier TDM Services - Whitepaper 1

4 However, this does not apply to the utilities who still have critical applications that require a TDM (Time Division Multiplexing) infrastructure to deliver guaranteed timing and latency for critical monitoring of remote locations. The packet based alternatives (wireless or MPLS) are not necessarily suitable solutions today. Much of the equipment being monitored is old and does not have an IP/Ethernet interface and the sheer cost of upgrading that equipment makes a simple transition impossible. The carriers lack of understanding or appreciation of the impact that the removal of these services will have is causing utilities considerable pain and leaving them no choice but to scramble to find workable alternatives. Physical Network Connectivity The biggest issue with replacing the carrier TDM services is how will the substation be physically connected to the network? Even as Packet based solutions are being viewed as the logical migration path, these solutions still need a physical pipe to run over. 2 Replacing Carrier TDM Services - Whitepaper

5 Fiber will typically be the preferred option, however the high cost (up to $75,000 per mile) can make this difficult to justify, particularly for low bandwidth requirements. Microwave is usually the second choice, when fiber is not a viable option. A Microwave link needs line of sight connectivity, which can be issue in certain geographic locations. Carriers offering Ethernet services as replacements for the discontinued TDM services will, in most cases, have to lay new physical media to support those services, passing the cost onto the customer in the form of a NRE (Non-Recurring Engineering) charge, which can be as high as the cost of laying fiber (which it may well be!) Within the power Industry, for those who operate their own private networks, and have substations that cannot be connected using fiber, microwave or a carrier supplied Ethernet service, there is another option which allows a cost effective, easy to install replacement infrastructure that supports the current legacy (TDM) applications and provides an orderly controlled migration to the Next Generation Network that utilizes a packet based system a modern, private satellite network. Solving the Conundrum with Satellite Satellite communications have in the past typically been viewed as a solution of last resort due to bandwidth limitations, latency concerns, operating costs and operational concerns. substation connectivity. Significant improvements in satellite technology have taken place over the last few years resulting in new modern, private satellite systems. These offer a solution that operationally address the requirements for replacing many of the discontinued TDM service connections in a quick and cost effective manner. Thus making satellite communications a very valid third option for The table below shows the key applications supported in today s substations Application Data rate/volume Latency allowance Reliability Security (at end point ) (one-way) SCADA Medium/Low Low High High Smart metering Low/Very low High Medium High Operations data Medium/low Low High High Video surveillance High/Medium Medium High High Inter-site rapid response (for High/Low Very low Very high Very High example, Teleprotection) Distribution automaton Low/low Low High High Distributed energy Medium/low Low High High management and control (DER, storage, PEV) Mobile workforce (Push to Low/low Low High High Talk) Corporate data Medium/low Medium Medium Medium Corporate voice Low/very low Low High Medium Replacing Carrier TDM Services - Whitepaper 3

6 Service Provider vs Private Satellite The following figure shows a typical old technology satellite system. Such systems are usually referred to as VSAT systems (Very Small Aperture Terminal) a reference to the antenna size, which was considered very small when such architectures were first developed. Typical VSAT System VSAT systems commonly share the following characteristics: A star architecture with a large antenna at a hub station and smaller antennas at the Remote Terminal Units (RTU). A terrestrial VPN from the hub to the utility Network Operational Center (NOC). A satellite owner who charges for bandwidth. A Teleport owner who charges for services on the system, i.e. a middleman between the satellite operator and the utility customer. The transmitter at the teleport operates at a data rate higher than the inbound transmitter (located at the Remote Terminal). 4 Replacing Carrier TDM Services - Whitepaper

7 Private Satellite System A modern private satellite system as shown in the diagram above delivers significant benefits over the traditional VSAT: The primary hub is located at the utility NOC, with a secondary hub located at the Backup NOC. o this eliminates the need for a VPN. o provides dedicated bandwidth to the hubs. Provides the utility with complete control of the network. Takes out the middleman. Reliability Power Industry studies show that the lost productivity to the economic life of an area affected by an outage can never be fully made up. The reliability standard in the Power Grid control, monitoring and operations systems is now higher than what the Carriers are able to deliver. In a Critical Infrastructure, network reliability is of paramount importance. The Goal is % (5 9s) up time, no more than 5 minutes of outage in any 12-month period-of-time, which is achievable with a satellite system. The carrier networks were once built to this standard however they no longer expect to meet this level of reliability and if they meet a 95% uptime or availability mark, they consider that good enough. That can mean several hours of down time on a regular basis. Which may be good enough for dial tone, but is not acceptable for a Power Grid. Replacing Carrier TDM Services - Whitepaper 5

8 Security Security is of paramount importance in a utility network as NERC/FERC continue to develop the CIP mandates. As a private network, satellite offers extremely high levels of security: The utility maintains full control, end to end of what is effectively a point to point connection. Physically hacking into the network is extremely difficult. The satellite system is totally segregated from any public infrastructure. The satellite links support full AES 256 encryption. A modern, private satellite system has no terrestrial back haul, so no VPN component. The satellite link is a OSI level 1 pipe offering a high level of cyber security. Latency A traditional VSAT system suffers from extended latency issues primarily due to the use of remotely located shared HUBs that add delays due to firewalls at the Teleport and the inclusion of the VPN terrestrial transport network. A private satellite network still has a propagation delay when the signal travels into space (the laws of physics cannot be overcome!). However, removing the shared hub and terrestrial VPN components bring that roundtrip delay below 0.7s, which is slightly more latent than terrestrial options, but works with the majority of utility applications. This propagation delay is highly predictable, so the telecom engineers can design the application support knowing they have a fixed, known delay using a range of flexible access schemes that minimize the latency to meet application response requirements, such as: Fixed TDMA. Enhanced Slotted Aloha (Random Access). Slot request and reservation (SRR). Any of these approaches can be intermixed to optimize the network. There is no one size fits all answer to a specific service area and application. Bandwidth Bandwidth availability on VSAT systems has always been an issue (both availability and cost!). Today s modern private systems can address both of these issues with bandwidth available from 19.2 kbps up to 4Mbps, in a very cost effective solution. 6 Replacing Carrier TDM Services - Whitepaper

9 Financial Considerations Traditional VSAT systems have always suffered from a high cost of operation due to incurring the following: VPN charges Teleport / Satellite reseller charges Satellite transponder charges Eliminating VPN and Teleport Charges A private satellite system eliminates the VPN and reseller/teleport charges and improves reliability by placing the center of the star (the Hub) on a utility s location, usually at the Network Operations Centre. The key to making it practical to have the satellite hub on the user site is to reduce the size of the hub antenna. It is relatively straight forward to mount a 1.8 m antenna on the roof of an office building, but very challenging to mount a 3.5 m antenna in the same location, with the 3.5 m antenna weighing in about 8 times heavier than a 1.8 m antenna. The use of smaller antennas is accomplished by minimizing the satellite link rate (but not the user data rate). Minimizing Transponder Charges To reduce the satellite transponder charges the system needs to operate at the minimum viable data rate for the target data volumes and delay times. In most utility systems there is usually an imbalance between the inbound and outbound direction, with the inbound (towards the NOC dominating). With modern satellite variable rate modems and link independence each direction can be optimized individually. This imbalance in traffic flows is the opposite of most corporate networks. Traditional TDMA satellite systems which are corporate focused, leave spectrum poorly utilized due to their shared nature. VSAT systems assume symmetric or outbound dominance, leaving a lot of unused capacity in the outbound direction when supporting utility systems, but data charges are billed as if traffic is flowing symmetrically. A modern private system adjusts each direction independently so that the spectrum is optimally used with data rates that are configured independently for each direction so that the satellite spectrum is used to best effect. Replacing Carrier TDM Services - Whitepaper 7

10 Reducing Installation Costs The major costs associated with installing a satellite system are related to: Antenna mounting. RF cable installation. Electrical and ground wiring. RF cabling and electrical wiring do not vary much between systems, but antenna size does. Why does a VSAT system use a big hub antenna and why does a modern private system operate with a small antenna? The answer is simple: over-the-satellite bit rate. Commercial service offerings use high data rate satellite carriers operating in a TDM/TDMA structure where multiple customers share carriers with each other. Each has a small contracted data rate, but the overall system runs at a high data rate. When the terminal is transmitting or receiving it is running at the system rate (fast), not the contracted data rate (slow). TDM/TDMA VSAT System The error performance of a digital data communication system is set by the ratio of the receive energy per bit (Eb) and the noise in the system (NO) or Eb/NO. Note that this is energy (joules) per bit, not energy per unit of time (watts). At faster data rates there is less time to transmit the same amount of energy into the radio channel meaning that a higher transmit power is needed. Thus the power scales directly with the transmitted data rate, not the user information sent. 8 Replacing Carrier TDM Services - Whitepaper

11 For example, if we consider two satellite systems, one operating at 1 Mbps and one operating at 19.2kbps (User A), the 1 Mbps will require over 50 times (17 db) the power for the same link error performance and availability. The extra receive power can be obtained using one, or more of the following techniques: Increasing the transmit antenna size Increasing the transmit power Increasing the receive antenna size Reducing the noise performance of the receiver Adding more error correction overhead The first two items in the above list increase the effective transmitted power (called EIRP, Effective Isotropic Radiated Power). The latter three items improve the performance of the receiver. In practice VSAT receivers have low noise figures compared to other radio systems and the amount of improvement left to be had is small. Increasing overhead increases the amount of satellite bandwidth consumed with most systems using 1/2, 2/3 or 3/4 rate codes. Since all users are charged for bandwidth, modern private systems are designed not to consume any more satellite bandwidth with FEC than necessary. From a practical perspective higher speed system designers use both larger antennas and larger power amplifiers to achieve that 17.2 db in extra system gain when going from 19.2 kbps to 1 Mbps. It is the nature of most geostationary satellite systems that enable the modern private system to have the antenna sizes asymmetric, that is by having a larger antenna at one end it can reduce the antenna size at the other. To reduce overall system costs, the center of the star (Hub) uses the larger antenna size and the remotes the smaller. The table below shows a typical approach to making up a system gain difference of 17 db. Parameter TSAT System 1 Mbps VSAT Note User Data Rate 19.2 kbps 19.2 kbps Both systems supporting same system max throughput. Satellite link Rate 19.2 kbps 1 Mbps Ignore overhead as all systems add similar amounts of control and FEC. Hub Antenna 1.8 m 3.5 m 5.7 db difference Remote antenna 1.2 m 1.8 m 3.5 db difference Transmit power 0.33 W 2.1 W 8.0 db difference By operating at a data rate as close as practical to the user data rate the modern private system, with its variable bit-rate modems, gives a system with smaller antennas. These smaller antennas lower the installation cost and allows the hub to be located on the utility s premises close to the NOC. With a modern Replacing Carrier TDM Services - Whitepaper 9

12 private system, the entire earth station system is on the utility s property eliminating the delays, expenses and reliability issues associated with teleports and VPNs. Conclusion The announcements from the carriers that the TDM services are to be discontinued has created consternation and worry among the utility companies. The enormity of the transition in a relatively short time frame resulted in the Utilities Telecom Council (UTC) setting up a working group to help their members with the decisions involved in addressing the changes. There are alternative infrastructure solutions available today, each with its benefits and limitations as shown in the table below: Feature Carrier Services TDM Microwave Fiber Satellite Line protection (POTT/DCB) Yes yes yes yes Current Differential yes no yes no SCADA yes yes yes yes Backhaul Communications yes yes yes yes Broadband yes yes yes yes Data Rate Mbps Mbps Gbps Mbps Time to deploy No longer available long long short Service disruption high low high low Cost effectiveness Low* medium** low high *Carriers are increasing existing circuit significantly **Assumes licensed radio Modern private satellite systems have been designed for use in critical infrastructure utility networks, supporting a wide range of utility applications in an optimum manner through use of variable bit-rate modems and asymmetric data flows that match a utility s unique requirements. The modern private satellite architecture results in the following benefits: Smaller and lower cost antennas. Minimum satellite bandwidth costs. Lower cost installations. A simplified network with no need for a Teleport or VPN delays and charges. Entirely utility-owned and located infrastructure. 10 Replacing Carrier TDM Services - Whitepaper

13 FAE Telecom s position is that there is no one single answer, no one size fits all solution. Each substation connection will need to be looked at individually and the ideal connection selected based on the application criteria. Fiber is typically the primary choice when the high cost can be justified. If fiber is not an option then the second choice is usually microwave, however geography, distance, and installation time are key factors that can rule out a microwave link. Modern private satellite systems provide an extremely viable third option that offers: Ubiquitous coverage - covers anywhere, regardless of distance or topography. Quick and easy to install, and maintain. Direct connectivity, no intermediary hops. Minimal impact on environment/landscape. Different modes of failure. A replacement carrier service is the solution of last resort. The lack of understanding and appreciation of the utilities requirements and the carriers current business model make them a very unstable partner for building a critical monitoring and control network. All Purpose Transport Network The key issue with the replacement strategy is adopting a migration strategy that allows the replacement infrastructure to support the TDM based applications that are critical to monitoring the network today and that allows a managed organized migration to the packet based applications that are the future. The three key alternatives fiber, microwave and satellite all allow that migration. They are pipes that the utilities own and control and will carry any type of communications traffic. The migration path will be what we call an APTN, All Purpose Transport Network. This is a multifaceted, multi-component infrastructure that allows for diverse application support. At the edge, will be multiple access infrastructures, fiber, microwave and satellite. In the regional network, it will be fiber (SONET/COE) or microwave and in the core fiber (SONET/COE). Running over this infrastructure will be multiple applications with traffic management at the edge being TDM, COE, MPLS- TP or a mix. At the regional network this could be COE, MPLS- TP, MPLS or again a mix. Replacing Carrier TDM Services - Whitepaper 11

14 1 Liberty Lane East #13 Hampton, N.H GO-FAETEL Replacing Carrier TDM Services - Whitepaper