1. Introduction The nature of the public network has changed. Demand for Internet Protocol (IP) data is growing at a compound annual rate of between 100% and 800%1, while voice demand remains stable. What was once a predominantly circuit switched network handling mainly circuit switched voice traffic has become a circuitswitched network handling mainly IP data. Because the nature of the traffic is not well matched to the underlying technology, this network is proving very costly to scale. User spending has not increased proportionally to the rate of bandwidth increase, and carrier revenue growth is stuck at the lower end of 10% to 20% per year. The result is that carriers are building themselves out of business. Over the last 10 years, as data traffic has grown both in importance and volume, technologies such as frame relay, ATM, and Point-to-Point Protocol (PPP) have been developed to force fit data onto the circuit network. While these protocols provided virtual connections a useful approach for many services they have proven too inefficient, costly and complex to scale to the levels necessary to satisfy the insatiable demand for data services. More recently, Gigabit Ethernet (GigE) has been adopted by many network service providers as a way to network user data without the burden of SONET/SDH and ATM. GigE has shortcomings when applied in carrier networks were recognized and for these problems, a technology called Resilient Packet Ring Technology were developed. RPR retains the best attributes of SONET/SDH, ATM, and Gigabit Ethernet. RPR is optimized for differentiated IP and other packet data services, while providing uncompromised quality for circuit voice and private line services. It works in point-to-point, linear, ring, or mesh networks, providing ring survivability in less than 50 milliseconds. RPR dynamically and statistically multiplexes all services into the entire available bandwidth in both directions on the ring while preserving bandwidth and service quality guarantees on a per- -1-
customer, per-service basis. And it does all this at a fraction of the cost of legacy SONET/SDH and ATM solutions. Data, rather than voice circuits, dominates today's bandwidth requirements. New services such as IP VPN, voice over IP (VoIP), and digital video are no longer confined within the corporate local-area network (LAN). These applications are placing new requirements on metropolitan-area network (MAN) and wide-area network (WAN) transport. RPR is uniquely positioned to fulfill these bandwidth and feature requirements as networks transition from circuit-dominated to packet-optimized infrastructures. Table 1. Resilient Packet Ring Technology Key Features Resilience Services Efficiency Scalable Proactive span protection automatically avoids failed spans within 50 ms. Support for latency/jitter sensitive traffic such as voice and video. Support for committed information rate (CIR) services. Spatial reuse: Unlike SONET/SDH, bandwidth is consumed only between the source and destination nodes. Packets are removed at their destination, leaving this bandwidth available to downstream nodes on the ring. Supports topologies of more than 100 nodes per ring. Automatic topology discovery mechanism. 2. RPR Operation -2-
RPR technology uses a dual counter rotating fiber ring topology. Both rings (inner and outer) are used to transport working traffic between nodes. By utilizing both fibers, instead of keeping a spare fiber for protection, RPR utilizes the total available ring bandwidth. These fibers or ringlets are also used to carry control (topology updates, protection, and bandwidth control) messages. Control messages flow in the opposite direction of the traffic that they represent. For instance, outer-ring traffic-control information is carried on the inner ring to upstream nodes. Figure 1. RPR Terminology By using bandwidth-control messages, a RPR node can dynamically negotiate for bandwidth with the other nodes on the ring. RPR has the ability to differentiate between low- and high-priority packets. Just like other quality of service (QoS) aware systems, nodes have the ability to transmit high-priority packets before those of low priority. In addition, RPR nodes also have a transit path, through which packets destined to downstream nodes on the ring flow. With a transit buffer capable of holding multiple packets, RPR nodes have the ability to transmit higher-priority packets while temporarily holding other lower-priority packets in the transit buffer. Nodes with smaller transit buffers can use bandwidth-control messages to ensure that bandwidth reserved for high-priority services stays available. The RPR MAC -3-
One of the basic building blocks of RPR is Media Access Control (MAC). As a Layer-2 network protocol, the MAC layer contains much of the functionality for the RPR network. The RPR MAC is responsible for providing access to the fiber media. The RPR MAC can receive, transit, and transmit packets. Figure 2. MAC Block Diagram Receive Decision: Every station has a 48-bit MAC address. The MAC will receive any packets with a matching destination address. The MAC can receive both unicast and multicast packets. Multicast packets are copied to the host and allowed to continue through the transit path. Matching unicast packets are stripped from the ring and do not consume bandwidth on downstream spans. There are also control packets that are meant for the neighboring node; these packets do not need a destination or source address. Transit Path: Nodes with a non-matching destination address are allowed to continue circulating around the ring. Unlike point-to-point protocols such as Ethernet, RPR packets undergo minimal processing per hop on a ring. RPR packets are only inspected for a matching address and header errors (TTL=0, Parity, CRC). -4-
Transmit and Bandwidth Control: The RPR MAC can transmit both high- and low-priority packets. The bandwidth algorithm controls whether a node is within its negotiated bandwidth allotment for low-priority packets. High-priority packets are not subject to the bandwidth-control algorithm. Protection: RPR has the ability to protect the network from single span (node or fiber) failures. When a failure occurs, protection messages are quickly dispatched. RPR has two protection mechanisms: Wrapping: Nodes neighboring the failed span will direct packets away from the failure by wrapping traffic around to the other fiber (ringlet). This mechanism requires that only two nodes participate in the protection event. Other nodes on the ring can send traffic as normal. Figure 3. Wrapped Traffic Flow Steering: The protection mechanism notifies all nodes on the ring of the failed span. Every node on the ring will adjust their topology maps to avoid this span. Regardless of the protection mechanism used, the ring will be protected within 50 ms. Topology Discovery : RPR has a topology discovery mechanism that allows nodes on the ring to be inserted/removed without manual management -5-
intervention. After a node joins a ring, it will circulate a topology discovery message to learn the MAC addresses of the other stations. Nodes also send these messages periodically (1 to 10 seconds). Each node that receives a topology message appends its MAC address and passes it to its neighbour. Eventually, the packet returns to its source with a topology map (list of addresses) of the ring. Routers are able to use the address resolution protocol (ARP) mechanism to determine which RPR MAC address belongs to the destination address of an IP packet. RPR switches and bridges will have a list of stations that they can reach through a RPR MAC address. The topology map will be used to determine which direction on the ring will provide the best path to the destination. Physical Layer RPR packets can be transported over both SONET and Ethernet physical layers. The SONET/SDH physical layer offers robust error and performance monitoring. RPR packets can be encapsulated within the synchronous payload envelope (SPE) using a high-level data-link control (HDLC) like or generic framing protocol (GFP) encapsulation. A robust Layer-1 protocol, SONET/SDH provides information such as loss of signal and signal degrade for use by the RPR protection mechanism. When using a SONET/SDH physical layer, RPR can be carried over SONET/SDH TDM transport or dark fiber. Ethernet provides an economical physical layer for RPR networks. RPR packets are transmitted with the required inter-packet gap (IPG). RPR Systems using the SONET physical layer will not interoperate with Ethernet physical-layer-based systems on the same ring. -6-
MAC Frame format Figure 4. RPR MAC frame. Destination Address: This one or six byte address (see dual mode addressing) is the MAC address of the ring node to which the frame is being transmitted, and therefore does not change from link to link on the ring. This address can also be a broadcast address Source Address: This one or six byte address is the MAC address of the ring node from which the frame is being transmitted, and therefore also does not change from link to link. Payload Type: This two-byte field tells the system what type of payload follows the RPR fields. For example MPEG, ATM or Ethernet. -7-
Class of Service (CoS): The three bit CoS field allows the identification of up to eight Classes of Services, including Expedited Forwarding (EF), six levels of Assured Forwarding (AF1 through AF6), and Best Effort (BE). Extension (E) Bit: This field indicates that there is an extension to the RPT header. This allows for fields that may be added in the future. Time To Live (TTL): The one byte TTL field is included to allow the RPT ring topology. It ensures that under no circumstances do RPT frames continue to circulate in a loop indefinitely. Flow ID (optional): The 20-bit flow ID field maps a virtual connection from ingress to egress over the RPT ring. It allows the simple manual or automatic setup of connection oriented services such as Time Division Multiplexed (TDM) circuit emulated services and Ethernet virtual leased lines through the packetswitched network. Header Error Check (HEC): Borrowing a concept from ATM, the two byte Header Error Check (checksum) provides a way to test the integrity of the header, allowing for persistent delivery of frames despite errors in the payload. Cyclic Redundancy Check (CRC): The four byte (32 bit) Cyclic Redundancy Check (CRC-32) works differently in RPT than it does for standard Ethernet. -8-
3. Comparing RPR to Other Solutions Resilient Packet Ring (RPR) technology was designed to combine SONET s carrier-class functionalities with Ethernet s high bandwidth utilization and granularity. Additionally, RPR technology offers fairness that has been lacking in today s Ethernet solutions. RPR has a MAC layer technology as discussed earlier, being standardized in the IEEE 802.17 workgroup. This employs spatial reuse to maximize bandwidth utilization, provides a distributed fairness algorithm, and ensures high-speed traffic protection similar to SONET Automatic Protection Switching (APS). RPR allows full ring bandwidth to be utilized under normal conditions and protects traffic in the case of a nodal failure or fiber cut using a priority scheme, alleviating the need for SONET-based protection. Furthermore, because the RPR MAC layer can run on top of a SONET PHY, RPR-based networks can provide performance-monitoring features similar to those of SONET. RPR technology offers all of these carrier-class functionalities, while at the same time keeping Ethernet s advantages of low equipment cost, high bandwidth granularity, and statistical multiplexing capability. The following table summarizes the differences between traditional Ethernet and RPR: Table 2. Comparison between Gigabit Ethernet and RPR GIGABIT ETHERNET Enterprise-class equipment Data service only Point-to-point or mesh topology (No rings) Protection in 50 seconds Simple management Limited scalability RPR Carrier-class equipment -9- Data, circuit or video service Point-to-point, linear, ring, or mesh topology Protection in 50 milliseconds or less Full FCAPS management 254 nodes per ring, multiple rings
The following table compares SONET/SDH to RPT: Table 3. Comparison between SONET/SDH and RPT SONET/SDHT RPT Manual topology configuration Auto-topology discovery 16 nodes per ring 254 nodes per ring Fixed, dedicated management Management bandwidth used as needed bandwidth Time division multiplexing Statistical multiplexing Manual provisioning of bandwidth Manual or dynamic provisioning and routes No service class awareness Differentiated services in eight classes Fixed direction traffic routing Least-cost traffic routing All traffic is protected Per-connection protection 50% of all bandwidth reserved unused for protection Bandwidth reserved as needed, but still usable by unprotected services Protection per bi-directional span Per-fiber protection -10-
4. Using RPR with SONET and Ethernet Ethernet is the most common interface within enterprise networks. Although the optimal customer interface and LAN solution, Ethernet does not work well within metro networks. Most RPR edge systems offer an Ethernet interface as the customer interface. SONET/SDH can be used to transport RPRs that use the SONET/SDH physical layer. A 2.5G RPR can be carried as two optical carrier (OC) 48/synchronous transfer mode (STM) 16 channels (inner and outer rings) inside an OC 192/STM 64 SONET/SDH ring. This allows customers to utilize their existing transport infrastructure to deliver legacy TDM services, while RPR delivers data services efficiently. New SONET advancements such as virtual concatenation and the link-capacity adjustment scheme (LCAS) allow more precise and dynamic allocation of bandwidth dedicated to RPR services. Figure 5. RPR over SONET with Ethernet Customer Interfaces -11-
5. Technical aspects of RPR Technical aspects of RPR are multicast, spatial reuse, fairness, fast protection and quality of service. Multicast One RPR multicast packet can be transmitted around the ring and can be received by multiple nodes. Mesh topologies require multicast packets to be replicated over all possible paths, wasting bandwidth. Spatial Reuse RPT has the ability to switch traffic over multiple spans of the ring simultaneously. Hence, the bandwidth on a particular span between ring nodes is utilized asynchronously with regard to the bandwidth on other spans. This allows bandwidth to be added or dropped on a span-by-span basis, which ensures maximum utilization of bandwidth in the ring, especially when traffic patterns are highly distributed. For example, traffic passing from node A to node B will be dropped at node B, allowing new traffic to be inserted at node B for transmission to downstream nodes C, D, etc. Figure 6. RPR Spatial Reuse -12-
Fairness Fairness is one of the most important features in carrier-class networks. Fairness is achieved when the traffic characteristics of two service flows that have the same service level agreement (SLA) are identical, regardless of their network source and destination. The RPR protocol can guarantee fairness across the metropolitan network. Each node on the metro ring executes an algorithm designed to ensure that each node on the ring will get its fair share of bandwidth. However, ring nodes use the spatial reuse capability to use additional available bandwidth, which is greater than their fair share on local ring segments, as long as it doesn t affect other ring nodes. More specifically, the ring supports weighted fairness, proportional to the bandwidth each user buys. Figure 7 demonstrates the operation of the fairness algorithm. Before employing the fairness algorithm, all nodes transmit at their peak usage rate. At some point Node 1 experiences congestion and requests upstream nodes to reduce their transmission rate to the allowed rate. After convergence, each node will receive its fair share of bandwidth, while nodes that can locally transmit higher rates, like nodes 2 and 3, without generating congestion on other nodes, continue to transmit their peak usage rate. Other data optimized technologies, such as Ethernet, do not provide the carrierclass fairness guaranteed in RPR-based networks. Ethernet switches prioritize traffic locally, on every interface on the ring, thus creating unfair conditions for traffic that has to traverse through several nodes on its way to the destination node. -13-
Figure 7: Fairness in an RPR Ring Figure 8 demonstrates the lack of fairness in Ethernet-based networks. In this example, a 10Gbps Ethernet ring connects between the nodes. Nodes 4, 5 and 6-14-
all transmit traffic to node 1, creating congestion between node 1 and node 6. Under these conditions, node 6 will get half of the total available bandwidth, while nodes 4 and 5 will each get 25% of the available bandwidth. -15-
Figure 8: Unfairness in an Ethernet Ring Fast Protection and Restoration The feature that more than any other makes RPR a carrier-class technology is its fast self-healing capability which allows the ring to automatically recover from a fiber cut or node failure. This is done by wrapping traffic onto the alternate fiber within the SONET-class 50msec restoration timeframe. RPR technology provides carriers with more than SONET-class fast protection and performance monitoring capabilities; it also enables them to achieve it without dedicating protection bandwidth, as in the case of SONET, thus eliminating the need to sacrifice 50% of the ring bandwidth for protection. Two mechanisms are proposed for fast protection in the RPR MAC - Wrap and Steer. Each of these mechanisms has its own advantages and limitations, and both can be used on an RPR ring utilizing the Selective Wrap Independent Steer (SWIS) scheme. -16-
Figure 9 shows an example of the data paths taken before and after a fiber cut event when Wrap is being used. Before the fiber cut, node 3 sends traffic to node 1 via node 2 (Figure 9a). When the fiber cut occurs (between node 1 and node 2), node 1 will wrap the inner ring to the outer and node 2 will wrap the outer to the inner ring. After the wrap, traffic from node 3 to node 1 initially traverses through the non-optimal path (passing through node 2, back to node 3 and into the originally assigned port on node 1 - Figure 9b). This is immediately done and can inherently meet the 50ms criteria, as it is a local decision of the nodes that identify the fault (in this case, nodes 1 and 2). Subsequently, higher layer protocols discover the new ring topology and a new optimal path is used (Figure 9c). This can take several seconds to converge in a robust manner, after which the double ring capacity is restored (even though not evenly distributed, depending on the location of the fault and the ring traffic pattern). Thus, unlike SONET rings that always pay the redundancy tax to achieve protection, after a failure; an RPR ring reduces its capacity only for several seconds. Other data optimized protocols, such as Ethernet, do not provide the SONETclass restoration time. Typically, spanning tree tools are used for protection in these cases. In a mesh topology, these tools provide protection time of the order of 1 second, but in ring topologies, protection time is degraded to the order of 10 seconds or more. Figure 9: Fast-Protection in RPR Using Wrap -17-
In summary, RPR s fast protection and restoration capability prevents service loss for high priority critical traffic. And just as SONET does, RPR enables carriers to continue to support their existing critical traffic over a data optimized network, without compromising their guaranteed 99.999% service availability. Quality of Service Quality of Service (QoS) is required in order to let a carrier effectively charge for the services it provides. ATM promised to deliver multiple services due to its rich QoS feature set. However, a carrier s service offering should be simple. Customers should clearly understand the service differentiators for which they re required to pay. Often, a too-rich QoS feature set causes a complicated and incomprehensible service offering. Furthermore, different service quality features are required for different types of applications. Data transfer applications require low packet loss rate, while real-time applications, such as voice, require low latency and low delay variation. There are several parameters that more than others govern the characteristics of a delivered service: Service availability, delay, delay variation and packet loss rate. Service availability depends on the reliability of the network equipment, as well as on the network survivability characteristics. Delay occurs in a network when a packet waits in a switch queue for other packets to pass. Delay variation is the difference in delay of several packets belonging to a common traffic flow. Highfrequency delay variation is called jitter while low-frequency delay variation is called wander. The most important parameter for a carrier selling bandwidth services is service availability. In order to remain competitive, a carrier must guarantee five-nines -18-
(99.999%) service availability, irrespective of other service characteristics. The respective requirement from the network is that all traffic flows should remain active under any circumstances, including during a protection event. During a protection event, and until high layer protocols converge, the available bandwidth on an OC-192 RPR ring reduces to 10Gbps. In order to guarantee service for every traffic flow under these circumstances, a portion of each flow s bandwidth should be allocated on this guaranteed throughput. This way, even during a protection event, service availability is guaranteed and none of the services is preempted. Traffic delay is a minor issue in high throughput rings. An end-to-end delay of several tens of milliseconds is usually regarded as acceptable, while in an OC- 192 ring, the expected delay for the longest Ethernet frames (1518 bytes) is shorter by three orders of magnitude (about 5(sec). Delay variation control is essential to ensure proper transport of TDM services over an RPR ring. Control can be achieved by synchronizing the nodes on the RPR ring by a common timing source. -19-
6. RPR Market Development ISP Networks Without barriers of traditional SONET/SDH infrastructures, RPR solutions are helping ISPs to deliver reliable Internet services (such and IP and video) and address the growing bandwidth service requirements for the next-generation intra-point of presence (POP), exchange point, and server farm/storage applications. Regional Metro Networks RPR enables regional metro networks designed to deliver Internet services (such as IP, VoIP, and video) to the metro. RPR regional metro solutions are available for transport over dark fiber, over wavelength division multiplexing (WDM), and over SONET/SDH, cable MSO, and enterprise/campus MANs. Metro Access Networks RPR enables metro access solutions for service providers looking to deliver Internet services (such as IP, VoIP, and VPN) to metro access networks with direct Ethernet connectivity for multi-tenant/multidwelling customers and edge programmability. -20-
7. Summary of features and benefits of RPR RPR provides the following benefits to service provider networks: Packet-optimized, Layer 1 independent protocol that allows the integration of DWDM, transport, switching and routing functions in a single platform. Differentiated data services, with advanced QoS mechanisms. Point-to-point and multipoint services. Topology flexibility, including the ability to have spans running at different data rates and with different numbers of fibers and wavelengths, as well as the ability to have asymmetry in the bandwidth. End-to-end networking through a standard, heterogeneous IP/MPLS network. Full interworking with standard IP, MPLS, Ethernet, and circuit-based networks. Maximum utilization of the fiber bandwidth under both normal and protection-switched conditions. Minimum disruption of user services during a protection-switching event. Specifically, full restoration of all protected service well within the 50-millisecond period specified by SONET and SDH standards. Maximum configurability of the switch as to its behavior under all conditions. Ease of provisioning and management of the ring. -21-
Significantly simplified network operations vs. ATM and SONET/SDH approaches. Faster deployment of services. 8. Conclusion The main objective of Resilient Packet Ring technology is to enable a true alternative to SONET transport for packet networks, providing carriers with resiliency, fast protection and restoration, and performance monitoring comparable to those of SONET networks. RPR was designed to combine SONET strengths of high availability, reliability and TDM services support, with Ethernet s low-cost, superior bandwidth utilization and high service granularity characteristics. Unlike SONET, RPR provides an Ethernet-like cost curve as well as superior bandwidth utilization, both in its Ethernet-like statistical multiplexing, and in its spatial reuse capabilities. Spatial reuse provides an extremely efficient use of shared media traffic in metro/access rings, since it is expected that in the near future most traffic originating in a metro area will remain within the same metro/access ring. Unlike next-generation SONET solutions that integrate both transport and data switching in the same network element, RPR is a transport technology that fits into existing carriers operations model, this tremendously reduces the required operational expenses in deployment as well as the maintenance expenses associated with the manual provisioning process of today s transport networks. -22-
Unlike Ethernet transport technology, RPR provides five-nines availability using SONET-grade fast protection and restoration, carrier-class fairness, the ability to transparently carry and groom TDM traffic, and SONET-like reliability and performance monitoring capabilities. RPRs provide a reliable, efficient, and service-aware transport for both enterprise and service-provider networks. Combining the best features of legacy SONET/SDH and Ethernet into one layer, RPR maximizes profitability while delivering carrier-class service. RPR will enable the convergence of voice, video, and data services transport. RPR is being promoted and standardized by industry leaders as well as by innovative startup companies, and is positioned to take a major role in deployment of next generation carrier-class networks. Standardization The IEEE 802.17 working group is working on the industry standard for an RPR MAC. This group was formed in January 2001 and expects to complete the standard in mid-2003. -23-
References 1. www.iec.org 2. www.ieee.org 3. Data Networks by Dimitri Bertsekas and Robert Gallagar. 4. Computer Networks by Andrew.S. and Taneabaum 5. Computer Network A systems approach by Larry.L.Petterson and Bruces. David -24-
CONTENTS 1. INTRODUCTION 2. RPR OPERATION RPR MAC MAC FRAME FORMAT 3. COMPARING RPR TO OTHER SOLUTIONS. 4. USING RPR WITH SONET AND ETHERNET 5. TECHNICAL ASPECTS OF RPR 6. RPR MARKET DEVELOPMENT 7. SUMMARY OF FEATURES AND BENEFITS OF RPR 8. CONCLUSION 9. REFERENCES -25-
ACKNOWLEDGEMENT -26-