Cross-Over Spanning Trees

Transcription

1 Cross-Over Spanning Trees Enhancing Metro Ethernet Resilience and Load Balancing Minh Huynh and Prasant Mohapatra Computer Science Department University of California at Davis Davis, CA. USA. {mahuynh, Stuart Goose Technology To Business Siemens Corporation Berkeley, CA. USA Abstract The economics and familiarity of Ethernet technology is motivating the vision of wide-scale adoption of Metro Ethernet Networks (MEN). Despite the progress made by the community on additional Ethernet standardization and commercialization of the first generation of MEN, the fundamental technology does not meet the expectations that carriers have traditionally held in terms of network resiliency and load management. These two important features of MEN have been addressed in this paper. We propose a new concept of Cross-Over Spanning Trees (COST) that increases the resiliency of the MEN while provisioning the support for load balancing. As a result, the capacity in terms of network throughput is greatly enhanced while almost avoiding any re-convergence time in case of failures. The gain ranges from 1.9% to 7.3% of the total traffic in the face of failure; while load balancing increases an additional 1.7% to 37% of the total throughput. Keywords- Cross-Over Spanning Trees, Load Balancing, Metro Ethernet Network, MSTP, Resilience, RSTP. I. INTRODUCTION The most common technology used for local area networks is the Ethernet protocol, which has been the predominant technology for a long period of time. Ethernet is a simple and cost-effective protocol that provides a variety of services. Despite the occasional challengers, such as fibre channel, infiniband architecture, the evolution of Ethernet has been continuing. The recent standardization of Gigabit Ethernet [1] protocol has propelled it for consideration in the scope of metropolitan area networks. Moreover, several companies are focusing their developments, products, and services for Metro Ethernet Networks (MEN). MENs [13] are comprised of a metro core network and several access networks. All the access networks connect to the core at one or two gateway Ethernet switches. The customers networks are connected to the access network, and the metro core helps in interconnecting the access networks. Packets hop through multiple switches in both access and metro core networks. Redundant links are used both in core as well as access networks. The main challenges in the context of MEN include resiliency, load balancing, and support for QoS []. Current Ethernet solutions deploy the Spanning Tree Protocol and its variants to manage the topology autonomously. However, they are inadequate in all of the three areas. In this work, we address the resiliency and load balancing aspects of MEN. We have introduced a novel approach, called Cross-Over Spanning Tree (COST) protocol, which allows switching between multiple spanning trees without forming any cycles. This feature enhances the resiliency as well as facilitates load balancing. In addition to fast recovery, it also increases the capacity of the network in terms of the achievable throughput. Considering the impact of the COST approach, we believe that COST has potential to provide enhanced services with a low overhead. The encouraging experimental results presented in this paper were obtained using the OPNET [1] simulation product to quantify the resiliency and the gain in terms of the network throughput. The behaviors of the Ethernet switches within OPNET Modeler were modified to imbue the COST approach. In the resilience test scenarios, COST yields an increase of 1.9% and 7.3% of the total traffic comparing to Multiple Spanning Tree Protocol (MSTP) and Rapid Spanning Tree Protocol (RSTP), respectively. In addition, when the network is overloaded and imbalanced, COST gains an additional 1.7% and 37% of the total traffic comparing to MSTP and RSTP, respectively. As a result, COST reduces ten times the delay in MSTP and RSTP. The organization of the paper is as follows: Section II describes the spanning tree protocol family that is currently deployed in Ethernet and their limitations. Section III explains the concept of COST including an overview, some details in the implementation, and an example. Section IV illustrates the simulation setup. Section V shows COST in operation in the face of failure comparing to RSTP and MSTP. Section VI demonstrates COST s load balancing capability against RSTP and MSTP. Related works are presented in section VIII, and the paper is concluded in Section IX. II. MOTIVATION Traditionally, Ethernet-based networks use spanning tree protocol (IEEE.1d) for routing packets in the network. Spanning Tree Protocol [1] is standardized in IEEE.1d. It is a layer protocol that can be implemented in switches and bridges. The spanning tree protocol (STP) essentially uses a shortest-path approach in forming a tree that is overlaid on top of the mesh-oriented Ethernet networks. Spanning tree is used primarily to avoid formation of cycles or loops in the network. Unlike IP packets, Ethernet frames do not have a time-to-live (TTL) field. STP prevents loops in the network by blocking

2 redundant links. Therefore, the load is concentrated on a single link which leaves it at risk of failures and with no load balancing mechanism. The root of the tree is chosen based on the bridge priority, and the path cost to the root is propagated throughout so that each switch can determine the state of its ports. Only the ports that are in the forwarding state can forward incoming frames. This ensures a shortest single path to the root. Whenever there is a change in the topology, switches rerun the protocol that can take 3 to seconds. At any one time, only one spanning tree dictates the network. Although STP has been used for most Ethernet networks, it has several shortcomings in the context of its use for MEN. These shortcomings are enumerated as follows: 1. Spanning trees restrict the number of ports being used. In high-capacity Ethernets, this restriction translates to a very low utilization of the network.. STP has poor resiliency: a very high convergence time (3s to s) after a link failure. 3. STP does not have any mechanisms to balance load across the network.. STP does not support QoS. An improvement of STP is the Rapid Spanning Tree Protocol RSTP [] specified in IEEE.1w. RSTP reduces the number of port states from five in STP to three: discarding, learning, and forwarding. Through faster aging time and rapid transition to forwarding state, RSTP is able to reduce the convergence time to between 1 and 3 seconds. It is understood that depending on the network topology, this value varies. In addition, the topology change notification is propagated throughout the network simultaneously, unlike STP, in which a switch first notifies the root, then the root broadcast the changes. Similar to STP, there is only one spanning tree over the whole network. RSTP still blocks redundant links to ensure loop free paths leaving the network underutilized, vulnerable to failures, and with no load balancing. MSTP or Multiple Spanning Tree Protocol [] is defined in IEEE.1s. MSTP uses a common spanning tree that connects all of the regions in the topology. The regions in MSTP are multiple instances of the spanning tree. Each instance is an instance of the RSTP. An instance of RSTP governs a region, where each region has its own regional root. The regional roots are in turn connected to the common root that belongs to the common spanning tree. Since MSTP runs pure RSTP as the underlying protocol, it inherits some drawbacks of RSTP as well. However, a failure in MSTP can be isolated into a separate region leaving the traffic flows in other regions untouched. In addition, the administrators can perform light load balancing manually by assigning certain flows to a specific spanning tree. COST goes a step further in resilience and load balancing provisioning. COST eliminates the performance hit whenever there is a failure and gives more flexibility for load balancing. III. CONCEPTUAL APPROACH TO COST In this section, we describe the basic philosophy behind the COST protocol and its potential for provisioning enhanced performance and services. A. COST Philosophy In the spanning tree protocol and in most of its variants, at any point of time, only one spanning tree is used. The use of this spanning tree is facilitated by blocked ports in various combinations in each of the Ethernet switches. Although many protocols have proposed the enhancement of the basic STP, they still use only a single spanning tree for one flow at any point of time in any segment of the network. These protocols take relatively longer to recover from faults and also have no support for balancing load across the network. The primary motivation behind the design of COST is to allow the flexibility of using more than one spanning trees while a flow is en route to its destination. This flexibility allows the usage of more ports per switches. However, to avoid the formation of cycles in the network, certain restrictions are imposed. The basic methodology for implementing the COST philosophy is to identify multiple spanning trees and number them sequentially to form an ordered list. The VLAN ids [3] can be used as the sequences for the spanning trees. Frames of a flow start using one spanning tree and can be switched over to the next spanning tree (none of the other variants of spanning tree allow this flexibility) in sequence. This procedure can be repeated until the frame reaches the spanning tree which has the highest id in the sequence, as seen in Figure 1. At no point in time is a frame allowed to change from a spanning tree with a higher id to a spanning tree with a lower id. A flow is switched, or crossed-over, from one spanning tree to another whenever there is a: link failure, or load imbalance X Figure 1 A sketch of the 3 layers of spanning tree. Prior to a failed link (indicated by a cross), COST elevates the traffic (indicated by dashed lines) to the next highest spanning tree in the sequence. Traffic traverse from right to left B. Loop Free Guarantee As mentioned earlier, a cross-over from one spanning tree to another is allowed only from a lower numbered Spanning Tree (ST) to a higher one; the reverse cross-over is not permitted. Therefore, infinite loops cannot occur in COST because of this monotonic increase property. In other words, X

3 COST does not lead frames into an infinite loop when switching between spanning trees. Initially, COST uses the first ST. If there is a problem with a link on the path of the first ST, COST switches to the second ST. If there is no problem, COST stays on the first ST. COST forms multiple spanning trees by creating a set of single independent spanning trees by running RSTP for each ST. Each of these STs guarantees loop free because the RSTP blocks all the redundant links. Therefore, if COST stays on the first ST and never switches, then no loop would occur, guaranteed by RSTP. When a problem occurs on the path on the first ST, COST switches to the second ST. Since COST prohibits the switching from a high ranking tree to a low ranking tree, a flow cannot switch to the first ST. Therefore, it stays on this second ST until it gets dropped or delivered. It is possible that frames might get into a local loop because of link sharing between STs. In other words, the frames might repeat the link they have traversed before. However, they will not get stuck in the loop infinitely because they will not repeat any path supervised by the first ST. Then the loop free property is guaranteed because the second ST is independent from the first ST and it is also guaranteed by the RSTP loop free property individually. Inductively, loop free property is guaranteed as COST climbs up the ladder of STs. C. Implementation Issues When implementing COST, there are other issues that must be taken into consideration. First, COST must be backward compatible with current protocol. To do this, we leverage the MSTP protocol,.1s, to implement the functionality and operations needed by COST. Since MSTP is backward compatible with RSTP and STP, COST can interoperate with them and MSTP as well. Thus, COST retains the advantages of MSTP while providing enhanced features in terms of resiliency, capacity, and load balancing. This also saves the cost of upgrading current switches to switches that support COST by a gradual upgrade procedure. The decision of whether to elevate to the next Spanning Tree is done on a per frame basis. As each frame arrives at a switch, it looks at the outgoing link of the current frame in order to decide. Thus, the failure and load imbalance is detected locally by each switch staying true to the nature of Ethernet. Therefore, if a link is problematic, only a part of the end-to-end path need rewrite the header for certain affected flows. Then it is possible for a flow to be in multiple STs at one time. For example, if a flow takes the following path A-B-C-E starting with ST 1. A problem occurs at link B-C. Without any user intervention, the flow is adjusted so that a new path is chosen, A-B-D-E. From A-B, the flow still uses the same ST 1, but from B through E, the flow runs on ST. The key in COST enhanced performance is the avoidance of the lengthy re-convergence procedure. Therefore, the reconvergence in MSTP must be changed as follows. When a switch detects a fault or a link recovery on one of its ports, the Spanning Tree Algorithm (STA) cannot initiate the port state/ role re-selection. The STA must be also stopped from flushing entries in the filtering database and its forwarding tables. The switch acts as if nothing has happened and switches the traffic to the new Spanning Tree for a soft re-convergence. When a link recovers, instead of setting the recovered port to blocking state and performing the re-convergence, the switch only needs to reinstate the original role of the port per Spanning Tree. The original intension of having VLAN tags is for isolation of traffic. When COST uses VLAN tags as id for ST, VLAN s initial objective is still preserved. Instead of mapping a VLAN id to a traffic group, COST can be implemented to map a set of VLAN ids that represent the spanning trees to a traffic group. The VLAN partition is implementation dependence. The shortage of VLAN ids can be an issue but there are proposals to perform VLAN stacking or Q-in-Q [1][1]. This technique increases the number of VLAN tag from 1 to. Viking [], a related work, also uses VLAN tag as the identification for multiple spanning trees. After a long period in operation, it is possible that a majority of the traffic runs on the last available Spanning Tree. This impairs COST greatly because COST is not allowed to switch to a lower order Spanning Tree. Thus, COST is helpless to the majority of the traffic when they run into problems. This is an indication for the administrator to schedule a total reconvergence. D. Exemplar Simulation Using a Simple Topology For the initial explanation, a very simple topology is used to illustrate of the benefits of COST. Our primary goal is to show the feasibility in terms of COST operation and to quantify its potential benefits in terms of resiliency and load balancing. For this initial demonstration experiment, consider the simple topology shown in Figure. There are three sources sending traffic to one destination. The frames are routed through three Ethernet switches. Multiple links are available between the switches. Several spanning trees can be formed between the sources and the destination. The spanning trees use VLAN ids for sequencing. In Figure, from top to bottom, each link is on a separate VLAN. The VLAN ids are 1,, 3 and. In this experiment, a total of about 1.9 Mbps of constant traffic is sent by sources src, src_, and src_1 to the destination dest. The traffic starts to flow at 1s to allow the Spanning Tree initiation to complete. The maximum load on each link is set to % and the link capacity is 1Mbps. Thus, at most a link will carry Mbps. The experiment demonstrates what happens when the link (vlan1) between switch1 and switch fails at 1s and then recovers at 1s. src src_ src_1 switch1 Figure VLAN1 VLAN VLAN3 switch VLAN VLAN1 VLAN VLAN3 VLAN switch3 A simple exemplar topology dest Initially, the vlan1 and vlan links carry * Mbps; the remaining.9 Mbps is on vlan3. No traffic flows initially on vlan. From Figure 3, when the switch1< >switch (vlan1) link fails at 1s, the throughput on the switch1< >switch

4 (vlan3) link increases to carry the traffic from the failed link. The switch1< >switch (vlan) link increases to carry the remaining traffic. When the switch1< >switch (vlan1) link recovers, the state of the link returns to the original. switch1 <-> switch (vlan1) switch1 <-> switch3(vlan3) switch1 <-> switch(vlan) switch <-> switch(vlan) Figure 3 Throughput on each link between switch1 and switch. A simple illustration of COST resilience, link recovery/reuse, and load balancing to a threshold. The traffic was monitored between switch and switch3, as seen in Figure. Since there is no failure on the downstream direction in this part of the network, the graph shows no switching of spanning trees when the link failed. Hence, effects of the failed part of the network are isolated from the others. The graph in Figure shows a consistent cumulative throughput of 1.9 Mbps received at the destination. switch <-> switch3 (vlan1) switch <-> switch3(vlan3) switch <-> switch3(vlan) switch <-> switch3(vlan) Figure Throughput on each link between switch and switch3. The graph shows that the effect of a link failure between switch1 and switch were not felt here because COST manages the resilience locally Figure time (s) The cumulative throughput as observed by the receiving host during the link failures. IV. SIMULATION DESIGN The OPNET [1] simulator tool was chosen because of its comprehensive implementation of Ethernet. OPNET includes implementation for RSTP, MSTP, and VLAN which are crucial to the evaluation of COST. COST will be evaluated on two topologies: a representative topology of Metro Area Network (MAN), and a x grid topology as seen in Figure and Figure 7, respectively. The topology from Figure is widely recognized in Metro Area Network (MAN) and is similar to the metro network topology from Riverstone Networks [17]. A grid topology which inheritably contains high degree nodes is put to the test to show the impact of COST on various topologies. As long as there are multiple backup paths, the network performance can enjoy the benefit of COST. In the MAN topology, RSTP has only one spanning tree configured on each side of the router. The initial RSTP spanning tree is shown in Figure 3 (Appendix). The root of the spanning tree of the network in question is located at the switch core. On the other hand, MSTP and COST have four spanning trees configured. The common root is at core but the regional root for vlan1 (ST 1) and vlan (ST ) is at core1, the regional root for vlan (ST ) and vlan3 (ST 3) is at core. The spanning tree configuration can be viewed in Figure 3 through Figure (Appendix). Each VLAN represents one spanning tree. Similar to RSTP, the spanning tree stops at the router. Likewise, RSTP has one spanning tree operating in the x grid topology. The root of the tree is located at node_1 which is center of the topology. Conversely, MSTP and COST is configured with six spanning trees. The common root is at node_33. The regional roots for ST 1 through ST are in the following order: node_33, node_9, node_3, node_1, node_1, node_7. As specified in the.1d standard, there are a maximum of 7 hops. In order to form a stable spanning tree for a topology of this size, the hop count parameter is increased.

5 Figure access1 CSsrc1 core aggregator1 access DBsrc1 DBserver server_switch core core1 access3 DBsrc router core access CSsrc core aggregator access DBsrc3 core3 access CSsrc3 A representative Metro Area Network (MAN) topology node_3 node_ node_ node_1 node_1 node_31 node_7 node_1 node_13 node_19 server1 node_3 node_ node_ node_1 node_ server node_33 node_9 node_3 node_1 node_1 server3 node_3 node_1 node_ node_1 node_ node_3 node_11 node_ node_17 node_3 1s: aggregator1 < > core1 fails 1s: aggregator1 < > core fails 1s: aggregator < > core1 fails s: aggregator1 < > core1 recovers Results of the simulation experiment for RSTP, MSTP and COST are presented in the next section. Cumulative throughput is used as the metric for comparing the resilience. For MSTP, each traffic source is assigned to a spanning tree in a round robin fashion from left to right in Figure. For example, CSsrc1, DBsrc1, DBsrc, CSsrc, DBsrc3, and CSsrc3 are assigned to vlan1, vlan, vlan3, vlan, vlan1, and vlan, respectively. Vlan1, vlan, vlan3, and vlan each correspond to a different spanning tree. COST has the same traffic profiles as MSTP except that all sources initially begin with vlan1 (ST 1). ) Load Imbalanced Scenarios The topology in this load balancing scenario is similar to that which was used for the resilience scenario (seen in Figure ). However, we add two additional sources, CSsrc and DBsrc, to access3 switch and access switch, respectively. There are eight traffic flows from CSsrc{1,,3,} and DBsrc{1,, 3,} to DBserver. Each flow is a bi-directional UDP stream of approximately.3mbps. The simulation runs for s. Again, all traffic starts to flow at 1s. There is no failure within this experiment. All of the links below core1 and core are 1Mbps. All the links above core 1 and core are 1Gbps. The reason that we choose 1Mbps was for simulation efficiency. It is faster to overload the 1Mbps link, and there is still enough memory to run the simulation. The load is spread out evenly for MSTP, i.e. each spanning tree has two sources. CSsrc1 and DBsrc3 traverse on the first ST; DBsrc1 and CSsrc3 traverse on the second ST; DBsrc and CSsrc traverse on the third ST; and CSsrc and DBsrc traverse on the fourth ST. B1 S1 node_ G1 node_ B Figure 7 S node_ G node_7 B3 S3 node_ G3 A x grid topology node_9 A. Metro Area Network Topology Within this topology, resilience and load balance will be evaluated. Although they are being evaluated separately, they can work well together as demonstrated in the simple example in Section III.D. The description and specific parameters used are included. 1) Failure Scenarios There are six traffic flows from CSsrc{1,, 3} and DBsrc{1,, 3} to DBserver. Each flow is a bi-directional UDP stream of approximately.3mbps. All links have capacity of 1Gbps. The simulation runs for s. Again, all traffic starts to flow at 1s allowing the standard Spanning Tree initialization to complete. The link failures and link recovery are scheduled as follows: B S G B. Grid Topology Similar to the MAN topology, resilience and load balance are put to the test in separate scenarios. The resilience test in this topology is more rigorous in that it includes both node failures and link failures. Whenever there is a node failure, all the links attached to the node also fail as well. 1) Failures Scenarios There are four flows to each of the server in Figure 7. Each flow is a bi-directional UDP stream of approximately.3mbps. All links have capacity of 1Mbps. This capacity is enough to carry the traffic without causing any congestion. The simulation runs for 1s. Again, all traffic starts to flow at 1s. There are a total of failed links and six failed nodes. The link failures and node failures are scheduled as follows: 11s: node_7 fails 11s: node_ < > node_9 fails 1s: node_1 fails 13s: node_13 fails 1s: node_1 fails 1s: node_7 < > node_ fails

6 1s: node_1 fails 1: node_ < > node_1 fails 1s: node_3 fails 1s: node_3 < > node_33 fails In MSTP scenario, the flows are group by the destination to put into the corresponding tree. For example, since S1, S, S3, and S are going to server1, they run on the same spanning tree. However, all sources started in the same tree in COST. ) Load Imbalanced Scenarios The setting in this load balancing experiment is similar to that which was used for the resilience experiment (seen in Figure 7). However, the links are now 1Mbps. The reason is because of the simulation efficiency and resources as before. Since the bottleneck for the RSTP scenario is the links on the path from node_33 to node_1, we upgrade these links to 1Mbps for all the three protocols. Each flow is a bidirectional UDP stream of approximately.3mbps. The simulation ran for 17s. Again, all traffic starts to flow at 1s. There are no link failures within this experiment. V. ENHANCED ETHERNET RESILIENCE As alluded to earlier, resilience is of particular importance for carriers and this is one area for which Ethernet is well recognized as being very weak. COST was specifically formulated to address the inherent weakness of Ethernet resilience. An experiment is presented in this section in which RSTP, MSTP and COST are evaluated for their resilience in the face of link failures and recoveries. The results of the MAN topology from Figure are presented first and in details to show the behavior of COST. Then the results from the grid topology are overviewed to demonstrate the impact of COST on a denser topology. A. Performance in Metro Area Network Topology This subsection shows the performance of each protocol in the face of failures separately. Then a superimposed graph shows how they are stacked up against each other. 1) RSTP The throughput for RSTP as observed by the receiving host during the link failures is depicted in Figure. As expected, when a link fails, RSTP re-converges and the new link takes over showing by a dip in the throughput. Figure shows the effect of failure in the network at different times. The first dip accounts for the failure at 1s. The opened gap takes 1s for RSTP to re-converge. Following the re-convergence, the link aggregator1< >core is not used, which explains why no dip is seen for the link failure at 1s. The second dip accounts for the link failure at 1s; while the third dip is the result of the link recovery at s. Similarly, the second re-convergence takes 1s and the third re-convergence takes 7s Figure time (s) the throughput as observed by the receiving host during the link failures for RSTP ) MSTP Figure 9 illustrates the impact of failures in the MSTP network. The first dip accounts for the failure at 1s. The second dip occurs at 1 on account of aggregator1 < > core failure. Unlike RSTP, MSTP utilizes more links; therefore, it receives a performance hit on the second failure. However, the hits in RSTP are much more damaging than in MSTP. It is shown in Figure 9 that the dip is not as deep as in Figure. The third and last dip are the result of aggregator < > core1 failure and the recovery of aggregator1 < > core1, respectively. On average, each gap in Figure 9 takes 7 seconds to pick up the rate again Figure time (s) The throughput as observed by the receiving host during the link failures for MSTP 3) COST The intermediate throughput on the links between the core switches and the aggregator switches can be seen in Figure 1. As prescribed by the monotonically increasing property of COST, the traffic is initially sent on vlan1 (aggregator1< >core1 link). Following a failure, the link for vlan (core< >aggregator1 link) takes over and when that fails, the link for vlan3 (aggregator1< >aggregator link) takes over. When aggregator1< >core1 recovers, the recently arrived frames do not have to crossover so that the rate of aggregator1< >core1 picks up again showing the dynamic nature of COST.

7 aggregator1 <-> core1 core <-> aggregator1 aggregator1 <-> aggregator 1 1 RSTP MSTP COST Figure 1 The throughput as observed on various links in the topology as links fail and COST re-routes traffic resiliently to maintain constant throughput ) Comparison of RSTP, MSTP and COST For this scenario, the comparative performance of RSTP, MSTP, and COST can be visualized by superimposing the cumulative throughput graphs of RSTP, MSTP, and COST as in Figure 11. While the maximum throughput is the same in this experiment, it can be seen that during the failure periods COST is able to maintain a sustained throughput. Despite incurring one or more link failures, COST has been designed in such a way that it is usually possible to avoid the need for the spanning tree re-convergence algorithm to be executed. If one or more failed links recover before COST exhausts the available VLANs (monotonic increase), then it is possible that no re-convergence is ever required even though links fail. Therefore, COST is able to operate continuously without interruption to the service. This is a clear advantage of COST over RSTP and MSTP. To observe the impact of the dip in the throughput graph, the loss percentage, which is the area of the dipped region, of RSTP and MSTP is calculated against COST. The results show that RSTP loses 7.3% of the total received traffic compared to COST; and MSTP loses 1.9% of the total received traffic compared to COST. These losses are significant in the MEN setting where the link capacity reaches the gigabits range. In addition, COST provides uninterrupted operations for real time services. Figure 11 In contrast with RSTP and MSTP, COST maintained a constant throughput to the receiving host despite link failures and recoveries B. Performance in a Grid Network In this scenario, Figure 1 shows the cumulative throughput collected at the three servers. Both RSTP and MSTP start to take the performance hit after 13s; whereas COST keeps constant throughput until 1s. In contrast to RSTP and MSTP, COST does not re-converge; therefore, after numerous failures without any recovery, some traffic flows start to run on the last spanning tree. This is shown in the figure where COST s graph line does not come back up to the maximum rate. Up until 1s, RSTP loses 7.9% of the total traffic comparing to COST, and MSTP loses 1.9% of the total traffic comparing to COST. After 1s, majority of COST s traffics run on the last spanning tree. Therefore COST s throughput drops to 3.1MB/s. As mentioned before, COST s weak point is when a majority of the traffic uses the last spanning tree. This is the threshold where a scheduled re-convergence for COST should take place. However, if there are some recoveries, then re-convergence might not be necessary. During the period where COST has some flows on the last spanning tree to most flows on the last spanning tree, COST still outperforms RSTP by 9% but loses to MSTP by %.

8 Throughput (MB/s) 7 3 COST RSTP MSTP its maximum capacity. Despite approximately. Mbps arriving at aggregator1, the switch s output was only 1 Mbps. RSTP cannot use the other two links to carry the remaining traffic as they are blocked forcing aggregator1 to drop the remaining traffic. At aggregator in Figure 1,.Mbps arrived and since the core1< >aggregator link capacity can handle it; it doesn t have to drop any packet. However, if an overload situation occurs, excess traffic will be dropped because the aggregator< >core is blocked time (s) Figure 1 The cumulative throughput of three protocols shows RSTP and MSTP with their reconvergences while COST starts having some flows running on the last spanning tree aggregator1 <-> core1 core <-> aggregator1 aggregator1 <-> aggregator VI. ETHERNET SWITCH LOAD BALANCING By being able to distribute the traffic, or to load balance, across various links in a network, it is possible to increase the capacity and utility of the network. However, none of the existing Ethernet protocols allow the carrier to dictate load balancing across all the links in the network. The COST algorithm will facilitate load balancing across all links in the metro Ethernet. The carriers will thus have an option for balancing load across the network, as well as to control the load on individual links. This will be an attractive feature for the carriers as they can exploit maximal throughput, and thereby capacity from the network. Similar to the resilience simulation, the results of topology from Figure are presented first and in details showing the behavior of COST. Then the results from the grid topology will be shown from the overall perspective. A. Performance in Metro Area Network The load balancing performance for MAN topology is examined in this section starting with RSTP. In order to see the inefficiency in RSTP and MSTP, the utilization of the intermediate links between the core switches and the aggregator switches are shown. The left side of the network refers to the links connected to aggregator1; and the right side of the network refers to the links connected to aggregator. 1) RSTP In this section, the traffic on intermediate links in the network was measured to show the inefficient link utilization of RSTP due to its lack of load balancing. The metric that we use in the graphs for demonstrating the efficacy of the respective protocols is again the cumulative throughput. At aggregator1 switch, there are three links that can potentially carry the traffic into the core network. However, in order to prevent loops, RSTP blocks two of those links. As shown in Figure 13, the aggregator1< >core1 link is loaded to Figure 13 The throughput as observed on various links in the left side of the access network topology shows under utilized links 1 core1 <-> aggregator aggregator <-> core Figure 1 The throughput as observed on various links in the right side of the access network topology also shows under utilized links ) MSTP At aggregator1,.3 Mbps arrives for ST 1, another.3mbps comes from ST, and.mbps is for ST 3. Since ST 3 root is at core and the link aggregator1< >core blocks ST 3, the.mbps must then go to aggregator. At aggregator, there are.3 Mbps for ST 1,.3 Mbps for ST,.Mbps for ST 3 that comes from aggregator1, and. Mbps for ST. Because ST and ST 3 share the same link, the link aggregator < > core is maxed out and is forced to drop some traffic. The link aggregator < > core1 only carries ST 1 traffic which is.3mbps. This is shown in Figure 1. The traffic for ST is sent over to aggregator1 because the root for

9 ST is at core1 and only link aggregator1 < > aggregator allows traffic for ST. The link aggregator1 < > core1 carries the combine traffic of ST 1 and ST (arriving from aggregator); thus it is maxed out at 1Mbps. The link aggregator1 < > core only carries traffic for ST; thus, it carries only.3mbps. And the link aggregator1 < > aggregator sends ST 3 traffic to core as explained earlier; thus, it has.mbps. This is shown in Figure 1. This shows that it is possible to perform load balancing in MSTP but it is not efficient. Even if the access ports are reconfigured to distribute the load balancing the network, it only applies to this specific situation. Due to the dynamic and unpredictable nature of packet switching traffic, there is no one static configuration that works for all. Therefore, MSTP is not as responsive to current traffic condition as COST. 1 1 core <-> aggregator1 aggregator1 <-> core1 aggregator1 <-> aggregator Figure 1 The throughput as observed on various links on the left side of the access network topology the traffic to the next ST unless it is the last ST. In this case, the last ST is ST. Initially, all traffic starts in ST 1 with each source sending.3mbps. There are.mbps arriving at aggregator1 on ST 1. The other 1. Mbps (.Mbps from access3 and another.mbps from access) is sent to aggregator on ST because the load balance threshold for the link is %. The 1.Mbps is now on ST. In addition, there are.mbps arriving at aggregator on ST 1. Of the.mbps arrived at aggregator1, Mbps is sent to aggregator1 < > core1 link on ST 1, Mbps is sent to aggregator1 < > core link on ST, and Mbps is sent on the agrregator1 < > aggregator on ST 3 toward aggregator. The remaining.mbps cross-over to ST and is sent out on link aggregator1 < > core1. Since the aggregator1 < > core1 link is shared by the last ST, it is allowed to have more than the % threshold. On the right side, aggregator sends Mbps to aggregator < > core1 link on ST 1 and upgrades the remaining.mbps to ST. Now, aggregator sends the combined.mbps + 1.Mbps on ST to aggregator < > core link. The arriving Mbps to aggregator from aggregator1 on ST 3 needs to be sent out on link aggregator < > core because that link is shared by ST and ST 3 to reach the root at core. However, the aggregator < > core link is capped at Mbps; thus, 1.Mbps traffic must be elevated to ST and sent back toward the root of ST at core1. Then the link aggregator1 < > core1 receives an additional of traffic for a total of = 1.Mbps. Since the link can only handle 1Mbps, it has to drop some packets. This is shown in Figure 17 and Figure aggregator <-> core core1 <-> aggregator Throughtput (Mbps) aggregator1 <-> core1 core <-> aggregator1 aggregator1 <-> aggregator Figure 17 The throughput as observed on various links on the left side of the topology showing COST improving and balancing link utilization Figure 1 The throughput as observed on various links on the right side of the access network topology 3) COST As with the RSTP experiment, the traffic on intermediate links in the network was measured to show the resulting link utilization. In this instance of the experiment, the link utilization threshold for load balancing was set at %. This means that for a link capacity of 1 Mbps the switch will permit at most Mbps on that link before it will try to switch

10 1 3 3 aggregator <-> core core1 <-> aggregator 1 1 RSTP MSTP COST Figure 1 The throughput as observed on various links on the right side of the topology showing COST improving and balancing link utilization Figure 19 The cumulative throughput of RSTP, MSTP and COST for MAN topology ) Comparison of RSTP, MSTP, and COST For this experiment, the comparative performance of RSTP, MSTP, and COST can be visualized by superimposing the cumulative throughput graphs of RSTP, MSTP, and COST as in Figure 19. As the bottleneck for RSTP is at aggregator1 where only 1Mbps can be sent out, by combining the traffic from aggregator the end host receives only approximately 17 to 1Mbps. MSTP does a better job to redistribute the load. However, it is difficult to find an optimum assignment of the loads for a balanced network. Heuristic assignments provide some balancing but might not be enough. In contrast, COST can dynamically redistribute the traffic if the current link is congested without any static pre-configuration; thus, it is able to accommodate more incoming traffic. Figure 19 shows the overlaid throughput of three protocols at the receiver. Comparing to a perfect condition where there is no loss from the senders, RSTP loses.1%; MSTP loses 1.9%, and COST loses 9.1% of the total traffic. A direct consequence of a congested network is the increase in delay as seen in Figure. This is the packet delay measured at the destination by taking a sample mean within an interval. RSTP and MSTP have a significantly higher delay than COST with RSTP being the highest. The rate at which RSTP increases its delay is.33 second per second; MSTP increases its delay at. second per second; and COST s delay increases at.1 second per second. Note that the very high delay cases are not usable in any practical sense. However, the relative comparison in this graph shows the relative merits of COST. delay (s) RSTP MSTP COST Figure The increasing delays for RSTP, MSTP and COST B. Performance in the Grid Topology In this grid topology, there is heavy congestion that forces the switches to drop some traffic flows. Therefore, none of the protocols can achieve the maximum throughput which is at.mb/s. Figure 1 shows that COST is able to achieve higher throughput than both RSTP and MSTP. COST delivers.% and 9.% more of the total traffic than RSTP and MSTP, respectively. It is interesting to note that in this case, the performance of RSTP and MSTP is relatively the same. Even with multiple spanning trees, MSTP still drops the same amount of traffic as RSTP. This is due to inefficient construction of the spanning tree. There are works that address this inefficiency in spanning tree construction in [][][9][1]. Therefore, it is possible to achieve higher throughput if the spanning tree structure is optimized.

11 Throughput (MB/s) 3 1 COST RSTP MSTP time (s) Figure 1 The cumulative throughput of RSTP, MSTP, and COST for grid topology VII. OUT OF ORDER PROBLEM Since COST allows a flow to be in multiple Spanning Tree at the same time, different frames belonging to the same flow might take different paths. Therefore, it is possible for end hosts to receive out of order frames. If out of order frames force TCP retransmission, it diminishes TCP performance greatly. We have run a set of simulation to test the effect of out of order packet on RSTP, MSTP and COST. Due to limited space, we only show the results for the K receiver buffer scenario. In this scenario, there are sources uploading to a server. Each source uploads a single file of size MB. The topology is the one from Figure. Figure shows that COST finishes sending the file before both of RSTP and MSTP. This shows that COST s out-of-order-packet does not make an impact on the performance of TCP. It is interesting to note here that RSTP performs better than MSTP. If MSTP distribute all loads onto one ST, it behaves just like RSTP. However, when the load is spread equally among the ST, its throughput gets worse. Throughput (MB/s) COST MSTP RSTP Figure TCP throughput for RSTP, MSTP, and COST VIII. RELATED WORKS Viking is a Multiple Spanning Tree architecture proposed by Sharma et al []. Viking precomputes multiple spanning trees so that it can change to a backup ST in the event of a failure. The paths are computed based on the weight that is assigned to each link. The weight is derived from the criticality of the corresponding link. Viking's approach is more complex than COST, in that the k shortest primary paths and the k backup paths for each primary must be determined. A path aggregation algorithm is then run to merge the paths into the spanning tree, which necessitates processing power. Viking also proposes to use a client-server model, opposed to COST which is a distributed solution on each switch. Viking needs to be informed by the end-hosts to update the server on the condition of network before the spanning trees are periodically recomputed. Therefore, its load balancing is not as dynamic as COST's real time load balancing. Ethereal [7], a real time connection oriented architecture supporting best effort and assured service traffic at the link layer, proposes to use the propagation order spanning tree for faster re-convergence of the ST once a failure has been detected. Unlike COST, Ethereal under utilizes the link capacity by blocking redundant ports and restrict a flow to only one Spanning Tree. SmartBridge [9] and STAR [1] are also two other approaches to improve upon the STP. They both find an alternate route that is shorter than the corresponding path on the Spanning Tree. SmartBridge requires the full knowledge of the topology. STAR is an overlay approach where STARaware switches are the super nodes of the topology. STAR calculates the shortest path from a super node to the next using the distance vector. Lim et al. [11] address the underutilization of the standard Spanning Tree. They also recognize that the simple priority queuing of.1 potentially starves low priority traffic when the high priority traffic dominate a significant fraction of the traffic. Each multimedia traffic flow uses the Spanning Tree that is built for the tuple <traffic type, VLAN>. While nonmultimedia traffic flows use the Spanning Tree that is built for a traffic type. In contrast to COST, each flow stays in the designated ST and no crossing over is allowed. Another approach to load balancing is Tree-Based Turn- Prohibition (TBTP) []. TBTP constructs a less restrictive spanning tree by blocking a small number of pairs of links around nodes, called turn, so that all cycles in a network can be broken. The benefit of TBTP is proportional to the degree of the nodes and MEN access networks have a low node degree. However, TBTP did not improve on the recovery time of the standard spanning tree protocol. Since TBTP relies on the standard STP to re-converge before it can re-compute its routing, the recovery time is in the magnitude of seconds. IX. CONCLUSION In this paper, we proposed a new concept of cross-over spanning trees (COST) for routing packets in the MEN. We have presented results from a preliminary study that demonstrates the potential benefits that can be offered to the carriers by COST. The implementation of COST has a low complexity overhead and can leverage MSTP support already

12 commonly available in Ethernet chipsets. In this work we have focused primarily on the resiliency and load balancing aspects of COST. Results obtained through simulation experiments using OPNET revealed the following potentials of COST: COST is very resilient to failures. It requires no reconvergence in face of multiple link failures. The throughput provided by COST is much higher than that of RSTP and MSTP. It helps in seamless re-integration and usage when failed links recover, thereby it may never need to reconverge. The process thus increases the MTBF of switches. COST provides load balancing of Ethernet frames throughout the metro access network, which is a new Ethernet layer feature. This gives carriers control over Ethernet that they previously never had. REFERENCES [1] IEEE Information technology - telecommunications and information exchange between systems - local and metropolitan area networks - common specifications. Part 3: Media Access Control (MAC) bridges, ISO/IEC 1-3, ANSI/IEEE Std.1D, 199. [] IEEE Standard for Local and Metropolitan Area Networks Common specifications Part 3: Media Access Control (MAC) Bridges Amendment : Rapid Reconfiguration Amendment to IEEE Std.1D, 199 Edition. IEEE Std.1w-1 [3] IEEE Standard for Local and Metropolitan Area Networks: Virtual Bridged Local Area Networks. IEEE Std.1Q-199 [] IEEE Standards for Local and metropolitan area networks Virtual Bridged Local Area Networks Amendment 3: Multiple Spanning Trees Amendment to IEEE Std.1Q, 199 Edition. IEEE Std.1s- [] Personal Communications, SBC Communications. [] S. Sharma, K. Gopalan, S. Nanda, T. Chiueh Viking: A Multi- Spanning-Tree Ethernet Architecture for Metropolitan Area and Cluster Networks Proceedings of IEEE INFOCOM. [7] S. Varadarajan, T. Chiueh Automatic Fault Detection and Recovery in Real Time Switched Ethernet Networks Proceedings of IEEE INFOCOM [] F. De Pellegrini, D. Starobinski, M. G. Karpovsky, and L. B. Levitin. Scalable Cycle-Breaking Algorithms for Gigabit Ethernet Backbones Proceedings IEEE INFOCOM [9] T. L. Rodeheffer, C. A. Thekkath, D. C. Anderson. SmartBridge: A Scalable Bridge Architecture Proceedings ACM SIGCOMM [1] K. Lui, W. C. Lee, K. Nahrstedt. STAR: A Transparent Spanning Tree Bridge Protocol with Alternate Routing ACM SIGCOMM Computer Communications Review Volume 3, Number 3: July. [11] Y. Lim, H. Yu, S. Das, S. S. Lee, M. Gerla QoS-aware Multiple Spanning Tree Mechanism over a Bridged LAN Environment Proceedings IEEE GLOBECOM 3 [1] OPNET simulator, [13] MEF, Metro Ethernet Networks A Technical Overview [1] IEEE Std.3z-199, Gigabit Ethernet, [1] Nortel Networks Service Delivery Technologies for Metro Ethernet Networks Nortel Networks Whitepaper Sept pdf [1] Riverstone Networks Scalability of Ethernet Services Networks [17] G. Holland. Carrier Class Metro Networking: The High Availability Features of Riverstone s RS Metro Routers Riverstone Networks White Paper #13. access1 core aggregator1 access access3 APPENDIX core core1 core core access core3 aggregator access access Figure 3 ST 1 configuration for MSTP and COST and the initial ST configuration for RSTP before any failure. access1 core aggregator1 access core core1 access3 core core access core3 aggregator access access Figure ST configuration for MSTP and COST access1 core aggregator1 access core core1 access3 core core access core3 aggregator access access Figure ST 3 configuration for MSTP and COST access1 core aggregator1 access core core1 access3 core core access core3 aggregator access access Figure ST configuration for MSTP and COST