Multipoint Bridged Ethernet Using MPLS Virtual Private LAN Services

Multipoint Bridged Ethernet Using MPLS Virtual Private LAN Services By Scott Kotrla How VPLS Compares to Existing VPN Technologies Multipoint Bridged Ethernet is a service offering that attempts to recreate the functionality of an Ethernet Local Area Network (LAN) in a distributed fashion over the Metropolitan Area Network (MAN) and Wide Area Network (WAN). Virtual Private LAN Services (VPLS) is an implementation of Multipoint Bridged Ethernet (MBE) that uses MultiProtocol Label Switching (MPLS) to implement multiple MBE service instances in a scalable manner with Quality of Service (QoS) and customer isolation. MBE service uses the commodity pricing of Ethernet interfaces and customers familiarity with Ethernet in the LAN to remove the barriers between LAN, MAN, and WAN communication. Current customer premises equipment (CPE) is maintained and supported by the same staff supporting the LAN, and CPE cost efficiencies can be used to purchase additional bandwidth. While the technology has the potential to offer many customer benefits, it comes at the cost of adding additional complexity to the service provider network. Many MBE benefits can be obtained using Ethernet as an access method to existing Virtual Private Network (VPN) services, either through native access or through the InterWorking Functions (IWF) facilitated by MPLS. Multipoint Bridged Ethernet VPN Like network-based IP VPNs, MBE is an any-to-any service offering, which makes MBE both functionally and architecturally more similar to a Layer 3 VPN (L3VPN) than an Ethernet point-to-point Layer 2 VPN (L2VPN). The primary difference between the two any-to-any VPN technologies is the way packets are forwarded to their destination location at the service edge either by IP address or by Ethernet Media Access Control (MAC) address. This relatively small change in the forwarding plane has some significant ramifications on the IP service transparency experienced by an enterprise, as well as the network scalability experienced by the service provider. Also, similar to L3VPN technology, scalable MBE service is typically built using MPLS as described by ongoing work in the Internet Engineering Taskforce (IETF) under the name of VPLS. Because of the similarities, many network devices that support L3VPN also support MBE in the same platform. Page of 10

Advantages One logical sub-interface on each router Flexible bandwidth with support for bursting No changes required at existing sites for adding a site Complete flexibility in choice of L3 protocols and architecture Easy migration from existing VPN service No easy provisioning once logically connected to service Disdvantages Packet replication and associated bandwidth consumption Requires Ethernet connectivity to the customer Figure 1: Multipoint Bridged Ethernet VPN Ethernet Bridging Functionality Ethernet bridging allows devices in an Ethernet broadcast domain to communicate with each other automatically without any configuration. Unlike IP addresses, which are manually configured on a device or subnet basis, Ethernet MAC addresses are assigned to devices by the manufacturer and are globally unique. This is managed by breaking the address range into a dedicated address block per device manufacturer and allowing each device manufacturer to assign addresses to devices from their dedicated address range as needed. MAC Learning In an Ethernet bridge, the table that tracks MAC address port origination is called the MAC forwarding table. The process of associating source MAC addresses with their originating ports is called MAC learning. All that is needed for communication between two devices in an Ethernet network is the MAC address of the distant device. Since most devices use IP to communicate, Address Resolution Protocol (ARP) is employed to bind an IP address to an Ethernet MAC address. ARP leverages the broadcast capabilities of Ethernet to send an ARP request to every device on the Ethernet network by targeting the broadcast MAC address of all Fs in hex. If a device with the target IP address is on the Ethernet network, it will respond with its MAC address which is then used for future communication to the device. Because every Ethernet packet has a source and destination MAC address, the ARP response can be sent to the specific destination MAC address of the device that made the ARP request. As these ARP packets are traveling across the LAN, the Ethernet bridges (i.e., switches) on the LAN examine the source MAC address in every packet in an attempt to learn the location of each device on the network. Like a broadcast packet, a packet with an unknown destination MAC address is sent to all devices on the Ethernet network until the location of the device is learned. Page 2 of 10

Virtual LANs Virtual LANs (VLANs) are used to create multiple broadcast domains in an Ethernet network that allow groups of users to communicate privately. Within a broadcast domain, traffic will be sent to ports that are members of the same VLAN only. Along with port and source MAC addresses, VLANs are stored in the MAC forwarding table. Devices that support multiple VLANs apply a VLAN tag to every packet that leaves the device. Devices that do not support VLANs are typically assigned a port-based VLAN in the Ethernet bridge to which they are directly connected. Trunks between Ethernet bridges typically contain all of the VLANs that are in use on the network. Spanning Tree Protocol Because Ethernet bridges are able to broadcast packets, bridging loops can be created if there are multiple paths through the network between two bridges. Spanning Tree Protocol (STP) is the bridging protocol designed to prevent bridging loops in the Ethernet network by shutting off ports to eliminate multiple paths between devices. In the simplest case, redundant links between two bridges with STP disabled create a bridging loop that can create a broadcast storm on the network as seen in Figure 2. In Figure 2, a single broadcast packet sent into port 1 is broadcast to ports 2 and 3 (shown in red). At port 2, the first packet is broadcast to ports 4 and 3 (shown in blue). At port 3, the second packet is broadcast to ports 4 and 2 (indicated in green). This continues indefinitely and, as additional packets are sent from ports 1 and 4 into the network, eventually all of the bandwidth on every link of the network is consumed. STP prevents this from occurring by blocking the connection between ports 3 and 3, so that there are no bridging loops. Figure 2: Bridging Loop Ethernet in the LAN Understanding the benefits that MPLS provides in implementing MBE service requires an understanding of the limitations of Ethernet in the LAN. Unlike a WAN where there are different varieties of Ethernet service for different applications, Ethernet and MBE are synonymous in the LAN and have the following attributes. VLANs identify broadcast domains. VLANs are globally significant, creating a limitation of 4,094 broadcast domains per network. Any VLAN can consume all available space in the MAC forwarding table. Any VLAN can consume all available bandwidth on a port. Broadcast traffic is treated with the same priority as non-broadcast traffic. Broadcast storms are prevented through the use of STP, which ensures that there are no loops in the network. Virtual Private LAN Service VPLS is a standards effort that is currently underway in the L2VPN working group of the Internet Engineering Task Force (IETF). The working group aims to create an interoperable standard for MBE over MPLS. There are two different approaches to signaling pseudowires (PW) between MBE service instances on different MPLS Provider Edge (PE) routers. One approach uses extensions to Label Discovery Protocol (LDP) to build connections across an MPLS network the same way that point-to-point virtual private lines are signaled across an MPLS network. The other approach uses extension to Border Gateway Protocol (BGP) to build connections across an MPLS network, which is the same way that network-based Private IP VPNs are signaled. Both protocols achieve the same result creating a more scalable architecture for metro, national, and global MBE services. Benefits of VPLS Over Traditional Ethernet Many of the benefits of VPLS over traditional bridged Ethernet are the result of the different markets at which the two technologies are targeted. Ethernet bridges are employed by individual customers for use in the LAN, while MPLS PEs are used by service providers for serving multiple customers in the WAN. Page 3 of 10

Some of the benefits of VPLS include: VLANs identify Ethernet flows and can be mapped to broadcast domains. VLANs are locally significant, allowing for more than 4,094 broadcast domains per network. Space in the MAC forwarding table can be limited on a per-vpls service instance basis. VLAN bandwidth can be policed/shaped to the amount the customer purchased. Broadcast traffic can be limited to a fraction of the provisioned bandwidth. Broadcast storms are prevented through the use of split horizon, which prevents broadcast loops. Split Horizon The main architectural difference between traditional bridged-ethernet and VPLS is the use of split horizon instead of STP. STP suffers from the following problems: Slow reconvergence after network failures on the order of minutes for very large networks Possibility of high latency through the network, since the shortest path through the network can be blocked Possibility of service disruption if the spanning tree blocks the only path to an end destination Stranded bandwidth caused by blocked ports Split horizon solves many of the scalability problems that STP experiences in very large deployments. It uses the tunnel capabilities of MPLS to build a logical full mesh of connections called Label Switched Paths (LSP) between VPLS service nodes. Because all of the VPLS service nodes are fully meshed, VPLS service nodes only forward traffic to and from customer devices served from the VPLS service node. Traffic received from a VPLS service node is never sent back to other VPLS service nodes, because the device expects that a full mesh of connectivity exists between all VPLS service nodes. In Figure 3, VPLS PE4 will never forward traffic from VPLS PE1 to VPLS PE2, even in the event of a network failure. Instead, the VPLS service node relies on the underlying MPLS tunnels to reroute traffic and restore connectivity. VLAN 1 VLAN 2 VLAN 3 GE VPLS PE1 VPLS PE3 Private MPLS Network VPLS PE2 VPLS PE4 VLAN 7 GE VLAN 6 VLAN 5 Legend High Speed Connection (relative to ) Low Speed Connection (relative to ) Layer 2 Switch Layer 2 Aggregator Building Aggregator CPA Network Element VPLS Network Element VLAN 4 Figure 3: VPLS Service Example Figure 3 depicts implementing VPLS using Converged Packet Access (CPA). For simplicity s sake redundancy is not depicted in this figure. The figure breaks CPA aggregation and VPLS service into separate functions, but, in some instances, these functions could reside in the same platform. In this example, the enterprise has seven locations served by three VPLS service nodes. Each service node is a different metro area. Page 4 of 10

These VLANs are being applied in one of the following three manners. Each customer location is transmitting untagged traffic, and the VLANs are being pushed by the CPA network. Each customer location is transmitting the same VLAN tag, and the VLANs are being swapped by the CPA network to the listed values. Each customer location is transmitting multiple VLAN tags, and the VLANs are being pushed by the CPA network as stacked carrier tags. Interaction Between CPA and VPLS Bridging only occurs in the VPLS PE function. Traffic between customer locations in the same metro area has to go all the way to the VPLS PE for bridging to occur. The CPA network is designed to provision large numbers of point-to-point VLAN flows from the customer s premises to the various service edges; one of which is the VPLS service edge. The handoff to the VPLS service edge is via a single gigabit Ethernet (GE) interface that carries all of the aggregated VPLS traffic within the metro area. VLANs are locally significant in the CPA network and their value is swapped as needed to the next available value. The same VLAN value is not shown at every site, because all of the traffic in the metro area eventually will traverse a single connection to the service edge. Figure 3 only shows devices that are engaged in packet processing. Lower layer devices like TDM ADMs and optical ADMs (OADM) can be placed between devices as needed for transport between physical locations. VPLS Packet Walkthrough In Figure 3, a broadcast packet sent from the customer location designated as VLAN 1 will traverse the CPA network to the GE connected to VPLS PE1. At VPLS PE1, it will be replicated and sent back to the same GE as VLAN 2 and VLAN 3, which will be groomed through the CPA network to the other two customer locations in the metro served by VPLS PE1. The broadcast packet also will be replicated and sent across the MPLS network to VPLS PE2 and VPLS PE4. Because VPLS PE3 has not been provisioned as part of the VPLS service instance, the broadcast packet is not sent to VPLS PE3. At VPLS PE4, the packet will be forwarded to the GE shown as VLAN 7. Finally, at VPLS PE2, the broadcast packet will be replicated and sent down the GE as VLAN 4, VLAN 5, and VLAN 6, which will be groomed through the CPA network to the final three customer locations served by VPLS PE2. Private IP vs. VPLS Figure 4 shows a network diagram of a Private IP service with access via CPA. The diagrams look almost identical to the VPLS service shown in Figure 3, and, in many cases, the same MPLE PEs can provide both services. VLAN 1 VLAN 2 VLAN 3 GE Private IP PE1 Private IP PE3 Private MPLS Network Legend Private IP PE2 Private IP PE4 VLAN 7 GE VLAN 6 High Speed Connection (relative to ) Low Speed Connection (relative to ) Layer 2 Switch Layer 2 Aggregator Building Aggregator CPA Network Element Private IP Network Element VLAN 5 VLAN 4 Figure 4: Private IP Service Example Page of 10

Network Costs Many cost benefits of VPLS over legacy VPN services actually are the benefits of Ethernet interfaces and not Ethernet bridging. If VPLS and Private IP services use the same access network, the same MPLS core, and, in some cases, the same PEs to provide the service, why would there be any cost difference in the two technologies? Encapsulating an IP datagram in Ethernet does not magically reduce network cost. Because of MAC address randomness, changes in the IP world that reduce the number of IP routes and increase scalability are actually useless in the world of Ethernet bridging. This means that VPLS is less scalable than Private IP with the same hardware, which actually makes the service more expensive to deploy than Private IP service. VPLS Scaling Challenges Key scaling limitations of VPLS compared to Private IP include: Limits on the number of MAC addresses Reduced bandwidth from broadcasts Total network size These scaling limitations are the result of the MAC learning that reduces configuration (compared to Private IP) and are at the very heart of bridged networks. That means these limitations will be difficult to overcome. As VPLS networks scale nationally and globally, MAC learning takes longer and bandwidth efficiency is decreased from additional broadcasts. How to Know If VPLS Is Right for You Private IP is an established service that can meet the needs of the majority of Verizon Business s customers. However, there are some customers that would be better suited for a future Verizon Business VPLS offering for a variety of reasons: Non-IP protocols required Special routing protocol requirements, including MPLS More comfortable with a Layer 2 service Less reconfiguration to move from existing Layer 2 service Router configuration more consistent at every location For customers with these and similar requirements, VPLS may be the right solution. Page 6 of 10

Appendix A This Appendix includes information on existing VPN service offerings, which includes: Physical Private Line Network Virtual Private Line VPN CPE-Based IP VPN Network-Based Private IP VPN Physical Private Line Network A network composed of physical layer circuits is actually more of a Private Network than a VPN. This type of network can be composed of traditional Time Division Multiplexing (TDM) circuits, TDM circuits with Ethernet handoffs, wavelengths, or dark fibers. The distinguishing characteristic of this network is that every connection between locations requires additional physical ports. For a full mesh of connectivity between n locations, the number of ports increases on the order of n2. Adding new locations requires adding as well as configuring new physical ports at existing locations. Since all connections are point-to-point, the CPE decides through which circuit any given packet is sent. Because physical connections between locations exist, bandwidth must be purchased in large increments. Utilization often is low within the physical pipes between locations, because they each have to be sized for peak capacity. Increasing bandwidth usually requires swapping out low-speed uplinks for higher-speed uplinks, which is cost and time prohibitive. However, this network provides complete flexibility in link layer (Ethernet, Frame Relay, ATM, PPP) choice and network layer (IP and IPX) protocols, including routing protocols and architectures. Furthermore, security and QoS are very strong because of the physical layer isolation between customers. Advantages Complete flexibility in choice of L2/L3 protocols and architecture High perceived security Perfect QoS Bandwidth scales from low to high Disdvantages Multiple TDM interfaces on each router Confined to TDM bandwidth increments with no bursting Adding a site requires new circuits all across the network High cost Figure A1: Physical Private Line Network Page 7 of 10

Virtual Private Line VPN The most popular virtual private line VPN is Frame Relay, but the same architecture can be used to build VPNs using ATM and Ethernet technologies. Prior to the emergence of Ethernet in the MAN and WAN environments, this type of VPN was synonymous with Layer 2 VPN (L2VPN). Presently this type of VPN is more accurately called a point-to-point L2VPN, because each connection only is between two locations. The major difference between a Frame Relay VPN and a network composed of physical private lines is the notion of logical ports within the physical port. Instead of all the traffic leaving a physical port and going to the same destination, traffic can be forwarded to different locations based on the tag carried within the frame (e.g., DLCI for Frame Relay, VLAN for Ethernet, VPI/VCI for ATM). This eliminates the need for previously mentioned n2 physical ports. Instead, a single physical port can be subdivided into multiple logical ports for connection to multiple locations across a switched network potentially a huge cost advantage. Since all connections are point-to-point, the CPE decides through which virtual circuit (VC) any given packet will be sent. Adding new locations requires adding new logical interfaces at existing locations. This can be done remotely, since no new interfaces or cabling is required. Because there are logical connections between locations, bandwidth can be purchased in much smaller increments than TDM channels or wavelengths can provide. Furthermore, the interface can be sized for the peak demand of individual connections instead of the peak demand of all connections in aggregate. Upgrading the bandwidth between sites also can be done much more quickly via remote provisioning assuming bandwidth is available on the physical port. Customers with this type of VPN service must use the link-layer protocol of the service (e.g., Ethernet, Frame Relay, or ATM), but any network protocols and routing protocols can be used similar to a network built with private lines. Because the service network has no knowledge of the IP layer, this type of VPN is IP-service transparent. Due to the amount of logical provisioning required for a full mesh, these types of VPNs often are built as hub-and-spoke networks with many small, remote locations having single connections to the main office and the main office hub having spoke connections to all of the remote locations. The perception of security is less for virtual private lines than for physical private lines, but the main security risk is misprovisioning, which can happen almost as easily in networks composed of physical private lines. QoS in virtual private line VPNs varies by service provider and service network, but it can become very poor if the network is largely oversubscribed. By leveraging the deployment of MPLS technology, different customer locations can use different protocols via MPLS IWF to convert between protocols as needed (e.g., Ethernet access to Frame Relay VPN). Advantages One physical interface on each router Complete flexibility in choice of L3 protocols and architecture Flexible bandwidth with support for bursting Disdvantages Multiple logical subinterfaces on each router Adding a site requires new connections all across the network Figure A2: Virtual Private Line VPN Page of 10

CPE-Based IP VPN CPE-based IP VPNs are actually a subclass of virtual private line VPNs and use the public Internet for lowcost connectivity, especially at low speeds. This type of VPN is very common for remote access applications where small remote offices need connectivity into a hub location. For this reason, these types of networks are commonly hub-and-spoke designed. Since the public Internet is insecure, encryption is employed to build secure tunnels between locations. These tunnels are the virtual private lines that make up this type of VPN. Adding new locations requires adding new encrypted tunnels at existing locations. This can be done remotely, since no new interfaces or cabling is required. Low cost access is partially offset by the cost of encryption hardware and software at each location. At higher bandwidths, this is often cost prohibitive. This type of VPN has the advantage of leveraging the ubiquity of the public IP network, spanning multiple providers across the globe. The primary disadvantage is the lack of QoS. The public Internet is a best-effort network and neither bandwidth nor priority can be guaranteed end-to-end. Advantages One physical interface on each router Complete flexibility in choice of L3 protocols and architecture Flexible bandwidth with support for bursting Low cost works across existing Internet service No provisioning Disdvantages Multiple logical sub-interfaces on each router Low QOS Low speed Figure A3: CPE-Based IP VPN Page 9 of 10

Network-Based Private IP VPN Network-Based Private IP VPNs are much simpler to operate and maintain than point-to-point L2VPNs. This type of VPN leverages MPLS to provide secure IP connectivity between customer locations and is often called Layer 3 VPN (L3VPN). Because IP is a network protocol instead of a link-layer protocol, customer connectivity is any-toany instead of point-to-point. This means that each customer router only has one logical connection (or two for diversity) into the service network. Instead of forwarding packets based on logical tags applied by the customer, the network routes the traffic to the proper destination based on the IP address. When new customer locations are added to the VPN, no new configuration is required at existing customer sites. While this is simpler, it does increase the complexity of the service network. Because the link-layer protocols only exist from the customer s premises to the service edge, different access methods can be used at each customer location without any additional configuration, complexity, or IWF such as mixing Ethernet and Frame Relay access. With network-based Private IP, bandwidth is much more flexible than it is with point-to-point L2VPNs. Instead of purchasing and provisioning bandwidth per logical connection, bandwidth is purchased and provisioned in aggregate as access links into the VPN. This decreases the required bandwidth because all of the bandwidth at a single location can be used for communication to another location at any given time. The main disadvantage of this VPN type is that customers are more restricted in their choice of network protocols, routing protocols, and routing architectures. Since the customer has to peer with the service provider at the IP layer, non-ip protocols are not supported and coordination on IP addressing, routing protocols, and tunneling technologies, such as customer MPLS, has to be arranged with the service provider. That means the VPN is not IP service transparent. This requires relinquishing some service flexibility in order to simplify the network. Advantages One logical subinterface on each router Flexible bandwidth with support for bursting Adding a site requires no changes at existing sites Disdvantages Limited choice of routing protocols and IP architectures Likely requires changes in IP network architecture IP only Coordination with service provider required on IP changes Figure A4: Network-Based Private IP VPN We never stop working for you. 2006 Verizon. All Rights Reserved. WP10881 01/06 The Verizon and Verizon Business names and logos and all other names, logos, and slogans identifying Verizon s products and services are trademarks and service marks or registered trademarks and service marks of Verizon Trademark Services LLC or its affiliates in the United States and/or other countries. All other trademarks and service marks are the property of their respective owners. Page 10 of 10