Situation Since 2005, Microsoft IT has been exploring the implementation of CMPLS for its corporate network. Carriers initially could not meet Microsoft IT requirements, especially in the areas of IPv6 and multicast support. As carrier offerings evolved and Microsoft IT decided to deploy CMPLS, Microsoft IT continued to encounter challenges, from technical specifications and support to business processes, such as billing for services. Solution Microsoft IT developed several processes and best practices to improve carrier selection, testing, certification, and deployment. Benefits More efficient process for vendor selection Improved partnerships with strategic suppliers More effective testing and certification Streamlined deployment Microsoft IT Processes Improve Deployment of Carrier-Provided Multiprotocol Label Switching Published: June 2012 Microsoft Information Technology (Microsoft IT) decided to implement carrier-provided Multiprotocol Label Switching (CMPLS) for its corporate network. However, due to the complexity of both the technology and Microsoft IT requirements, a variety of challenges arose. As a result, Microsoft IT developed several processes and best practices to improve the carrier selection, testing, certification, and deployment of CMPLS. This case study is designed for IT decision makers and engineers of enterprises that have decided to deploy CMPLS and have nonstandard technical requirements. The primary goal of this document is to provide lessons learned, best practices, and recommendations to help other organizations deploy CMPLS more efficiently and effectively. Situation Since 2005, Microsoft IT has been working to fulfill its strategy to incorporate Multiprotocol Label Switching (MPLS) into its corporate network. MPLS is a highly scalable, protocol independent technology. In MPLS, data packets are assigned labels, and packet-forwarding decisions are based solely on the contents of this label. Because the packet itself does not need to be examined, end-to-end circuits can be created across any type of transport medium, via any higher-layer protocol. This eliminates the dependency on a particular data-link-layer technology that is, asynchronous transfer mode (ATM), frame relay, Synchronous Optical Network (SONET), or Ethernet and the need for multiple Layer 2 networks to carry different types of traffic. MPLS may thus be used to deliver multiple virtual private networks (VPNs) on a single, common network infrastructure, any-toany site connectivity, and built-in business continuity and disaster recovery. Additionally, MPLS is the technology that carriers are now focusing on and investing in for delivering wide area network (WAN) and voice services, while phasing out the older technologies of ATM and frame relay. The deployment of CMPLS is a component of a larger Microsoft IT network convergence and internal MPLS deployment effort. As shown in Figure 1, this effort includes using a private MPLS backbone network and a carrier-provided MPLS to bring multiple applications and services, such as data, voice, video, and telepresence, from separate circuits and networks onto a single, common network.
Figure 1. MPLS and network convergence at Microsoft Although CMPLS was available in 2005, Microsoft IT chose to deploy CMPLS in its network only recently. The decision was due to two primary requirements that carriers could not initially meet: support for Internet Protocol version 6 (IPv6) addressing and support for the unique multicast needs of Microsoft. Between 2005 and 2009, Microsoft IT worked to encourage carriers to support its unique requirements. In 2007, Microsoft IT deployed MPLS in its private network in the Puget Sound region and began working to deploy MPLS across its corporate backbone network, while continuing to study carrier capabilities. Microsoft Requirements vs. Carrier Capabilities Lack of IPv6 Support in CMPLS IPv6 was designed to succeed Internet Protocol version 4 (IPv4). The growth of the Internet has created a need for more addresses than are possible with IPv4. The Internet Engineering Task Force (IETF) developed IPv6 to deal with this long-anticipated IPv4 address exhaustion. For many organizations, IPv6 support was not a primary requirement. In 2009, when Microsoft IT began to actively pursue the selection and certification of carriers, less than 1 percent of Internet traffic transiting major Internet exchanges was IPv6, and less than 25 percent of large enterprises were planning for IPv6 adoption. Due to this low demand, combined with the complexities of providing IPv6 support, this support was not a priority for most carriers until the recent depletion of IPv4 addresses drew more focus to the issue. However, IPv6 support has been critical for Microsoft for many years, because of the company's extensive software product development. The Windows operating system and other Microsoft products must be proven to work on IPv6 as part of internal deployments. Page 2
Additionally, because the Microsoft IT internal deployment of MPLS in its private network supports IPv6 and any CMPLS deployment would be an extension of that network, carrier support of IPv6 is vital. IPv4 has matured to a state that is very robust and supportable, whereas IPv6 support has a long way to go to achieve a high level of supportability. Therefore, Microsoft IT continues to drive network hardware vendors and service providers toward complete congruency of features and operational visibility between IPv4 and IPv6 protocols. Insufficient Support of IP Multicast IP multicast is a technique for one-to-many communication over an IP infrastructure in a network. An example of an application that uses multicast is video streaming. Multicast scales to a larger receiver population by not requiring prior knowledge of the identity or number of receivers. Multicast uses network infrastructure efficiently by requiring the source to send a packet only once, even if it must be delivered to a large number of receivers. Until recently, carrier MPLS offerings did not support Microsoft IT's multicast requirements because of scaling issues and lack of standards to support multicast. Carrier routers must maintain the state of every multicast stream, which can push the processing limits of routers. Microsoft IT currently has a worldwide deployment of multicast on its corporate network, so any deployment of CMPLS must provide equivalent support of the following policies and needs. Unrestricted points of origination. Microsoft policy allows anyone on its network to be a sender or a receiver of a multicast stream. A recent measurement showed more than 3,400 active multicast senders distributing content within the Microsoft corporate network. These sources include streaming events for company-wide events, group meetings, and online training. Product development. Microsoft is in a unique position because of the nature of its product development. Approximately 80 percent of multicast traffic on the network is development related with an average aggregate rate (single snapshot of global rendezvous points (locations of stream sources) of 750 megabits per second (Mbps), if all current active sources have a single receiver. Internal groups like Xbox, Microsoft Mediaroom, and Windows Media Player also must test new versions of their products that use multicast for content delivery. Software distribution. Windows Deployment Services uses multicast for delivering software updates within the Microsoft corporate network. Decision to Deploy CMPLS In 2009, it became clear that a handful of carriers were becoming able to support the full extent of Microsoft IT's requirements for CMPLS. With the depletion of IPv4 addresses, carriers began prioritizing IPv6 address support, and some carriers began agreeing to support the Microsoft IT requirements for multicast support. Microsoft IT decided to move forward with CMPLS, and the process of identifying carriers began. However, the complexity of both MPLS technology and Microsoft IT requirements, combined with the variety of ways that carriers can deploy an MPLS network, necessitated a more thorough selection, testing, and certification process. Page 3
Solution As Microsoft IT determined the capabilities of carriers' MPLS offerings, developed a testing and certification process, and began deployment at various pilot and production sites, a variety of challenges arose. Some of these were due to requirements that needed to be more specific. Others were the result of technical capabilities and support, network configurations, or business processes that needed to be changed or accounted for. Carrier Capabilities During the carrier selection process, Microsoft IT developed and refined a questionnaire designed to identify carriers that could meet Microsoft IT's specific requirements. This process uncovered several areas that required further clarification. Need to Specify How Requirements Must Be Met In some cases, although carriers had the capability to deliver several requirements, Microsoft IT later determined that they could not do so in the manner that Microsoft IT required. For example, some carriers could support IPv4, IPv6, and multicast. They therefore correctly answered "yes" to each of those questions. However, they did not clarify that they could not initially provide this support on the same port, as Microsoft IT required. Another issue with carriers' IPv6 support arose because Microsoft manages the customer edge in its environment. Microsoft IT therefore does not advertise third-party public address space within its corporate network. For some carriers, this requirement presented challenges in their provisioning systems, which were previously designed to support only their own addressing schemes. Another example of carrier capabilities matching what was requested, but not how it was needed, pertained to multicast. Again, carriers correctly answered "yes" to multicast support, but did so without knowing the scale and extent required. Therefore, Microsoft IT developed a new questionnaire that eliminated original assumptions by specifying more precisely how Microsoft IT needed carriers to deliver key capabilities. In working with carriers to meet its multicast needs, Microsoft IT drove the adoption of the Bootstrap Router (BSR) protocol within the carrier VPN service offerings. This solution supports automatic detection of rendezvous points to improve multicast support. Carriers certified this solution expediently and ubiquitously in response to this Microsoft IT requirement. Need to Specify Who Will Deliver the Requirements In some cases, carriers indicated that they could provide services to specific sites, but did not disclose that they would need to use partners to accomplish the connectivity. Due to the complexity of Microsoft IT's requirements, the variety of ways that a partner may deliver its MPLS service, and the level of testing required, Microsoft IT avoids Network-to-Network Interfaces (NNIs), which allow network handoffs from one carrier to another. Therefore, Microsoft IT added this clarification to the questionnaire. Need to Confirm That Carrier Processes and Systems Match Technical Capability Another lesson learned in the selection process is that carriers may have the technology to meet a requirement, but not the operational support, personnel, or processes in place to actually deliver it. In several instances, carriers indicated that they could deliver specific technical capabilities, but because there were no current product offerings in the billing or provisioning systems to support these, delays occurred. Some examples of this were support Page 4
for large ports (greater than 1 gigabit per second [Gbps]), the ability for customers to use their own IP addresses for IPv6, and support for MD5 authentication. Each of these situations caused delays with testing and deployment of some of those features. As a result, the carrier selection process now includes asking questions that address whether a carrier has the systems, engineering, and processes in place to support the technology requirements. Security Concerns Security concerns about traffic being routed incorrectly also arose during the carrier selection process. To resolve this, Microsoft IT created an Address Community Filter plan in which each address prefix was tagged to ensure that only these addresses would be accepted on the Microsoft network. Some carriers are now implementing the Address Community Filter Plan to ensure that Microsoft IT-assigned public IPv4/IPv6 addresses are not advertised or reachable from both the corporate network VPN and the public Internet. Latency Issues Latency will always be a function of distance. CMPLS can reduce latency compared to a huband-spoke design by shortening the round-trip distance between many locations. However, in some instances, latency can actually increase because of the locations of a carrier's points of presence (POPs). During testing, latency between some sites and the network services hub increased, because the closest carrier MPLS POP was farther away than the POP that serviced that site on the original network. Because carriers are still building their MPLS networks, fewer carrier POPs are currently enabled for CMPLS than for the network it replaces. Therefore, when Microsoft IT is selecting carriers in the future, it will consider the geographic location of their CMPLS POPs. Another example of needing to plan with the carrier for the location of services was discovered with sites that were being upgraded from Layer 2 VPN to Layer 3 VPN. In one case, a carrier had to change the Provider Edge (PE) router to one located in a different city which increased latency by 120 milliseconds between local sites. Ultimately, the carrier installed a PE to support Layer 3 service in the original POP location. However, the resolution of this issue caused delays and additional resource consumption. Lab and Pilot Testing and Certification Lab and Pilot Testing Time Lab testing took three to five months per carrier, based on the lead time for test circuits, for time to test, and for carriers to make adjustments to their systems to accommodate Microsoft IT's unique requirements. Pilots lasted 30 to 90 days, depending on whether Microsoft IT identified additional issues. As the testing and certification process evolved and documentation was completed, Microsoft IT hoped to shorten these times by providing a specific test and documentation plan and having carriers do more of the testing themselves. Even with this self-test plan in place for a significant portion of the testing, Microsoft IT will continue to confirm and test carrier capabilities in a lab against unique aspects of the Microsoft IT network. As of January 2012, three carriers had successfully completed a lab test and pilot, conducted by Microsoft IT engineering, and now have circuits in production in the United States, Latin America, and Asia. A fourth carrier has begun conducting a lab test in its own facility, based on the test plan that Microsoft IT engineering provided. Microsoft IT will still conduct Page 5
additional testing to validate the carrier results, but having the carrier use its own lab facility is expected to significantly reduce the requirement of Microsoft time and resources. Issues with Configurations During the testing and certification process, several issues with configurations affected the turn-up of pilot circuits. In one case, a carrier's router was not configured with as-override set for the Microsoft VPN, preventing the Microsoft router at that location from detecting all internal routes. Because of this issue, the Partner Edge router could not detect any of the other Microsoft sites that were connected to the carrier's MPLS cloud. In another case, a carrier's router was missing a filter configuration at the pilot hub location. This configuration (used to filter and mark customer VPNs from internally used carrier VPNs such that internal prefixes are never advertised to customers) caused all Microsoft IT learned prefixes at the hub location to be treated as internal carrier prefixes. Thus, they were not advertised back into the Microsoft IT VPN, and the Microsoft IT hub router could not detect them. The Microsoft pilot hub router could therefore not detect the Microsoft offices that were on the carrier's MPLS network, leaving sites unable to connect to any of the others. Because some of the necessary configurations were previously nonstandard in the carrier deployment, Microsoft IT resolved some challenges by having those configurations consistently enabled across the carrier network. This was identified as an issue in the pilot, and the carrier modified its process to ensure that these configurations were included in the turn-up checklist. Microsoft IT still encountered challenges during some of the initial production CMPLS circuit turn-ups. Microsoft IT attributed these challenges to insufficient training of carrier personnel with the updated turn-up process. Production Deployment Implementation of CMPLS across 60 sites in the United States has taken approximately 16 months. In the United States, 51 CMPLS circuits are currently deployed across the Microsoft corporate network. There are an additional 12 Microsoft Technology Center (MTC) locations in North America on a separate CMPLS network from a second carrier. The MTC private network did not have the same set of requirements as the Microsoft corporate network and was deployed before any of the carriers completed CMPLS certification with Microsoft IT. Multiple sites now also exist in Asia, Latin America, and the EMEA (Europe, Middle East and Africa) region. During the implementation and deployment at some of these sites, the following issues arose. Local loop Ethernet/802.1Q support. Microsoft IT encountered some problems when it deployed some CMPLS circuits in Asia by using Ethernet as the local loop/last mile (circuit connection from Microsoft site to local carrier POP). The access provider that the CMPLS carrier selected did not support the IEEE 802.1Q standard, which supports virtual LANs across the circuit. This selection resulted in the inability to virtualize the circuit to establish multiple VPNs. The Ethernet circuit had to be replaced with a point-to-point access circuit to resolve the problem. Relying on the CMPLS carrier to verify this requirement with its local access providers increased the potential for problems. This issue did not occur on serial circuits (E1 and E3) that used frame relay encapsulation. Microsoft IT has provided this information to carriers to include in their processes for selecting and contracting local access providers. Lead times and scheduling. The access circuits determined the delivery intervals for the new CMPLS circuits. Similar to ATM and Private Line circuits, the average delivery interval Page 6
for US circuits was 104 calendar days for those circuits that did not require additional facilities or fiber installations. Microsoft IT expects this process to take even longer in less developed regions, and it has adjusted deployment planning to account for this timing. Circuit order specifications. Because this is a Layer 3 service, Microsoft IT had to change its ordering process to include a formal network design for each circuit at the beginning of the project. This is required because of the additional Layer 3 information that Microsoft IT needs at the time of order, including VPN name, VPN capacity, Class of Service (CoS) queue distribution, partner and customer router IPv4 and IPv6 addresses, and autonomous system number. This led to a new form, created to capture and integrate this information into the circuit ordering process and provide all required details up front to engineering. Hardware requirements at sites migrating from ATM, Private Line, Internet. Locations that were moving from ATM to CMPLS with DS3 or smaller access circuits had to replace their interface card with a new DS3 or T1 card because the ATM card could not be reused. Sites that were converting from Internet VPN offices to CMPLS were generally able to reuse their DS3 network interface card. Sites that required interfaces larger than DS3 varied, depending on the model of router deployed at a site. Hub port capacity requirements and traffic patterns. Microsoft IT sized the CMPLS port at the network hub in the Microsoft headquarters (Redmond, Washington) to accommodate 100 percent of the aggregate bandwidth to individual sites. After reviewing production traffic reports, Microsoft IT observed that the average utilization on this hub port was only approximately 50 percent (with peaks approaching 70 percent) of the aggregate site bandwidth. This was due in part to the nature of daily increases in traffic during work hours spread across several time zones, where there is a smoothing effect on hub utilization. Additionally, a portion of the traffic goes directly between sites, bypassing the hub site. Microsoft IT will continue to analyze traffic patterns to determine whether it can reduce the hub port sizing. Challenges with Out of Band (OOB) management. Though not unique to CMPLS circuit turn-ups, Microsoft IT estimated that around 80 percent of the sites had issues with their OOB connections to facilitate router configurations during the circuit turn-up. This was due to a combination of factors, including problems with the earlier modem supporting the terminal server and missing or disconnected phone lines used for OOB management. For a coordinated migration, the functionality of the OOB connection should be verified at each location in advance of the scheduled turn-up. Multicast testing challenges. The implementation engineer usually depends on a local resource at the site to conduct a test to verify that multicast is working over the CMPLS circuit. Occasionally, issues have occurred in completing that test, and the engineer has limited ability to conduct the test remotely. A tool to enable this testing from a remote location must be identified to ensure full testing of the circuit. Summary of Lessons Learned Overall lessons that Microsoft IT learned in carrier selection, testing, and deployment include the following: The number and complexity of MPLS features provide a great deal of flexibility in how carriers design and deploy it. Therefore, technical requirements need to be extremely specific and precise to eliminate misinterpretation. Page 7
A carrier's technical resources that support testing, pilots, and initial deployments must be identified and committed up front to avoid lost time troubleshooting unexpected issues that will likely be encountered throughout the process. The carrier's technical resources for circuit turn-ups require a higher level of expertise to support the complexity of MPLS compared to earlier WAN technologies. Carriers must have a balance between the flexibility to support nonstandard requirements but still provide well-defined processes that are followed to ensure proper implementation. The back-end systems (such as provisioning and billing) often limit what functionality carriers can support. An organization should plan on delays for system modifications to accommodate nonstandard features and configurations. Benefits By developing a certification process and resolving and documenting the many challenges that appeared in testing, certification, and deployment, Microsoft IT has begun to realize the following benefits. More efficient process for vendor selection With a refined questionnaire that more clearly describes its requirements, Microsoft IT can quickly short-list eligible carriers. By eliminating carriers that cannot meet its unique needs, Microsoft IT can focus more time and energy on the most qualified carriers. In addition to streamlining resources and eliminating wasted effort in the selection process, the focus on fewer global carriers is consistent with the Microsoft IT effort to consolidate services with strategic suppliers, which is expected to bring additional efficiencies. Improved partnerships with strategic suppliers Another benefit of focusing on fewer carriers is the potential to improve the partnership between Microsoft IT and the selected strategic suppliers. Microsoft IT's requirements have helped influence carriers to modify their service offerings. This is especially important because of the complexity, range of features, and variety of ways that carriers can deliver MPLS. Microsoft IT expects that as it reduces the number of carriers, and those fewer carriers bid for larger, region-wide contracts, the carriers will be more likely to make investments in their MPLS networks that are consistent with Microsoft IT's evolving needs. More efficient carrier testing and certification By identifying and resolving issues with testing, and by creating a test and documentation plan for carriers, Microsoft IT expects to shorten the time needed to effectively test and certify carriers. An improved testing process will help Microsoft IT prioritize carriers for certification, based on the technical capabilities they can demonstrate. These changes will expand the options in each region and will help minimize delays of CMPLS deployment due to lack of carrier coverage or capability in a particular region. As additional carriers become certified, the potential for negotiated savings will increase as more carriers compete for the business. Streamlined deployment With lessons learned and best practices from earlier deployments documented and applied, Microsoft IT expects to complete future deployments more smoothly, with less wasted effort, rework, and interruption to workflow. Page 8
Conclusion The Microsoft IT WAN is constantly changing to support business needs and continue lowering the cost of the service. Microsoft IT chose CMPLS as the preferred technology for WAN transport to edge locations globally. The complexity of CMPLS technology and the variety of ways that carriers can configure and deploy their MPLS networks led Microsoft IT to determine that it needed a more thorough carrier selection, testing, and certification process for any CMPLS deployment. Challenges that Microsoft IT experienced during the initial deployment of CMPLS in the United States included delays in implementation due to an incomplete understanding of the requirements, failed circuit turn-ups resulting from insufficient availability of technical resources, and lack of back-end systems to order and bill for newly supported features. The improved carrier certification process has addressed these and other challenges. Microsoft IT has now completed CMPLS certification of three global carriers, with additional carriers in queue to complete the process. The effort to certify a selected set of carriers aligns with the Microsoft IT goal of consolidating to fewer carriers that provide global/regional solutions. Microsoft IT has deployed CMPLS service in the United States and part of Europe, Asia, and Latin America. Additional deployments are planned for Europe, Canada, and Latin America. Microsoft IT has managed the requests for proposals (RFPs) for those services more effectively by focusing on the certified carriers and those that have a product roadmap that aligns with Microsoft IT's requirements. Partnering with strategic suppliers that understand Microsoft IT's strategy for managing the corporate WAN will be a priority for the future. Focusing efforts with this smaller set of carriers enables a more in-depth working relationship that allows Microsoft IT to identify and work through complex technical and business issues more efficiently than it can do with multiple, smaller providers. This will be beneficial not only for other CMPLS projects but also for future carrier offerings that require close coordination between the customer and the supplier. For More Information For more information about Microsoft products or services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada Order Centre at (800) 933-4750. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information via the World Wide Web, go to: http://www.microsoft.com http://www.microsoft.com/technet/itshowcase 2012 Microsoft Corporation. All rights reserved. This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. Microsoft, Lync, Office 365, Windows, Windows Media, and Xbox are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners. Page 9