Issue 1.0 BTnet Resilient Extra White Paper for BT People and Prospective s This document provides a summary of the methods and configuration of the Resilient options available for BTnet services. Copyright British Telecommunications plc, 2003. All rights reserved. BT maintains that all reasonable care and skill has been used in the compilation of this publication. However, BT shall not be under any liability for loss or damage (including consequential loss) whatsoever or howsoever arising as a result of the use of this publication by the reader, his servants, agents or any third party. All third-party trademarks are hereby acknowledged. 1 of 11
Contents 1 INTRODUCTION...3 2 RESILIENT - FOR STANDARD BTNET FLEX WITH A MANAGED NTE...4 3 RESILIENT EXTRA - FOR STANDARD BTNET FLEX WITH MANAGED NTE...5 4 BTNET 100 RESILIENT EXTRA SOLUTION USING BGP WITH PRIVATE AS...6 5 BTNET FLEX NRO RESILIENT EXTRA...8 6 APPENDIX A - LOW LEVEL CONFIGURATION...9 6.1 BTNET100 RESILIENT EXTRA...9 6.1.1 Router Configuration...9 7 DOCUMENT INFORMATION...11 7.1 GLOSSARY OF TERMS...11 2 of 11
1 Introduction This document describes the solutions used for providing 'Resilient' and 'Resilient Extra' BTnet services. There are two main variants, 'Resilient' and 'Resilient Extra'. In summary, the 'Resilient' option provides the customer with a single NTE router, that has two links to the same BTnet access router, whereas the 'Resilient Extra' option provides the customer with 2 access connections from 2 different access routers. These would normally connect to two separate NTEs at the customer premises. The following sections provide more detail on the two options, including Resilient and three variants of Resilient Extra: Resilient for standard BTnet Flex with a managed NTE Resilient Extra for standard BTnet Flex with managed NTEs Resilient Extra for BTnet100 using BGP with Private AS Resilient Extra for BTnet Flex No Router Option 3 of 11
2 Resilient - for standard BTnet Flex with a managed NTE This option provides an entry level resilient solution for the access link only. It is available as an option to BTnet Flex customers supplied with a managed NTE router. A secondary link is provided from the same BTnet Access Router on a second sub-interface, and a PVC is configured to a second interface on the NTE router as shown below. The customer connects their network via an Ethernet connection to the NTE LAN interface as below. The customer only needs to configure a default route from their network to an IP address supplied by BT. 2x Static routes to customer network on different sub-interfaces 1 with higher metric (backup link) 2x Default routes to BT core network on different sub-interfaces CR1 1 with higher metric (backup link) To the Internet BTnet Core AR MSIP Shared STM-1 to MSIP Primary Secondary NTE NTE Router Ethernet LAN interface Network Figure 1 - BTnet Flex Resilient The secondary link is only used in the event of a failure of the primary link. Downstream Resilience is provided by using two static routes pointing to the customer network. These are configured on the BTnet Access Router via the two sub-interfaces that connect to the NTE router via two separate PVCs. The customer s network must be advertised down each of the lines, but the advertisement pointing down the secondary link has a higher cost metric tagged onto it, to ensure that the router does not use it unless the main line has failed. A similar method is used to provide the resilient link in the upstream direction for traffic heading towards the Internet. 4 of 11
3 Resilient Extra - for standard BTnet Flex with managed NTE The 'Resilient Extra' configuration is intended to improve on the 'Resilient' service by providing additional protection in the event of an NTE router or Access router failure. Therefore, Resilient Extra is a complete backup solution that provides the customer with a second NTE router, which connects to a second access PoP, which is homed off a different core PoP to their main connection. The backup PVC connects to a different access router from the main PVC. It is effectively the same as delivering two separate links to the customer premises. The customer connects their router onto the same Ethernet LAN as the NTE routers. The customer only needs to configure a default route from their network to an HSRP virtual IP address supplied by BT. In order to maximise the resilience offered to the customer without running a dynamic routing protocol between the NTE routers and the customer s equipment, Cisco hot standby routing protocol (HSRP) is used between the NTE routers, and BGP AS2856 is extended (in very limited form) to the customer s premises. To route correctly to the customer, their address range should be divided such that the normal network allocation is advertised on the Backup NTE, and the same allocation, divided into two parts, is advertised on the Primary NTE. 2x Over-specific static routes to the customer network redistributed into BGP added on NTE and advertised to BTnet Core Primary Priority 105 AR1 NTE1 CR1 To the Internet BTnet Core HSRP with Virtual IP Address Router Network AR2 AR2 Secondary NTE2 Priority 100 CR2 1x Summarised static routes to the customer network redistributed into BGP added on NTE and advertised to BTnet Core Figure 2 - BTnet Flex Resilient Extra The customer network address block is split into 2, and both halves are advertised via link 1. The undivided block is advertised via link 2. This ensures that traffic is correctly routed traffic, as it will always follow the more specific static route. These routes are re-distributed into BGP on the NTEs, and specify the Ethernet interface AND next hop address of the customer's router. This ensures that a failure of the Ethernet interface on the NTE can also be detected. The customer router has a default outbound route to the HSRP virtual IP address, and the NTE with the highest 'Priority' metric is selected by the HSRP routing protocol to send traffic towards the Internet via the BT core network. NTE1 has a Priority of 105, and NTE2 has a Priority of 100 under normal conditions. HSRP monitors the WAN interface of NTE1, so that if a failure occurs somewhere on Link 1, then the Priority on NTE1 is reduced to 95. This has the effect of HSRP selecting NTE2 to use for sending outbound traffic to the Internet, as NTE2 then becomes the NTE with the highest Priority metric. The NTEs have static default routes configured to the access routers for sending outbound traffic. This results in ALL traffic normally being sent and received via the primary link. Traffic is only sent via the secondary link in the event of the primary failing. In summary, HSRP is used for the Outbound Resilient Extra routing changes, and BGP is used for the Inbound Resilient Extra routing changes. 5 of 11
4 BTnet 100 Resilient Extra Solution using BGP with Private AS The BTnet 100 Service is delivered using an Ethernet Access line utilising a single fibre pair connecting from customers sites to the BTnet Access PoP then via a resilient access network to the core network and Global Internet. The customer connects their routers to the service via Ethernet connections to the BTnet NTUs. Note: This solution requires the customer to carry out detailed BGP configuration on their routers as defined below and in the Appendix. Resilient Extra customers are provided with two links that are connected to two different access PoPs and two different Core PoPs. One of the links is designated as the primary and the other as the secondary. Under normal conditions, traffic will only be carried over the primary link. NB as the access circuit is distance dependant it is possible to opt for both primary and secondary links to use the same (nearest the customer) access PoP although separate core PoPs would still be used to increase the resilience. The BTnet 100 resilient extra solution uses BGP with a Private AS to control the routing. The customer routers must be configured to advertise the full routes of the customer network to the BT core network, using BGP with a Private AS. BGP Private AS Full routes to customer networks sent to core using Private AS BGP routes advertised to core with Local preference of 190 AR1 Primary Default route MED=0 CR1 CR1 To the Internet BTnet Core Default route MED=1 Fibre NTU router Network routes advertised to core with Local preference of 180 AR2 AR2 Secondary BGP Private AS CR2 Full routes to customer networks sent to core using Private AS BGP CR2 Figure 3 - BTnet 100 Resilient Extra using BGP with Private AS The solution involves setting up a BGP peering to the customer using a Private AS number and the customer routes are then advertised into BTnet via the customer router. All customers will use the AS number 65002 unless they are already using a Private AS number in which case as their existing number may be used. The ability to have multiple customers using the same Private AS number is dependent on the fact that the customer only has a default route back to BTnet. s must advertise their main aggregate routes via BOTH of their BTnet ebgp peerings. The Access / NTE router checks any received route against a prefix-list and only matching routes are accepted. The Access router then sets the local-preference associated with that route to ensure correct routing of the customer s inbound traffic. Routes received on the Primary Access router have their local-preference set to 190 and routes received on the Secondary Access / NTE router have their local-preference set to 180. To allow customers to route to the BTnet network and beyond a default route is advertised to the customer via both ebgp peerings. The default route advertisements will have their metric set so that under normal conditions the Primary link will be preferred. The default route 6 of 11
advertised from the Primary Access / NTE router will have the metric set to 0 and the default route advertised from the Secondary Access / NTE will have the metric set to 1. This results in ALL traffic normally being sent and received via the primary link. Traffic is only sent via the secondary link in the event of the primary failing. Please refer to section 6.1 of the Appendix for a suggested configuration of the customer routers. 7 of 11
5 BTnet Flex NRO Resilient Extra BTnet Flex NRO resilient extra solution is similar to the standard Resilient Extra solution as described in Section 3. However, as no NTEs are provided with this service, the resilient extra routing is controlled on the access routers rather than the NTEs. This enables BT to still have control over downstream resilience. The customer routers have a default outbound route to the access routers. This results in ALL traffic normally being sent and received via the primary link. Traffic is only sent via the secondary link in the event of the primary failing. 2x Over-specific static routes to the customer network redistributed into BGP and advertised to BTnet Core AR1 Primary CR1 CR1 To the Internet BTnet Core router Network AR2 1x Summarised static routes to the Secondary CR2 customer network redistributed into BGP and advertised to BTnet Core Figure 4 - BTnet Flex NRO Resilient Extra CR2 The customer's network address block is split into 2, and both halves are advertised via Access Router 1 (AR1) on the primary link. The undivided block is advertised from Access Router 2 (AR2) on the secondary link. This ensures that traffic is correctly routed traffic, as it will always follow the more specific static route. These routes are re-distributed into BGP on the Access Routers. 8 of 11
6 Appendix A - Low level configuration Note: The configuration shown below is a suggested configuration only, and the exact configuration required will be specific to the router type(s) used by the customer. 6.1 BTnet100 Resilient Extra 6.1.1 Router Configuration This section describes a suggested configuration to add resilient extra to BTnet100 customer routers. Please note, this is in addition to the configuration required to support the standard service. The customer router must peer with the Access router it connects to. The customer router receives only a default route from the access router, rather than taking full Internet routes, as this is not required to operate the service, and can create unnecessary loads on router processor. Primary Router Peering Configuration router bgp <65002 or 's Private AS Number> no synchronization bgp deterministic-med neighbor <Access Router Sub-Interface Address> send-community neighbor <Access Router Sub-Interface Address> remote-as 2856 neighbor <Access Router Sub-Interface Address> timers 10 30 neighbor <Access Router Sub-Interface Address> prefix-list default-only in no auto-summary! The default-only prefix-list should be configured to deny all routes except a default route. The access router will normally be configured not to send any routes, this filter is an additional safeguard to prevent the router running out of memory in the event that the access router is misconfigured. ip prefix-list default-only permit 0.0.0.0/0 To advertise each route, the customer must add an appropriate network statement in the BGP configuration. If the customer has multiple address blocks, then an additional network statement will be required for each one. router bgp <65002 or 's Private AS Number> network <Full Network> mask <Full Mask> There are a number of options available to the customer to control outbound routing. By default, the customer network should send traffic to the customer router attached to the primary link. This is achieved by configuring it as the default gateway from the customer network. The following configuration can be added to the primary customer router to direct outbound traffic destined for the primary router to go via the secondary router in the event of a failure of the primary link. ip route 0.0.0.0 0.0.0.0 <IP Address of Secondary Router LAN Port> 250 9 of 11
This sets a default route, but has a higher cost than the default route received by the primary customer router from the BT core network. A secondary gateway (of the Secondary customer router) can be added in some operating system network configurations such as Microsoft Windows, so that the secondary route is taken in the event of the primary customer router having a complete failure or failure of BGP routing. Secondary Router Peering Configuration router bgp <65002 or 's Private AS Number> no synchronization bgp deterministic-med neighbor <Access Router Sub-Interface Address> send-community neighbor <Access Router Sub-Interface Address> remote-as 2856 neighbor <Access Router Sub-Interface Address> timers 10 30 neighbor <Access Router Sub-Interface Address> prefix-list default-only in no auto-summary! The default-only prefix-list should be configured to deny all routes except a default route. The access router will normally be configured not to send any routes, this filter is an additional safeguard to prevent the router running out of memory in the event that the access router is misconfigured. ip prefix-list default-only permit 0.0.0.0/0 To advertise each route, the customer must add an appropriate network statement in the BGP configuration. If the customer has multiple address blocks, then an additional network statement will be required for each one. router bgp <65002 or 's Private AS Number> network <Full Network> mask <Full Mask> 10 of 11
7 Document Information 7.1 Glossary Of Terms Acronym AR AS ATM BGP CR HSRP IP LAN NRO NTE OSPF PVC WAN Expansions BTnet Core Network Access Router Autonomous System [IETF] Asynchronous Transfer Mode Border Gateway Protocol [IETF] Owned Router Hot Standby Router Protocol [IETF] Internet Protocol [IETF] Local Area Network No Router Option BTnet Router on customer premises (Network Termination Equipment) Open Shortest Path First protocol [IETF] Permanent Virtual Circuit/Channel Wide Area Network --- End of document --- 11 of 11