BGP Traffic Orchestration

Transcription

1 BGP Traffic Orchestration Asia Peering Forum 2017, Taipei Eric Ng 1

2 Key Peering Contacts Eric Ng Alibaba Network Architect Angela Gao Alibaba Procurement 2

3 Key Milestones of Alibaba Alibaba Group founded in Jack Ma s apartment; Alibaba.com and 1688.com (then Alibaba.com.cn) launched Aliwangwang instant messenger launched on Taobao Marketplace; Alipay launched Brands and retail platform Tmall.com launched to complement Taobao Marketplace Online group buying marketplace Juhuasuan and global consumer marketplace AliExpress launched Alibaba Group listed on the NYSE; Alipay s parent company Ant Financial Services Group formally established Online shopping website Taobao Marketplace launched Online marketing technology platform Alimama launched; Taobao Marketplace starts to monetize AliCloud founded in conjunction with Alibaba Group s 10 th anniversary celebration Cainiao Logistics launched; Alibaba Group campus relocated Global Shopping Festival generated RMB91.2 billion in GMV 3

4 Our network Infrastructure in China 34 provinces coverage 700+ Network/CDN nodes Multiple links to major ISPs 500 million + mobile user China Telecom China Unicom China Mobile Other Service Provider 4

5 Our network Global Infrastructure LDN MSK TYO SJC FRA HKG IAD DUB KL SIN 10+ Alibaba IDC 300+ Network/CDN nodes 83 millions+ Online retail customers SAO SYD 5

6 BGP Traffic Orchestration 6

7 Some facts in our network There are three major ISPs in China and some other local ISPs. We connect them in every province and region in China Adding them together will have hundreds of upstream links All our critical businesses traffic are riding on this huge network to reach our hundreds of million of Internet users in China Keep the outbound traffic local - Hot potato By default, our ASBR/IDC routers select the BGP best route learnt from the upstream providers link in the same region. 7

8 We rely on the BGP best path algorithm. BGP best path selection algorithm # Assuming the prefixes for comparison have the same length and the next-hop are reachable 1. Prefer the path with the highest local preference. 2. Prefer the path with the shortest AS Path. 3. Prefer the path with the lowest origin type. 4. Prefer the path with the lowest multi-exit discriminator (MED). 5. Prefer ebgp over ibgp paths. 6. Prefer the path with the lowest IGP metric to the BGP next hop. 7. Prefer the oldest path. 8. Prefer the route that comes from the BGP router with the lowest router ID/originator ID BGP, a path vector routing protocol, doesn t consider attributes like bandwidth, latency, cost. for path selection! 8

9 But the outcomes are Some of the IP upstream links are under utilized and not cost effective Performance to certain destinations are not as good as expected via the local IP upstream links 9

10 How can we optimize the utilization of multiple IP upstream links & improve the network performance?

11 New setup in our network New Controller/Agent deployment over the network Multiple agents deployed in each region for resilience Setup ibgp between Distributed Agents and IDC core routers/asbr/rr in the same region. Centralized Controllers Routing information collection from multiple agents Date collection from various Internal systems Conduct path simulation test Provide change recommendation Push changes to agents 11

12 In-house systems support to Controller Centralized network traffic monitoring system & Netflow Monitoring the traffic utilization of every backbone and IP transit link Feed the instant links utilization data to Controller Alinet An in-house network performance monitoring system. It monitors the network performance from various source IP distributed in our network, to all different target IPs. The target IPs are the pingable IP in different subnets, and the target IPs will be replaced by other pingable IP in our Global IP address pool from time to time, if they are no longer pingable. Asset management system Provide the exact physical location of network devices (SN, IDC, cage#, rack#, RU#) IS-IS Provide the BGP peer-address of each network device 12

13 Two key modules in Controller Path simulation module Based on the requirement set, the data collected from routers and various internal various monitoring system to find out the best path and the expected change in performance, cost and traffic utilization. Implementation module Push the optimized path change instruction to agent, and in turn reach the ASBR/IDC core routers. Engineers intervention is required. 13

14 Path simulation Requirement 14

15 Path simulation best path and performance Run the path simulation test multiple times to come out a more realistic picture 15

16 Path simulation Validation Combined with our internal route monitoring system (Not shown here) Engineers will check the simulation result before and after the implementation Based on the previous mentioned in-house traffic utilization and network performance tools 16

17 Implementation module Run the exabgp process in Agent Using the python flask to listen for prefix commands via HTTP POST calls from Controller and hooking it up to the exabgp process in Agent The individual route change by Controller can be applied to the granularity of one router. The Agent will then send the BPG update to all routers via ibgp. Advantages: Easy for deployment and maintenance Vendor neutral of the router 17

18 Example of BGP route update announced by Controller { "neighbor": { "ip": " ", "message": { "update": { "attribute": { "origin": "igp", "med": 20, "as-path": [ 3, 555, 334, ], "next-hop": "self", "local-preference": 1000, "atomic-aggregate": "", "community": [ ], "extended-community": [ ], "aggregator": "", "originator-id": "" }, "announce": [ " /24", " /24", " /24", " /24", " /24" ] } }, "asn": { "peer": "65533", "local": "65500" }, "address": { "peer": " ", "local": " " } }, "time": , "type": "update" } 18

19 Current status and next step Current status We have implemented the first phase of this BGP traffic orchestration in our network in China lately. Next steps Enhance the automation mechanism of the whole path simulation and implementation process for controller. Implement BMP to collect more information from ISP BGP Monitoring Protocol (RFC 7854) //BMP provides access to the Adj-RIB-In of a peer on an ongoing basis and a periodic dump of certain statistics the monitoring station can use for further analysis. The Adj-RIBs-In contains unprocessed routing information that has been advertised to the local BGP speaker by its peers." This is also referred to as the pre-policy Adj-RIB-In in this document.// 19

20 Peering Policy Alibaba ASN (Global ASN) Open peering policy We ve connected to major IXPs, like BBIX, HKIX, Equinix EIE, Megaport, DE-CIX, IX-Australia and we will connect more in coming future We announce regional routes in different interconnection locations, so it is recommended to peer with us in all locations. Please register your ASN in peeringdb, welcome to peer with Alibaba 20

21 21