Internet Peering. Why, How, Where,

Transcription

1 Internet Peering Why, How, Where,

2 Customer s Expectation 2

3 Or maybe this 3

4 IP Transit Provide access to The Internet Requires a circuit to an upstream ISP Could be local (domestic) or international Submarine circuits are fixed capacity, not tied to usage Also requires service from the upstream ISP Billing is based on usage, typically 95th percentile Repeat to get the level of redundancy required Two circuits to the same upstream ISP Circuits to two, or more, upstream ISPs

5 But it s really just 5

6 Until this happens 6

7 Or this 2 core1-vl400.vcc.kidanet.com.fj ( ) ms ms ms ( ) ms ms ms ( ) ms ms ms 5 gi rcr21.b sjc01.atlas.cogentco.com ( ) ms ms ms 6 be2063.ccr21.sjc01.atlas.cogentco.com ( ) ms be2095.ccr22.sjc01.atlas.cogentco.com ( ) ms be2063.ccr21.sjc01.atlas.cogentco.com ( ) ms 7 be3144.ccr41.sjc03.atlas.cogentco.com ( ) ms ms be3142.ccr41.sjc03.atlas.cogentco.com ( ) ms 8 zayo.sjc03.atlas.cogentco.com ( ) ms ms ms 9 ae16.cr2.sjc2.us.zip.zayo.com ( ) ms ms ms 10 ae27.cs2.sjc2.us.eth.zayo.com ( ) ms ms ms 11 ae3.cs2.sea1.us.eth.zayo.com ( ) ms ms ms 12 ae28.mpr1.sea1.us.zip.zayo.com ( ) ms ms ms i223.above.net ( ) ms ms ms 14 xe pe2.brwy.nsw.aarnet.net.au ( ) ms ms ms 15 ae9.bb1.a.syd.aarnet.net.au ( ) ms ms ms 16 ge bb1.a.suv.aarnet.net.au ( ) ms ms 7

8 What s wrong with this picture? Fintel customer in Suva Accessing content at the University of the South Pacific in Suva Packet travels > 25,000km Physical distance < 10km Adding latency Possibly jitter too Using expensive submarine capacity

9 Interconnection (aka Peering) Connection to a peer network Exchange of traffic to customers of each peer Requires a circuit to the peer (or to an Internet Exchange) Fixed cost based on capacity of the link May also require a cross connect in a data centre Could be fixed cost or more likely monthly recurring fee Traffic is settlement free Cost is the same if zero bytes exchanged or link saturated Don t saturate the link, customers will be grumpy J

10 We compete, why interconnect? International Connections If satellite, RTT is around 550ms per hop Compared to local traffic < 10ms round trip International bandwidth Costs significantly more than domestic bandwidth Don t congest it with local traffic Wastes money Harms overall performance (end-user experience)

11 Private Interconnect Autonomous System 334 border border ISP B ISP A Autonomous System 99

12 Interconnection (aka Peering) Local (loop) connections Can t be in a customer/transit relationship Sharing customer & infrastructure routes only Routes that generate revenue for you Share costs Two circuits, pay for one each 12

13 Results of Peering Both save money Local traffic stays local Better performance, better QoS, Expensive international bandwidth available for actual international traffic Everyone is happy (except submarine cable and satellite owners) 13

14 Scaling peering What happens when new ISPs enter the equation? Just repeat the process? Private peering means that each ISP has to buy circuits to every other peer (perhaps 2 for redundancy) For (n) peers each peer needs (n-1) half circuits Eg 5 peers => 4 half circuits 14

15 Why an Internet exchange Point (IXP) Private peering relies on just the two parties making best use of the circuit by building dedicated circuits to each peer n-1 peers => (n-1)/2 circuits With an IXP: Every participant has to buy just one whole circuit From their premises to the IXP fabric Improve latency performance between peers where traffic volume wouldn t justify a dedicated circuit Maximises the opportunity to fill the circuit Peak traffic may not be the same across all peers

16 Internet exchange Point (IXP) Need a location or facility that ISPs can access and can connect to each other over a common shared media Eg: Ethernet switch Should be a NEUTRAL venue Needs to have multiple telco circuit providers and/or allow any licenced provider to install services Needs controlled environment & access 16

17 Internet exchange Points Variety of shapes and sizes Commercial Community Tbps to Mbps Single location or Metropolitan Area scoped Purely a traffic exchange Value added services Layer 2 exchange point Ethernet Switches (100Gbps/10Gbps/1Gbps/100Mbps) 17

18 Internet exchange Point ISP1 ISP4 ISP2 IXP ISP5 ISP3 ISP6 Border routers in different Autonomous Systems 18

19 Layer 2 Exchange ISP 6 ISP 5 ISP 4 IXP Services: Root & TLD DNS, Routing Registry Looking Glass, etc Ethernet Switch IXP Management Network ISP 1 ISP 2 ISP 3 19

20 Layer 2 Exchange ISP 6 ISP 5 ISP 4 IXP Services: Root & TLD DNS, Routing Registry Looking Glass, etc Ethernet Switches IXP Management Network ISP 1 ISP 2 ISP 3 20

21 Layer 2 Exchange Two switches for redundancy ISPs use dual routers for redundancy or loadsharing Offer services for the common good Internet portals and search engines DNS Root & TLDs, NTP servers Routing Registry and Looking Glass 21

22 Layer 2 Exchange Requires neutral IXP management Usually funded equally by IXP participants 24x7 cover, support, value add services Secure and neutral location Configuration Private address space if non-transit and no value add services Otherwise public IPv4 (/24) and IPv6 (/48, /56, /64) ISPs require AS, basic IXP does not 22

23 Layer 2 Exchange Network Security Considerations LAN switch needs to be securely configured Management routers require AAA authentication, vty security IXP services must be behind router(s) with strong filters 23

24

25 Defining some terms

26 Types of Peering Private Peering Bi-lateral Peering Multi-lateral Peering 26

27 Private Peering Dedicated circuit between two peers Can use a cross connect within a data centre Or via dark fibre, telco circuit, microwave, Used where traffic levels high between two peers Expensive, cost shared between only two parties Often in pairs; each peer pays for one But ultimate in control 27

28 Bi-lateral Peering Uses an Ethernet switch at an Internet Exchange Single cross connect to the switch Peer can be remote (e.g. using Metro-Ethernet) Dedicated BGP peering between two peers Relies on the IXP to manage the switch Bandwidth shared by multiple peering relationships But direct relationship between the two peers More control (granularity) If bad things happen can turn down BGP on one peer 28

29 Multi-lateral Peering (MLPA) Uses an Ethernet switch at an Internet Exchange Single cross connect to the switch Single BGP peering session to a route server Easiest to setup, only one session Automatically peer with everyone else Reliant on IXP for both switch and route server Relationship is with the IXP Lesser control (granularity) If a peer has a problem less options to workaround 29

30 Types of Peering Policy Open Selective Restrictive 30

31 Open Peering Have a pulse peering Will peer with anyone Typically bi-lateral or multi-lateral at an existing facility Negligible additional cost so why not? Typically content providers have open peering policy 31

32 Selective Peering Conditional peering Ex: at an IXP, will ONLY peer bilaterally and NOT with the RS Some negotiation may be necessary May have some rules that peers must fulfil volumes, ratios, number of multiple connects May only peer outside of primary market 32

33 Restrictive Peering Rules! Has a (written) policy that defines if they will peer Often with rules, which are set so that they don t peer Often involves a minimum level of traffic Could require a test peering to check conformance Also can include a ratio in/out traffic levels 33

34

35 Controlling costs

36 Cost tied to circuit size (not byte count) Peering is typically settlement free No charge for the traffic exchanged Cost to peer Router interface Circuit to the peering fabric Charges imposed by the IXP All fixed, either capital expenditure or monthly recurring fee

37 Choosing a IXP Some markets have more than one Even if there is only one IXP it might appear in multiple locations E.g. LINX is built on two rings through multiple data centres across London Best location might be dictated by availability of IPLC, transit, or other factors

38 Which IXP? How many routes are available? How many other operators/providers are at the IX? What is the traffic to and from these destinations, and how much will it reduce the transit cost? What is the cost of co-lo space? Availability of power, type of cabinet, What is the cost of a circuit to the location? If similar to transit costs are you getting a benefit? What is the cost of remote-hands? For maintenance purposes to avoid serious outages 38

39 Remote locations If building to a remote location Make sure remote hands work at times when it s important to you Their 9-5 is not normally your office hours Check the skill set of the remote hands Maybe engage a local consultant to help

40

41 Worked Example Single International Transit versus Local IXP + Regional IXP + Transit

42 Worked Example ISP A is local access provider Some business customers (around 200 fixed links) Some co-located content provision (datacentre with 100 servers) Some consumers on broadband (5000 DSL/Cable/Wireless) They have a single transit provider Connect with a 16Mbps international leased link to their transit s PoP Transit link is highly congested 42

43 Worked Example (2) There are two other ISPs serving the same locality There is no interconnection between any of the three ISPs Local traffic (between all 3 ISPs) is traversing International connections Course of action for our ISP: Work to establish local IXP Establish presence at overseas co-location First Step Assess local versus international traffic ratio Use NetFlow on border router connecting to transit provider 43

44 Worked Example (3) Local/Non-local traffic ratio Local = traffic going to other two ISPs Non-local = traffic going elsewhere Example: balance is 30:70 Of 16Mbps, that means 5Mbps could stay in country and not congest International circuit 16Mbps transit costs $50 per Mbps per month local traffic charges = $250 per month, or $3000 per year for local traffic Circuit costs $100k per year => $30k is spent on local traffic Total is $33k per year for local traffic 44

45 Worked Example (4) IXP cost: Simple 8 port 10/100 managed switch plus co-lo space over 3 years could be around US$30k total => $3k per year per ISP One router to handle 5Mbps (e.g. 2801) would be around $3k (good for 3 years) One local 10Mbps circuit from ISP location to IXP location would be around $5k per year, no traffic charges Per ISP total: $11k Somewhat cheaper than $33k Business case for local peering is straightforward - $22k saving per annum 45

46 Worked Example (5) After IXP establishment 5Mbps removed from International link Leaving 5Mbps for more International traffic and that fills the link within weeks of the local traffic being removed Next step is to assess transit charges and optimise costs ISPs visits several major regional IXPs Assess routes available Compares routes available with traffic generated by those routes from its NetFlow data Discovers that 30% of traffic would transfer to one IXP via peering 46

47 Example: South Asian LINX Date: May 2013 Data: Route Server plus bilateral peering offers 70k prefixes IXP traffic averages 247Mbps/45Mbps Transit traffic averages 44Mbps/4Mbps Analysis: 85% of inbound traffic comes from 70k prefixes available by peering 15% of inbound traffic comes from remaining 380k prefixes from transit provider 47

48 Example: South Asian HKIX Date: May 2013 Data: Route Server plus bilateral peering offers 67k prefixes IXP traffic is 159Mbps/20Mbps Transit traffic is 108Mbps/50Mbps Analysis: 60% of inbound traffic comes from 67k prefixes available by peering 40% of inbound traffic comes from remaining 383k prefixes from transit provider 48

49 Example: South Asian ISP Summary: Traffic by Peering: 406Mbps/65Mbps Traffic by Transit: 152Mbps/54Mbps 73% of incoming traffic is by peering 55% of outbound traffic is by peering 49

50 Example: South Asian ISP Router at remote co-lo Benefits: can select peers, easy to swap transit providers Costs: co-lo space and remote hands Overall advantage: Can control what goes on the expensive connectivity back to home 50

51 Value propositions Peering at a local IXP Reduces latency & transit costs for local traffic Improves Internet quality perception Participating at a Regional IXP A means of offsetting transit costs Managing connection back to home network Improving Internet Quality perception for customers 51

52

53 IXP Design Considerations

54 IXP Capital Expenses Managed Ethernet Switch(es) mandatory Route Server(s) desirable Router(s) and server(s) to support other IXP services optional

55 IXP Operating Expenses Data Center space, including Rack Space Appropriate Electricity (AC or DC, with UPS/genset) Cooling CCTV Camera and other physical security measures Cross-connects Administration and Technical support Equipment maintenance Internet transit for other IXP services optional

56 IXP Organization Model Possible Models include: Donation / sponsorship Cost sharing by participants Fixed fee, Port speed based fee, Membership based organization Volunteers vs Outsourcing vs Staff Or any combinations of above Neutrality is the most important, even at the beginning Long term sustainability is the hard part so sustainable financial model has to be established in due course

57 Neutral Location as Starting Point May choose one of the followings as starting point: University Technology Park Carrier Neutral Data Center Government Data Center Submarine Cable Station Having multiple carrier options is the most important Should maintain neutrality continuously Expansion to multiple sites can be done gradually, coupled with growth

58 Requirements of IXP Site Proximity to the networks of the potential members / participants Options, availability, capacity and reliability of fiber carriers Support for additional fiber carriers 24x7 access for IXP authorized support personnel

59 Requirements of IXP Site Availability and stability of electricity supply, including UPS and backup power generator Do you need DC power? Sufficient cooling facilities Good physical security 24x7 surveillance and access control Availability of additional rack space for future growth

60 General Guidelines Governance Organisation-wise, multi-stakeholder bottom-up approach is proven to be the best approach for maximum acceptance of the community while government support is critical IXP should be as inclusive as possible in order to provide maximum benefits to the whole community which it serves Should be fair and consistent to every participant Should be open and transparent as much as possible

61 General Guidelines - Geography IXP should NOT be expanded beyond a metro area to avoid competing with participants and to maintain neutrality Should start with the city with the largest concentration of ISPs first and gradually set up separate infrastructure in other cities if needed

62 General Guidelines - Policies AUPs Acceptable Use Policy Minimal rules for connection Nobody is obliged to peer Agreements left to ISPs, not mandated by IXP

63 General Guidance - Fees Rely on donations Cost recovery Fixed membership fees Per port fees Once off fee or recurring Commercial

64 Services Offered Services offered should not compete with member ISPs e.g. web hosting at an IXP is a bad idea unless all members agree to it IXP operations should make performance and throughput statistics available to members Use tools such as LibreNMS (IXP Manager) to produce IX throughput graphs for member (or public) information

65 Services to Offer cctld DNS the country IXP could host the country s top level DNS e.g. SE. TLD is hosted at Netnod IXes in Sweden Offer back up of other country cctld DNS Root server Anycast instances of root servers (I.root-servers.net, F.rootservers.net etc are present at many IXes) gtld DNS.com &.net are provided by Verisign at many IXes

66 Services to Offer Route Server Helps scale IXes by providing easier BGP configuration operation for participants Technical detail covered later on Looking Glass One way of making the Route Server routes available for global view (e.g. Public or members-only access

67 Services to Offer Content Redistribution/Caching For example, Akamised update distribution service Network Time Protocol Locate a stratum 1 time source (GPS receiver, atomic clock, etc) at IXP Routing Registry Used to register the routing policy of the IXP membership (more later)

68

69 How to build an IXP

70 How to setup an IXP? The IXP core is an Ethernet switch Managed switch with reasonable security features Has superseded all other types of network devices for an IXP From the cheapest and smallest 12 or 24 port 100M/1G switch To the largest switches now handling 10GE, 40GE, 100GE interfaces 70

71 How to setup an IXP? Each ISP participating in the IXP brings a router to the IXP location Note: ISPs may connect directly to the IXP (availability of fibre connection) instead of a dedicated router at the IXP Router needs: One Ethernet port to connect to IXP switch One WAN port to connect to the WAN media leading back to the ISP backbone To be able to run BGP 71

72 How to setup an IXP? IXP switch located in one dedicated equipment rack Also includes other IXP operational equipment Routers from participant ISPs located in adjacent rack(s) Copper (UTP) connections made for 10/100Mbps or 1Gbps connections Fibre used for 10Gbps and 40Gbps 72

73 Peering Each participant needs to run BGP They need their own AS number Public ASN, NOT private ASN Each participant configures external BGP directly with the other participants in the IXP Peering with all participants or Peering with a subset of participants 73

74 Routing Advice ISP border routers at the IXP should NOT advertise default route or the full Internet routing table Carrying default or full table means that this router and the ISP network is open to abuse by other IXP members Correct configuration is only to carry routes offered to IXP peers on the IXP peering router Note: Some ISPs offer transit across IX fabrics They do so at their own risk see above 74

75 Routing (more) ISP border routers at the IXP should not be configured to carry the IXP LAN network within the IGP or ibgp Use next-hop-self BGP concept Don t generate ISP prefix aggregates on IXP peering router If connection from backbone to IXP router goes down, normal BGP failover will then be successful 75

76 Address Space Some IXPs use private addresses for the IX LAN Public address space means IXP network could be leaked to Internet which may be undesirable Because most ISPs filter RFC1918 address space, this avoids the problem Some IXPs use public addresses for the IX LAN Address space available from the RIRs IXP terms of participation often forbid the IX LAN to be carried in the ISP member backbone Does produce documentation for traceroute 76

77 APNIC Policy on IXP Address Space The End-User Assignments policy caters for IXP s Public Address space under IXP Address Assignment It requires that IXP have minimum 3 ISPs connected and have clear and open policy for joining The minimum IXP Assignment is /24 of IPv4 and /48 for IPv6 77

78 Hardware Ethernet switch needs to be managed Unmanaged switch means an unmanaged IXP Insist that IXP participants bring their own router moves buffering problem off the IXP Avoid spanning tree and other L2 security issues Run port-security (MAC filtering) to protect the IX security of the ISP connection is responsibility of the ISP, not the IXP 78

79 How to set up an IXP? The hard part with establishing an IXP is NOT the technical part, but for relevant stakeholders to come together to build a creditable governance structure for the IXP with which everyone is happy (TRUST) 79

80

81 Connecting to an IXP

82 IX Etiquette and Hygiene Connect using a layer 3 device Don t proxy ARP No CDP, RIP, EIGRP, OSPF, ISIS Don t steal default Don t leak the IX prefix to the Internet Do use consistent announcements Do register prefixes in an Internet Routing Registry 82

83 Filtering Announcements Only send infrastructure and customer routes Can use community tagging to easily identify them Filter what you accept Route filters (use Routing Registry data) AS path filters Maximum prefix count Minimum prefix size Typically a /24 for IPv4 (/48 for IPv6) May special case host routes for blackhole 83

84 PeeringDB Identifies your AS number Provides contacts for NOC Very useful when using a MLPA and need to contact peer Shows which facilities you use for peering IP numbers in use at those facilities Brief description of who you are, how to contact you, your traffic levels, type of customers, your peering policy 84

85 PeeringDB for Internet exchange Location and contact information Who is there, both on fabric and for cross connect useful for planning when building out or searching for peers 85

86 Tools to create router configuration Typically use Internet Routing Registry (IRR) data Mostly communicate with RADB, which mirrors other registry data IRRToolSet The first public tool set, current development unclear but probably stable BGPQ3 Newer tool, currently actively developed Creates filters so you will need to script using it 86

87 IRRToolSet Can create router configurations from policy defined in Routing Policy Specification Language (RPSL) Uses autnum, as-set and route objects rtconfig creates configuration file peval queries IRR data Example policy in autnum objects AS2764 & AS

88 BGPQ3 Creates AS path or route filters based on IRR data Supports a variety of formats IOS (both classic and XR), JUNOS, JSON, Bird Can also DIY format 88

89 BGPQ3 Example (IOS-XR) % bgpq3 -PXl prefixset-as38442 AS38442 P- prefix list X- IOS-XR l- name of generated entry no prefix-set prefixset-as38442 prefix-set prefixset-as /18, /22, /22, /20 end-set 89

90 Using communities for filters Set a community when you import a route from a customer or create a static (aggregate) route Use that community to control export to peers & transit Don t allow peers or transits to set it though Now when you add a prefix on a router it will automatically get exported on other routers without updating their prefix lists 90

91 Peering is not just technical A personal relationship helps Support your local NOG (Network Operator Group) If you expand to other markets try to attend their NOG or Peering Forums For Myanmar companies this might include: Apricot Asia Pacific SANOG South Asia SGNOG - Singapore RIPE Europe/Middle East NANOG North America 91

92

93 IXP Best Common Practices What could go wrong?

94 Concept Some Service Providers attempt to cash in on the reputation of IXPs Market Internet transit services as Internet Exchange Point We are exchanging packets with other ISPs, so we are an Internet Exchange Point! So-called Layer-3 Exchanges really Internet Transit Providers Router used rather than a Switch Most famous example: SingTel-IX

95 Competition Too many exchange points in one locale Competing exchanges defeats the purpose Los Angeles and Tokyo have multiple but it s a rarity Becomes expensive for ISPs to connect to all of them

96 Rules and Restrictions IXPs try to compete with their membership Offering services that ISPs would/do offer their customers IXPs run as a closed privileged club Ex: Restrictive membership criteria (closed shop) IXPs providing access to end users rather than just Service Providers IXPs interfering with ISP business decisions e.g. Mandatory Multi-Lateral Peering

97 Technical Design Errors Interconnected IXPs IXP in one location believes it should connect directly to the IXP in another location Who pays for the interconnect? How is traffic metered? Competes with the ISPs who already provide transit between the two locations (who then refuse to join IX, harming the viability of the IX) IXP spanning multiple data centres in a city work ok (e.g. LINX)

98 Technical Design Errors ISPs bridge the IXP LAN back to their offices We are poor, we can t afford a router Financial benefits of connecting to an IXP far outweigh the cost of a router In reality it allows the ISP to connect any devices to the IXP LAN with disastrous consequences for the security, integrity and reliability of the IXP

99 Routing Design Errors Route Server Mandated Mandatory peering has no history of success ISPs have no incentive to learn BGP Therefore have no incentive to understand peering relationships, peering policies, Entirely dependent on operator of RS for troubleshooting, configuration, reliability RS can t be run by committee! Route Server is to help scale peering at IXPs!

100

101 More Information

102 exchange Point Examples AMS-IX, DE-CIX and LINX in Europe Equinix, in every Equinix Data Centre? SIX in Seattle, Washington, USA SGIX in Singapore MyIX in Kuala Lumpur, Malaysia BBIX, JPIX and JPNAP in Tokyo, Japan HK-IX in Hong Kong IX Australia in Perth, Sydney, Melbourne, Brisbane All use Ethernet Switches

103 More info about IXPs Another excellent resource of IXP locations, papers, IXP statistics, etc Tele Geography: A collection of IXPs and interconnect points for ISPs Searchable database of Exchange Points, Networks & Facilities

104

105 Acknowledgement: Philip Smith Cisco Systems 105

106 Thank you