ISP Network Design. ISP Workshops

Transcription

1 ISP Network Desig ISP Workshops These materials are licesed uder the Creative Commos Attributio-NoCommercial 4.0 Iteratioal licese ( Last updated 27 th February

2 Ackowledgemets p This material origiated from the Cisco ISP/IXP Workshop Programme developed by Philip Smith & Barry Greee p Use of these materials is ecouraged as log as the source is fully ackowledged ad this otice remais i place p Bug fixes ad improvemets are welcomed Please workshop (at) bgp4all.com Philip Smith 2

3 ISP Network Desig p PoP Topologies ad Desig p Backboe Desig p Upstream Coectivity & Peerig p Addressig p Routig Protocols p Security p Out of Bad Maagemet p Operatioal Cosideratios 3

4 Poit of Presece Topologies 4

5 PoP Topologies p Core routers high speed truk coectios p Distributio routers ad Access routers high port desity p Border routers coectios to other providers p Service routers hostig ad servers p Some fuctios might be hadled by a sigle router 5

6 PoP Desig p Modular Desig p Aggregatio Services separated accordig to coectio speed customer service cotetio ratio security cosideratios 6

7 Modular PoP Desig Backboe lik to aother PoP ISP Services (DNS, Mail, News, FTP, WWW) Web Cache Other ISPs Hosted Services & Datacetre Backboe lik to aother PoP Cosumer Dial Access Network Core Cosumer cable, xdsl ad wireless Access Busiess customer aggregatio layer Chaelised circuits for leased lie circuit delivery Network Operatios Cetre GigE fibre truks for MetroE circuit delivery MetroE customer aggregatio layer 7

8 Modular Routig Protocol Desig p Modular IGP implemetatio IGP area per PoP Core routers i backboe area (Area 0/L2) Aggregatio/summarisatio where possible ito the core p Modular ibgp implemetatio BGP route reflector cluster Core routers are the route-reflectors Remaiig routers are cliets & peer with route-reflectors oly 8

9 Poit of Presece Desig 9

10 PoP Modules p Low Speed customer coectios ADSL2, Cable, Public Wireless, PSTN access Low badwidth eeds Low reveue, large umbers p Busiess customer coectios 10Mbps to 100Mbps speed rage Delivery over SDH or MetroEtheret liks Medium badwidth eeds Medium reveue, medium umbers 10

11 PoP Modules p Highbad Busiess customer coectios From 100Mbps to over 1Gbps Etheret, Fibre, or high ed SDH High badwidth eeds High reveue, low umbers 11

12 PoP Modules p PoP Core Two dedicated routers High Speed itercoect Backboe Liks ONLY Do ot touch them! p Border Network Dedicated border router to other ISPs The ISP s frot door Trasparet web cachig? Two i backboe is miimum guaratee for redudacy 12

13 PoP Modules p ISP Services DNS (cache, secodary) News (still relevat?) Mail (POP3, Relay, Ati-virus/ati-spam) WWW (server, proxy, cache) p Hosted Services/DataCetres Virtual Web, WWW (server, proxy, cache) Iformatio/Cotet Services Electroic Commerce 13

14 PoP Modules p Network Operatios Cetre Cosider primary ad backup locatios Network moitorig Statistics ad log gatherig Direct but secure access p Out of Bad Maagemet Network The ISP Network Safety Belt 14

15 PSTN & Broadbad Access Module Web Cache Telephoe Network DSLAM IP, ATM BRAS Access Network Gateway Routers Cable RAS Cable System To Core Routers Primary Rate T1/E1 Digital Modem Access Servers SSG, DHCP, TACACS+ or Radius Servers/Proxies, DNS resolver, Cotet 15

16 Busiess Customer Access Module Chaelised T1/E1 Aggregatio Edge Metro Etheret To Core Routers Direct Fibre Liks 16

17 High Speed Access Module Metro Etheret Aggregatio Edge Direct Fibre Liks To Core Routers Chaelised OC3/OC12 17

18 ISP Services Module To core routers Service Network Gateway Routers WWW cache DNS Secodar y POP3s IMAPs SMTP Host (i) SMTP Relay (out) DNS Resolver 18

19 Hosted Services Module To core routers Hosted Network Gateway Routers vla11 vla12 vla13 vla14 vla15 vla16 vla17 Customer 1 Customer 3 Customer 5 Customer 7 Customer 2 Customer 4 Customer 6 19

20 Border Module To local IXP NB: router has o default route + local AS routig table oly ISP1 ISP2 Network Border Routers To core routers 20

21 NOC Module To core routers Critical Services Module Out of Bad Maagemet Network Hosted Network Gateway Routers Firewall Corporate LAN asyc termial server NetFlow Aalyser TACACS+ server SYSLOG server Primary DNS Billig, Database ad Accoutig Systems Network Operatios Cetre Staff 21

22 Out of Bad Network Out of Bad Maagemet Network Router cosoles Termial server To the NOC NetFlow Collector NetFlow eabled routers Out of Bad Etheret 22

23 Backboe Network Desig 23

24 Backboe Desig p Routed Backboe p Switched Backboe ATM/Frame Relay core etwork Now obsolete p Poit-to-poit circuits x64k, T1/E1, T3/E3, OC3, OC12, GigE, OC48, 10GigE, OC192, OC768, 100GE p ATM/Frame Relay service from telco T3, OC3, OC12, delivery Easily upgradeable badwidth (CIR) Almost vaished i availability ow 24

25 Distributed Network Desig p PoP desig stadardised operatioal scalability ad simplicity p ISP essetial services distributed aroud backboe p NOC ad backup NOC p Redudat backboe liks 25

26 Distributed Network Desig ISP Services Customer coectios Backup Operatios Cetre POP Two Customer coectios Customer coectios ISP Services POP Three POP Oe ISP Services Exteral coectios Operatios Cetre Exteral coectios 26

27 Backboe Liks p ATM/Frame Relay Virtually disappeared due to overhead, extra equipmet, ad shared with other customers of the telco MPLS has replaced ATM & FR as the telco favourite p Leased Lie/Circuit Most popular with backboe providers IP over Optics ad Metro Etheret very commo i may parts of the world 27

28 Log Distace Backboe Liks p These usually cost more p Importat to pla for the future This meas at least two years ahead Stay i budget, stay realistic Uplaed emergecy upgrades will be disruptive without redudacy i the etwork ifrastructure 28

29 Log Distace Backboe Liks p Allow sufficiet capacity o alterative paths for failure situatios Sufficiet ca deped o the busiess strategy Sufficiet ca be as little as 20% Sufficiet is usually over 50% as this offers busiess cotiuity for customers i the case of lik failure Some busiesses choose 0% p Very short sighted, meaig they have o spare capacity at all!! 29

30 Log Distace Liks POP Two Log distace lik POP Three POP Oe Alterative/Backup Path 30

31 Metropolita Area Backboe Liks p Ted to be cheaper Circuit cocetratio Choose from multiple suppliers p Thik big More redudacy Less impact of upgrades Less impact of failures 31

32 Metropolita Area Backboe Liks POP Two Metropolita Liks POP Three POP Oe Metropolita Liks Traditioal Poit to Poit Liks 32

33 Upstream Coectivity ad Peerig 33

34 Trasits p Trasit provider is aother autoomous system which is used to provide the local etwork with access to other etworks Might be local or regioal oly But more usually the whole Iteret p Trasit providers eed to be chose wisely: Oly oe p o redudacy Too may p more difficult to load balace p o ecoomy of scale (costs more per Mbps) p hard to provide service quality p Recommedatio: at least two, o more tha three

35 Commo Mistakes p ISPs sig up with too may trasit providers Lots of small circuits (cost more per Mbps tha larger oes) Trasit rates per Mbps reduce with icreasig trasit badwidth purchased Hard to implemet reliable traffic egieerig that does t eed daily fie tuig depedig o customer activities p No diversity Chose trasit providers all reached over same satellite or same submarie cable Chose trasit providers have poor oward trasit ad peerig

36 Peers p A peer is aother autoomous system with which the local etwork has agreed to exchage locally sourced routes ad traffic p Private peer Private lik betwee two providers for the purpose of itercoectig p Public peer Iteret Exchage Poit, where providers meet ad freely decide who they will itercoect with p Recommedatio: peer as much as possible!

37 Commo Mistakes p Mistakig a trasit provider s Exchage busiess for a o-cost public peerig poit p Not workig hard to get as much peerig as possible Physically ear a peerig poit (IXP) but ot preset at it (Trasit sometimes is cheaper tha peerig!!) p Igorig/avoidig competitors because they are competitio Eve though potetially valuable peerig parter to give customers a better experiece

38 Private Itercoectio p Two service providers agree to itercoect their etworks They exchage prefixes they origiate ito the routig system (usually their aggregated address blocks) They share the cost of the ifrastructure to itercoect p Typically each payig half the cost of the lik (be it circuit, satellite, microwave, fibre, ) p Coected to their respective peerig routers Peerig routers oly carry domestic prefixes 38

39 Private Itercoectio Upstream Upstream PR PR ISP2 ISP1 p PR = peerig router Rus ibgp (iteral) ad ebgp (with peer) No default route No full BGP table Domestic prefixes oly p Peerig router used for all private itercoects 39

40 Public Itercoectio p Service provider participates i a Iteret Exchage Poit It exchages prefixes it origiates ito the routig system with the participats of the IXP It chooses who to peer with at the IXP p Bi-lateral peerig (like private itercoect) p Multi-lateral peerig (via IXP s route server) It provides the router at the IXP ad provides the coectivity from their PoP to the IXP The IXP router carries oly domestic prefixes 40

41 Public Itercoectio Upstream ISP6-PR ISP5-PR ISP4-PR IXP ISP1-PR ISP1 ISP3-PR ISP2-PR p ISP1-PR = peerig router of our ISP Rus ibgp (iteral) ad ebgp (with IXP peers) No default route No full BGP table Domestic prefixes oly p Physically located at the IXP 41

42 Public Itercoectio p The ISP s router IXP peerig router eeds careful cofiguratio: It is remote from the domestic backboe Should ot origiate ay domestic prefixes (As well as o default route, o full BGP table) Filterig of BGP aoucemets from IXP peers (i ad out) p Provisio of a secod lik to the IXP: (for redudacy or extra capacity) Usually meas istallig a secod router p p Coected to a secod switch (if the IXP has two more more switches) Itercoected with the origial router (ad part of ibgp mesh) 42

43 Public Itercoectio Upstream ISP6-PR ISP5-PR ISP1-PR2 ISP4-PR IXP ISP1-PR1 ISP1 ISP3-PR ISP2-PR p Provisio of a secod lik to the IXP meas cosiderig redudacy i the SP s backboe Two routers Two idepedet liks Separate switches (if IXP has two or more switches) 43

44 Upstream/Trasit Coectio p Two scearios: Trasit provider is i the locality p Which meas badwidth is cheap, pletiful, easy to provisio, ad easily upgraded Trasit provider is a log distace away p Over udersea cable, satellite, log-haul cross coutry fibre, etc p Each sceario has differet cosideratios which eed to be accouted for 44

45 Local Trasit Provider Trasit AR BR ISP1 p BR = ISP s Border Router Rus ibgp (iteral) ad ebgp (with trasit) Either receives default route or the full BGP table from upstream BGP policies are implemeted here (depedig o coectivity) Packet filterig is implemeted here (as required) 45

46 Distat Trasit Provider AR1 Trasit AR2 BR ISP1 p BR = ISP s Border Router Co-located i a co-lo cetre (typical) or i the upstream provider s premises Rus ibgp with rest of ISP1 backboe Rus ebgp with trasit provider router(s) Implemets BGP policies, packet filterig, etc Does ot origiate ay domestic prefixes 46

47 Distat Trasit Provider p Positioig a router close to the Trasit Provider s ifrastructure is strogly ecouraged: Log haul circuits are expesive, so the router allows the ISP to implemet appropriate filterig first Moves the bufferig problem away from the Trasit provider Remote co-lo allows the ISP to choose aother trasit provider ad migrate coectios with miimum dowtime 47

48 Distat Trasit Provider p Other poits to cosider: Does require remote hads support (Remote hads would plug or uplug cables, power cycle equipmet, replace equipmet, etc as istructed) Appropriate support cotract from equipmet vedor(s) Sesible to cosider two routers ad two log-haul liks for redudacy 48

49 Distat Trasit Provider AR1 Trasit AR2 BR1 BR2 ISP1 p Upgrade sceario: Provisio two routers Two idepedet circuits Cosider secod trasit provider ad/or turig up at a IXP 49

50 Summary p Desig cosideratios for: Private itercoects p Simple private peerig Public itercoects p Router co-lo at a IXP Local trasit provider p Simple upstream itercoect Log distace trasit provider p Router remote co-lo at datacetre or Trasit premises 50

51 Upstream Coectivity ad Peerig Case Study How Seacom chose their iteratioal peerig locatios ad trasit providers 51

52 Objective p Obtai high grade Iteret coectivity for the wholesale market i Africa to the rest of the world p Emphasis o: Reliability Itercoectivity desity Scalability 52

53 Metrics Needed i Determiig Solutio (1) p Focusig o operators that cover the destiatios mostly required by Africa i.e., Eglish-speakig (Europe, North America) p Iclude providers with good coectivity ito South America ad the Asia Pacific. p Little eed for providers who are strog i the Middle East, as demad from Africa for those regios is very, very low. 53

54 Metrics Needed i Determiig Solutio (2) p Split the operators betwee Marseille (where the SEACOM cable lads) ad Lodo (where there is good Iteret desity) To avoid outages due to backhaul failure across Europe Ad still maitai good access to the Iteret p Look at providers who are of similar size so as ot to fidget too much (or at all) with BGP tuig. p The providers eeded to support: 10Gbps ports Burstig badwidth/billig Future support for 100Gbps or N x 10Gbps 54

55 Metrics Needed i Determiig Solutio (3) p Implemet peerig at major exchage poits i Europe To off-set log term operatig costs re: upstream providers. 55

56 Implemetig Solutio p Coected to Level(3) ad GT-T (formerly Iteliquet, formerly Tiet) i Marseille p Coected to NTT ad TeliaSoera i Lodo p Peered i Lodo (LINX) p Peered i Amsterdam (AMS-IX) p BGP setup to prefer traffic beig exchaged at LINX ad AMS-IX p BGP setup to prefer traffic over the upstreams that we could ot peer away p No additioal tuig doe o either peered or trasit traffic, i.e., o prepedig, o de- aggregatio, etc. All traffic setup to flow aturally 56

57 Ed Result p 50% of traffic peered away i less tha 2x moths of peerig at LINX ad AMS-IX p 50% of traffic hadled by upstream providers p Equal traffic beig hadled by Level(3) ad GT-T i Marseille p Equal traffic beig hadled by TeliaSoera ad NTT i Lodo p Traffic distributio ratios across all the trasit providers is some 1:1:0.9:0.9 p This has bee steady state for the last 12x moths No BGP tuig has bee doe at all 57

58 Addressig 58

59 Where to get IP addresses ad AS umbers p Your upstream ISP p Africa AfriNIC p Asia ad the Pacific APNIC p North America ARIN p Lati America ad the Caribbea LACNIC p Europe ad Middle East RIPE NCC 59

60 Iteret Registry Regios 60

61 Gettig IP address space p Take part of upstream ISP s PA space or p Become a member of your Regioal Iteret Registry ad get your ow allocatio Require a pla for a year ahead Geeral policies are outlied i RFC2050, more specific details are o the idividual RIR website p There is o more IPv4 address space at IANA APNIC ad RIPE NCC are ow i their fial /8 IPv4 delegatio policy phase Limited IPv4 available IPv6 allocatios are simple to get i most RIR regios 61

62 What about RFC1918 addressig? p RFC1918 defies IPv4 addresses reserved for private Iterets Not to be used o Iteret backboes p Commoly used withi ed-user etworks NAT used to traslate from private iteral to public exteral addressig Allows the ed-user etwork to migrate ISPs without a major iteral reumberig exercise p ISPs must filter RFC1918 addressig at their etwork edge 62

63 What about RFC1918 addressig? p There is a log list of well kow problems: p Icludig: False belief it coserves address space Adverse effects o Traceroute Effects o Path MTU Discovery Uexpected iteractios with some NAT implemetatios Iteractios with edge ati-spoofig techiques Peerig usig loopbacks Adverse DNS Iteractio Serious Operatioal ad Troubleshootig issues Security Issues p false sese of security, defeatig existig security techiques 63

64 Private versus Globally Routable IP Addressig p Ifrastructure Security: ot improved by usig private addressig Still ca be attacked from iside, or from customers, or by reflectio techiques from the outside p Troubleshootig: made a order of magitude harder No Iteret view from routers Other ISPs caot distiguish betwee dow ad broke p Summary: ALWAYS use globally routable IP addressig for ISP Ifrastructure 64

65 Addressig Plas ISP Ifrastructure p Address block for router loop-back iterfaces p Address block for ifrastructure Per PoP or whole backboe Summarise betwee sites if it makes sese Allocate accordig to geuie requiremets, ot historic classful boudaries p Similar allocatio policies should be used for IPv6 as well ISPs just get a substatially larger block (relatively) so assigmets withi the backboe are easier to make 65

66 Addressig Plas Customer p Customers are assiged address space accordig to eed p Should ot be reserved or assiged o a per PoP basis ISP ibgp carries customer ets Aggregatio ot required ad usually ot desirable 66

67 Addressig Plas ISP Ifrastructure p Phase Oe / /24 Customer Address & p-t-p liks Ifrastructure Loopbacks 67

68 Addressig Plas ISP Ifrastructure p Phase Oe / /24 Customer Address & p-t-p liks Ifrastructure Loopbacks p Phase Two / /24 / Origial assigmets New Assigmets 68

69 Addressig Plas Plaig p Registries will usually allocate the ext block to be cotiguous with the first allocatio Miimum allocatio could be /21 Very likely that subsequet allocatio will make this up to a /20 So pla accordigly 69

70 Addressig Plas (cotd) p Documet ifrastructure allocatio Eases operatio, debuggig ad maagemet p Documet customer allocatio Cotaied i ibgp Eases operatio, debuggig ad maagemet Submit etwork object to RIR Database 70

71 Routig Protocols 71

72 Routig Protocols p IGP Iterior Gateway Protocol Carries ifrastructure addresses, poit-to-poit liks Examples are OSPF, IS-IS,... p EGP Exterior Gateway Protocol Carries customer prefixes ad Iteret routes Curret EGP is BGP versio 4 p No coectio betwee IGP ad EGP 72

73 Why Do We Need a IGP? p ISP backboe scalig Hierarchy Modular ifrastructure costructio Limitig scope of failure Healig of ifrastructure faults usig dyamic routig with fast covergece 73

74 Why Do We Need a EGP? p Scalig to large etwork Hierarchy Limit scope of failure p Policy Cotrol reachability to prefixes Merge separate orgaizatios Coect multiple IGPs 74

75 Iterior versus Exterior Routig Protocols p Iterior Automatic eighbour discovery Geerally trust your IGP routers Prefixes go to all IGP routers Bids routers i oe AS together p Exterior Specifically cofigured peers Coectig with outside etworks Set admiistrative boudaries Bids AS s together 75

76 Iterior versus Exterior Routig Protocols p Iterior Carries ISP ifrastructure addresses oly ISPs aim to keep the IGP small for efficiecy ad scalability p Exterior Carries customer prefixes Carries Iteret prefixes EGPs are idepedet of ISP etwork topology 76

77 Hierarchy of Routig Protocols Other ISPs BGP BGP ad OSPF/IS-IS BGP Static/BGP IXP Customers 77

78 Routig Protocols: Choosig a IGP p OSPF ad IS-IS have very similar properties Review the IS-IS vs OSPF presetatio p Which to choose? Choose which is appropriate for your operators experiece I most vedor releases, both OSPF ad IS-IS have sufficiet erd kobs to tweak the IGP s behaviour OSPF rus o IP IS-IS rus o ifrastructure, alogside IP IS-IS supports both IPv4 ad IPv6 OSPFv2 (IPv4) plus OSPFv3 (IPv6) 78

79 Routig Protocols: IGP Recommedatios p Keep the IGP routig table as small as possible If you ca cout the routers ad the poit-to-poit liks i the backboe, that total is the umber of IGP etries you should see p IGP details: Should oly have router loopbacks, backboe WAN poit-to-poit lik addresses, ad etwork addresses of ay LANs havig a IGP ruig o them Strogly recommeded to use iter-router autheticatio Use iter-area summarisatio if possible 79

80 Routig Protocols: More IGP recommedatios p To fie tue IGP table size more, cosider: Usig ip uumbered o customer poit-to-poit liks saves carryig that /30 i IGP p (If customer poit-to-poit /30 is required for moitorig purposes, the put this i ibgp) Use cotiguous addresses for backboe WAN liks i each area the summarise ito backboe area Do t summarise router loopback addresses as ibgp eeds those (for ext-hop) Use ibgp for carryig aythig which does ot cotribute to the IGP Routig process 80

81 Routig Protocols: ibgp Recommedatios p ibgp should carry everythig which does t cotribute to the IGP routig process Iteret routig table Customer assiged addresses Customer poit-to-poit liks Access etwork dyamic address pools, passive LANs, etc 81

82 Routig Protocols: More ibgp Recommedatios p Scalable ibgp features: Use eighbour autheticatio Use peer-groups to speed update process ad for cofiguratio efficiecy Use commuities for ease of filterig Use route-reflector hierarchy p Route reflector pair per PoP (overlaid clusters) 82

83 Security 83

84 Security p ISP Ifrastructure security p ISP Network security p Security is ot optioal! p ISPs eed to: Protect themselves Help protect their customers from the Iteret Protect the Iteret from their customers p The followig slides are geeral recommedatios Do more research o security before deployig ay etwork 84

85 ISP Ifrastructure Security p Router & Switch Security Use Secure Shell (SSH) for device access & maagemet p Do NOT use Telet Device maagemet access filters should oly allow NOC ad device-to-device access p Do NOT allow exteral access Use TACACS+ for user autheticatio ad authorisatio p Do NOT create user accouts o routers/switches 85

86 ISP Ifrastructure Security p Remote access For Operatios Egieers who eed access while ot i the NOC Create a SSH server host (this is all it does) p Or a Secure VPN access server Ops Egieers coect here, ad the they ca access the NOC ad etwork devices 86

87 ISP Ifrastructure Security p Other etwork devices? These probably do ot have sophisticated security techiques like routers or switches do Protect them at the LAN or poit-to-poit igress (o router) p Servers ad Services? p SNMP Protect servers o the LAN iterface o the router Cosider usig iptables &c o the servers too Apply access-list to the SNMP ports Should oly be accessible by maagemet system, ot the world 87

88 ISP Ifrastructure Security p Geeral Advice: Routers, Switches ad other etwork devices should ot be cotactable from outside the AS Achieved by blockig typical maagemet access protocols for the ifrastructure address block at the etwork perimeter p E.g. ssh, telet, http, smp, Use the ICSI Netalyser to check access levels: p Do t block everythig: BGP, traceroute ad ICMP still eed to work! 88

89 ISP Network Security p Effective filterig Protect etwork borders from traffic which should ot be o the public Iteret, for example: p LAN protocols (eg etbios) p Well kow exploit ports (used by worms ad viruses) p Drop traffic arrivig ad goig to private ad o-routable address space (IPv4 ad IPv6) Achieved by packet filters o border routers p Remote trigger blackhole filterig 89

90 ISP Network Security RTBF p Remote trigger blackhole filterig ISP NOC ijects prefixes which should ot be accessible across the AS ito the ibgp Prefixes have ext hop poitig to a blackhole address All ibgp speakig backboe routers cofigured to poit the blackhole address to the ull iterface Traffic destied to these blackhole prefixes are dropped by the first router they reach p Applicatio: Ay prefixes (icludig RFC1918) which should ot have routability across the ISP backboe 90

91 ISP Network Security RTBF p Remote trigger blackhole filterig example: Origi router: router bgp redistribute static route-map black-hole-trigger! ip route Null0 tag 66! route-map black-hole-trigger permit 10 match tag 66 set local-preferece 1000 set commuity o-export set ip ext-hop ! ibgp speakig backboe router: ip route ull0 91

92 ISP Network Security RTBF p Resultig routig table etries: gw1#sh ip bgp BGP routig table etry for /32, versio Paths: (1 available, best #1, table Default-IP-Routig-Table) Not advertised to ay peer Local from ( ) Origi IGP, metric 0, localpref 1000, valid, iteral, best Commuity: o-export gw1#sh ip route Routig etry for /32 Kow via "bgp 64509", distace 200, metric 0, type iteral Last update from :04:52 ago Routig Descriptor Blocks: * , from , 00:04:52 ago Route metric is 0, traffic share cout is 1 AS Hops 0 92

93 ISP Network Security urpf p Uicast Reverse Path Forwardig p Strogly recommeded to be used o all customer facig static iterfaces BCP 38 (tools.ietf.org/html/bcp38) Blocks all uroutable source addresses the customer may be usig Iexpesive way of filterig customer s coectio (whe compared with packet filters) p Ca be used for multihomed coectios too, but extreme care required 93

94 Aside: What is urpf? src= fa0/0 router FIB: /24 fa0/ /24 gi0/1 gi0/1 p Router compares source address of icomig packet with FIB etry If FIB etry iterface matches icomig iterface, the packet is forwarded If FIB etry iterface does ot match icomig iterface, the packet is dropped 94

95 Aside: What is urpf? src= fa0/0 router FIB: /24 fa0/ /24 gi0/1 gi0/1 p Router compares source address of icomig packet with FIB etry If FIB etry iterface matches icomig iterface, the packet is forwarded If FIB etry iterface does ot match icomig iterface, the packet is dropped 95

96 Security Summary p Implemet RTBF Iside ISP backboe Make it available to BGP customers too p They ca sed you the prefix you eed to block with a special commuity attached p You match o that commuity, ad set the ext-hop to the ull address p Implemet urpf For all static customers p Use SSH for device access p Use TACACS+ for autheticatio 96

97 Out of Bad Maagemet 97

98 Out of Bad Maagemet p Not optioal! p Allows access to etwork equipmet i times of failure p Esures quality of service to customers Miimises dowtime Miimises repair time Eases diagostics ad debuggig 98

99 Out of Bad Maagemet p OoB Example Access server: modem attached to allow NOC dial i cosole ports of all etwork equipmet coected to serial ports LAN ad/or WAN lik coects to etwork core, or via separate maagemet lik to NOC p Full remote cotrol access uder all circumstaces 99

100 Out of Bad Network Equipmet Rack Equipmet Rack Router, switch ad ISP server cosoles (Optioal) Out of bad WAN lik to other PoPs Modem access to PSTN for out of bad diali Etheret to the NOC 100

101 Out of Bad Maagemet p OoB Example Statistics gatherig: Routers are NetFlow ad syslog eabled Maagemet data is cogestio/failure sesitive Esures maagemet data itegrity i case of failure p Full remote iformatio uder all circumstaces 101

102 Test Laboratory 102

103 Test Laboratory p Desiged to look like a typical PoP Operated like a typical PoP p Used to trial ew services or ew software uder realistic coditios p Allows discovery ad fixig of potetial problems before they are itroduced to the etwork 103

104 Test Laboratory p Some ISPs dedicate equipmet to the lab p Other ISPs purchase ahead so that today s lab equipmet becomes tomorrow s PoP equipmet p Other ISPs use lab equipmet for hot spares i the evet of hardware failure 104

105 Test Laboratory p Ca t afford a test lab? Set aside oe spare router ad server to trial ew services Never ever try out ew hardware, software or services o the live etwork p Every major ISP i the US ad Europe has a test lab It s a serious cosideratio 105

106 Operatioal Cosideratios 106

107 Operatioal Cosideratios Why desig the world s best etwork whe you have ot thought about what operatioal good practices should be implemeted? 107

108 Operatioal Cosideratios Maiteace p Never work o the live etwork, o matter how trivial the modificatio may seem Establish maiteace periods which your customers are aware of p e.g. Tuesday 4-7am, Thursday 4-7am p Never do maiteace o the last workig day before the weeked Uless you wat to work all weeked cleaig up p Never do maiteace o the first workig day after the weeked Uless you wat to work all weeked preparig 108

109 Operatioal Cosideratios Support p Differetiate betwee customer support ad the Network Operatios Cetre Customer support fixes customer problems NOC deals with ad fixes backboe ad Iteret related problems p Network Egieerig team is last resort They desig the ext geeratio etwork, improve the routig desig, implemet ew services, etc They do ot ad should ot be doig support! 109

110 Operatioal Cosideratios NOC Commuicatios p NOC should kow cotact details for equivalet NOCs i upstream providers ad peers p Or cosider joiig the INOC-DBA system Voice over IP phoe system usig SIP Rus over the Iteret for more iformatio 110

111 ISP Network Desig Summary 111

112 ISP Desig Summary p KEEP IT SIMPLE & STUPID! (KISS) p Simple is elegat is scalable p Use Redudacy, Security, ad Techology to make life easier for yourself p Above all, esure quality of service for your customers 112

113 ISP Network Desig ISP Workshops 113