MULTI-STAGE CLOS ARCHITECTURES

Transcription

1 MULTI-STAG CLOS ARCHITCTURS Doug Hanks SR. DATA CNTR ARCHITCT CDBU JUNIPR NTWORKS

2 BIG PICTUR Corporate Office WAN Smart Phones Cell Tower Branch Office Data Centers

3 DATA CNTR ARCHITCTUR WAN dge Services Core Data Center Switching Block Aggregatio n Storage Access Compute

4 WHY MULTI-STAG NTWORKS? Scale No need for Layer 2 between racks Typically non-virtualized and OTT services Want to leverage data center overlay technologies VMware NSX Juniper Contrail

5 LARG SCAL Access Over 10,000 Ports Layer 3

6 CHARLS CLOS Ingre ss Middl e gres s

7 SPIN AND LAF Middle Spine Spine Leaf Leaf Leaf Leaf Ingress Scale gress

8 CLOS RQUIRMNTS Requirement OSPF IS-IS BGP Advertise prefixes Yes Yes Yes Scale Limited Limited Yes Traffic ngineering Limited Limited Yes Traffic Tagging Limited Limited Yes Multi-Vendor Stability Yes Yes ven more so

9 COMPONNTS ebgp ebgp ASN Spine 1 ASN Spine 2 ASN Leaf 3 ASN Leaf 4 ASN Leaf 5 ASN Leaf 6

10 DTAILS BGP ASN assignments IP address scheme P2P network assignments P2P address assignments Server-facing network assignments BGP export policies

11 XAMPL S S S S L L L L L L L L L L L L L L L L 4 spines + 16 leaves 64 links

12 PROBLM AT SCAL P2P Links P2P Link Scale Number of Leaves Links

13 SOLUTION

14 USAG

15 XAMPL localhost:sl dhanks$./sl -- spines=4 \ --spine-speed=40 \ --spine-ports=32 \ --leaf-down-speed=10 \ --leaf-down-ports=48 \ --leaf-up-ports=4 \ --leaf-up-speed=40 \ --base-prefix= /16 \ --p2p-mask=31 \ --autonomous-system=65000

16 JSON OUTPUT "leaf" : { "0" : { "as" : 65001, "name" : "L0", "port" : { "0" : { "address" : " /31", "ifd" : "et-0/0/0" }, "1" : { "address" : " /31", "ifd" : "et-0/0/1" }, "2" : { "address" : " /31", "ifd" : "et-0/0/2" }, "3" : { "address" : " /31", "ifd" : "et-0/0/3" } } }, "1" : { "as" : 65002, "name" : "L1", "port" : { "0" : { "address" : " /31", "ifd" : "et-0/0/0" }, "1" : { "address" : " /31", "ifd" : "et-0/0/1" }, "2" : { "address" : " /31", "ifd" : "et-0/0/2" }, "3" : { "address" : " /31", "ifd" : "et-0/0/3" } } },

17 DOWNLOAD

18 MULTI-STAG FUN WITH CLOS

19 MULTI-STAG CLOS XAMPL S S S S L L L L L L L L L L L L L L L L

20 MULTI-STAG CLOS XAMPL vspine vspine vspine S S S S S S S S S S S S L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L L

21 MULTI-STAG CLOS ROLS Spine Leaf Access Backplane of multi-stage CLOS vspine Always 1:1 Over- Subscription Combination of Spine and Leaf Provide BGP Route Acts as a logical switch Reflection Virtual peering point for Access Peers via ibgp to Leaf Over-Subscription dependent on nodes NNI of multi-stage CLOS the Spine and Leaf roles Variable Over-Subscription Peers via ibgp to Spine Single BGP Autonomous System nodes Number Peers via ebgp to Access Peers via ebgp to access switches nodes Provide access to end-points such as compute and storage Typically 3:1 Over-Subscription in NT and SP environments, and 1:1 for HPC Peers via ebgp to vspine nodes Provides L3 gateway services to end-points Provides Link Aggregation to end-points

22 MULTI-STAG CLOS BNFITS Massive scale over 73,000x10G access ports High performance variable over-subscription 1:1 to N:1 Pay as you grow start small and increment 1U at a time Low latency with fixed switches Very small blast radius upon failures in the network Standards based deign supports multiple vendors Deterministic latency with a fixed spine and leaf

23 MULTI-STAG CLOS ARCHITCTUR Spine 1:1 OS vspin e1 vspin e2 vspin e3 vspin e4 vspin e5 vspin e6 vspin e7 vspin e8 512x40G 512x40G 512x40G 512x40G 512x40G 512x40G 512x40G 512x40G Interfaces Interfaces Interfaces Interfaces Interfaces Interfaces Interfaces Interfaces 1:1 OS 1:1 OS 1:1 OS 1:1 OS 1:1 OS 1:1 OS 1:1 OS 1:1 OS Leaf 1:1 OS Access 3:1 OS

24 MULTI-STAG CLOS DTAIL Spine 16 nodes per vspine vspine1 512x40G 1:1 O/S vspine2 512x40G 1:1 O/S Leaf 40G 32 nodes per vspine Access 40G 512 nodes 49,152x10G

25 VSPIN OPTIONS OVR-SUBSCRIPTION Attribute Total 40G Interfaces 3:1 CLOS 768 1:1 CLOS 512 Spines 8 16 Leaves Total Switches Point-to-Point Connections Latency 1.6µ 1.6µ Control Planes Blast Radius 1/n 1/n

26 VSPIN TOPOLOGIS OVR-SUBSCRIPTION 3:1 CLOS vspine 1:1 CLOS vspine 8 Spines 16 Spines 1x40G 1x40G 32 Leaves 32 Leaves 24x40G per Leaf 16x40G per Leaf 768x40G per vspine 3:1 Over-Subscription 512x40G per vspine 1:1 Over-Subscription

27 MULTI-STAG CLOS TOPOLOGY 1 2 Spine Leaf vspine CLOS 768x40G 3:1 OS vspine CLOS 512x40G 1:1 OS Access 3:1 OS 3:1 OS

28 MULTI-STAG CLOS SUMMARY 1 3:1 OS vspine CLOS vspine 768x40G vspine 3W/10G port 1,088 switches 73,728x10G 9:1 2 OS 2 1:1 OS vspine CLOS vspine 512x40G vspine 3.6W/10G port 896 switches 49,152x10G 3:1 2 OS

29 MULTI-STAG CLOS BGP OVRVIW BGP ASN 1 BGP ASN 11 BGP RR Cluster ibgp ebgp Spine BFD Leaf BFD Access vspine

30 MULTI-STAG CLOS BGP DTAIL Spine BGP RR ibgp Down vspine1 ASN 1 vspine2 ASN 2 BGP BGP BGP BGP RR RR RR RR ibgp ibgp Leaf ibgp Up ebgp Down ebgp Access ebgp Up ASN 11 ASN 12 ASN 13 ASN 14 ASN 15 ASN 16 ASN 17 ASN 18

31 VSPIN STRIPING A B C D F G H I J

32 SUMMARY vspine = collapsed spine + leaf Multiple vspine options ibgp controlled 3-stage Clos Physical Chassis Vendor-specific technologies/fabrics Multi-Stage Clos reduces the blast radius BGP glues it all together BFD for high availability

33