Cloud Thinking in the Enterprise

Transcription

1 Cloud Thinking in the Enterprise Sean Flack CCIE# 24438

2 Traditional Datacenter Design OSPF/BGP Public Internet Edge LB FW Core L3 1 2 OSPF Distribution Layer L2/L3 1 2 LB LB n-1 n STP, VRRP TOR L2 1 m n-m+1 n Servers in a Rack 1 m n-m+1 n Receiver

3 Traditional Datacenter Design Properties Large Layer L2 Domain Multiple Control Protocols: OSPF, BGP, VRRP, STP,.. Oversubscription: 10:1 or even 50:1 is not unheard of. Optional: Multiple Admin Domains enabled using VRFs/VLANs and Firewalls. Multicast may be used in some environments Receiver

4 Traditional Datacenter Design Pros Simple from the host standpoint, all ports are virtually the same Cons Does not scale beyond a few dozen racks (L2) Complex! A cacophony of control protocols Layer-2 doesn t support ping and traceroute Low Availability: availability model of 1+1 Lower Performing: High oversusbcription No multipath

5 Traditional Datacenter Design Primary Problem The design is complex and as such doesn t scale

6 Desired Properties of a New Design Simple High Performing Cost Savings ($/bit) Virtually Infinite Scale Automation

7 Big Ideas in Networking Simple -> Layer 3 High Performing > Wide not Tall Cost Savings ($/bit) -> Merchant Silicon Large Scale -> Clos Topology Automation -> Software Architecture

8 Moore s Law Merchant Silicon # of transistors on a chip double every 2 years! Million-fold advance in 30 years! Billion-fold advance in 45 years! Moore s Law Applied to Networking Traditional Network Vendor ASICs: 4x10G per chip! Merchant Silicon ASICs: 128x10G or 32x40G per chip!

9 Merchant Single Switch Silicon Roadmap Technology 130 nm 65nm 40 nm 28 nm 10G ports ports Throughput 360MPPS 960MPPS 2 BPPS 4 BPPS Buffer Size 2 MB 8 MB 12 MB 16 MB Table Size 16K 128K 256K 256K Availability Improvement N/A 3X/3Y 2X/2Y 2X/2Y 9

10 Merchant Silicon and Moore s Law # of 10G Ports Millions of Transistors

11 Network Topology & Scale Ring 36 network ports, 6 switches (Scales, but Blocking) Star 56 network ports, 9 switches (Doesn't Scale, Blocking, SPOF) Tree (Scales to an extent, Blocking, SPOF) Fully Connected Network aka Full Mesh 18 network ports, 6 switches (Non-blocking, bug doesn't Scale) 3D Cube 40 network ports, 8 switches (Doesn't Scale, Blocking & Non-Minimal Routing) n = 8

12 Network Topology & Scale 4D Cube (Hypercube) 64 network ports, 16 switches (Still doesn't Scale, Blocking & Non-Minimal Routing) Planar Mesh with wraparound (aka Cylinder) 80 network ports, 16 switches (Scales, Blocking and not equi-distant) 2D ButterFly 32 network ports, 8 switches (Scales, Non-Blocking, Non-Minimal Routing) Flattened 2D ButterFly 32 network ports, 4 switches (Scales, Non-Blocking, Non-Minimal Routing) n = 8

13 Network Topology & Scale host ports Dragonfly Network 98 network ports, 28 switches (Scales, but blocking & non-minimal Routing) 2D Torus 64 network ports, 16 switches (Scales, but blocking & non-minimal Routing) 3D Torus 96 network ports, 32 switches (Scales, but blocking & non-minimal Routing) 2 Tier Clos 32 network ports, 12 switches (Non-Blocking, Scalable and Minimal Routing) n = 8

14 Network Topology & Scale Common Network Topologies include: Bus, Ring, Tree, Star, Mesh, Toroid, Dragonfly, Bufferfly, Crossbar, and Clos

15 Network Topology & Scale Only the following can be built as non-blocking: Crossbar (full-mesh), Flattened Butterfly, and Clos Crossbar (full-mesh) is meant for network in a box and is a single point of failure Butterfly requires non-minimal routing, thus doesn t work with IP routing protocols Clos is the only game in town

16 What is special about the Clos Topology? You can build non-blocking networks You can incrementally scale When using Single ASIC Merchant Silicon switches, it can be very cost effective Given how regular the network is, design and operations is simpler You don t have to use chassis, typically 1RU or 2RU switches.

17 Some Clos Formulas If the n is the number of ports per switch Then, in a Two-Tier Clos Network #10G ports towards TOR, N = n 2 /2 # of Switches, S = 3n/2 And, in a Three-Tier Clos Network #10G ports towards TOR, N = n 3 /4 # of Switches, S = 5n 2 /4

18 Two Tier Clos Network Tier-2 (Spine) 1 2 n-1 n OSPF/eBGP Tier-1 (Leaf) 1 2 n-1 n OSPF/eBGP TOR 1 m Servers

19 Two Tier Clos Network Assuming 3:1 oversubscription at the TOR (Typical Enterprise Level) Using 64x10G switches everywhere (n = 64): N = 2,048 S = 96 # of 10G Hosts, H = ~6k Using 128x10G switches everywhere (n = 128): N = 8,192 S = 192 # of 10G Hosts, H = ~24k

20 Three Tier Clos Network Tier-3 (Datacenter Spine) (n-1)/2 n/2 n/2 1 2 (n-1)/2 n/2 OSPF/eBGP Tier-2 (Spine) 1 2 (n-1)/2 n/2 1 2 (n-1)/2 n OSPF/eBGP Tier-1 (Leaf) 1 2 (n-1)/2 n/2 1 2 (n-1)/2 n 1 n OSPF/eBGP TOR 1 m 1 m Servers

21 Three Tier Clos Network Assuming 3:1 oversubscription at the TOR Using 64x10G switches everywhere (n = 64): N = 65,536 S = 5120 # of 10G Hosts, H = ~192k Using 128x10G switches everywhere (n = 128): N = 524,288 S = 20,480 # of 10G Hosts, H = ~1.5M

22 Clos Network Routing Protocols Careful consideration must be taken when choosing the routing protocol Link-state protocol (OSPF/IS-IS) nodes must retain full link-state data and are very chatty. Goal of leaf-spine is deterministic performance between two leaf nodes. A failure could lead to leaf-transit paths.

23 ebgp as the Leaf-Spine Protocol Spine nodes configured as a single AS. Each leaf node assigned it s own AS number. Allows 1023 AS numbers to be assigned ( )

24 ebgp as the Leaf-Spine Protocol Prevents any leaf from becoming a transit node due to BGP loop prevention mechanisms. Flooding dramatically reduced BGP doesn t expire routing state, speakers only propagate calculated best-paths. Added Benefits Strong routing policy framework Low impact upgrades of spine devices Drain a spine node by employing high MEDs before the upgrade.

25 Equal Cost Multi Pathing (ECMP) ECMP ensures effective use of all available bandwidth Provides failover and resiliency when any individual node or link fails Traffic balanced on a per-flow basis. SRC/DST IP address, SRC/DST UDP or TCP ports

26 Considerations for Clos Network Designs Choice of Routing Protocols BGP is preferred (Scales well) Packet Buffers Merchant Silicon typically implies smaller buffers (Typical 12MB per 128 ports) Become important if you have higher oversubscription or speed mismatches There is merchant silicon offering large buffers Dune Networks, Broadcom Arad (128MB per port!) Network Virtualization VxLAN and NVGRE are two ways of achieving this Yes, you can still do VLANs

27 Considerations for Clos Network Designs Cabling Lots of cables, cabling plan is important Consider 40G and 100G but do consider speed mismatches Path Monitoring How do you find a faulty link (needle in a haystack) Provide active probing of links

28 Side Effects of such a Network Design Simplicity Regular Topology Lends well to automation Lots of Cheap Bandwidth Don t have to worry so much about QoS Capacity Planning becomes easier

29 Thank-You