DATA CENTER FABRIC COOKBOOK

Transcription

1 Do It Yourself! DATA CENTER FABRIC COOKBOOK How to prepare something new from well known ingredients Emil Gągała

2 WHAT DOES AN IDEAL FABRIC LOOK LIKE? 2 Copyright 2011 Juniper Networks, Inc.

3 REQUIREMENTS - THE NETWORK FABRIC A Network Fabric has the. Scalability and resilience of a network 1. Any-to-any flat connectivity with fairness and full non-blocking 2. Low latency and jitter 3. No packet drops under congestion 4. Linear cost and power scaling with the number of interfaces 5. Support of virtual Layer 2 and Layer 3 networks and services 6. Modular distributed implementation that is highly reliable and scalable 7. Single logical device Performance and simplicity of a single switch 3 Copyright 2011 Juniper Networks, Inc.

4 FABRIC OF SWITCHES OR DISTRIBUTED SWITCH FABRIC Core Layer Aggregation Layer Access Layer 4 Copyright 2011 Juniper Networks, Inc.

5 FABRIC OF SWITCHES OR DISTRIBUTED SWITCH FABRIC One Network Flat, any-to-any connectivity Switch Fabric 5 Copyright 2011 Juniper Networks, Inc.

6 BUT WHAT ABOUT SCALING? Multi Tier Network Latency Scale vs. Latency Traditional design Ethernet Fabric Scale Bandwidth Scale vs. Bandwidth Fabric Servers NAS Traditional design Scale 6 Copyright 2011 Juniper Networks, Inc.

7 INGREDIENTS - THE NETWORK FABRIC 1. Control Plane Routing Engine 2. Switching Plane - Fabric 3. Forwarding Plane - I/O Modules Single device N=1 Single switch does not scale Single point of failure 7 Copyright 2011 Juniper Networks, Inc.

8 FABRIC CONTROL PLANE 10 Copyright 2011 Juniper Networks, Inc.

9 CONTROL PLANE Can we have a single Routing Engine to control all the TOR switches as line cards? Answer is NO: Key Reason: Scalability issues. Eg. On a typical router, RE distributes the complete forwarding state to all PFEs - This cannot scale to Fabric levels, with a few thousand TORs Protocol processing cannot scale with such a large number of interfaces - LACP:O(3000) sessions - ARP:O(100K 500K endpoints) Solution: Use Multiple REs Distributed and virtualized control plane 11 Copyright 2011 Juniper Networks, Inc.

10 3 BASIC FUNCTIONS ANY SWITCH HAS TO SOLVE 1. System Discovery Who are we? 2. Fabric Discovery How many ways can we send data to each other? 3. Control State Propagation How do we exchange control state between ourselves? 13 Copyright 2011 Juniper Networks, Inc.

11 THE 3 INTERNAL PROTOCOLS EXAMPLES 1. System Discovery IS-IS Based Runs on Control Plane Ethernet Network 2. Fabric Topology Discovery IS-IS Based 3. Control State Propagation Fabric Control Protocol BGP Based 14 Copyright 2011 Juniper Networks, Inc.

12 LOGICAL SYSTEM VIEW AFTER SYSTEM DISCOVERY 1. System Discovery IS-IS Based Runs on Control Plane Ethernet Network One Big LAN Node 1 Node 2 Node 128 Fabric Manager Interconnect 1 Interconnect 4 Fabric Control 1 Fabric NNG RE Control 2 (Active) NNG RE (Backup) 15 Copyright 2011 Juniper Networks, Inc.

13 WHY TO CENTRALIZE FABRIC DATA PLANE FORWARDING LOGIC? 2. Fabric Topology Discovery IS-IS Based Link 1: 4 Gbps Centralize what you can, distribute what you must A Link 2: 2 Gbps Link 3: 4 Gbps With Spanning TRILL (FabricPath) Tree With Fabric Only Both one 4Gbps link links is used are used All links are used Complex Distributed configuration equal cost constraint to make sure Spraythat weights rules out Link the 4Gbps intelligently 2 link is send used appropriate traffic on each link Effective fabric bandwidth 48 Gbps Effective fabric bandwidth : 10 Gbps 16 Copyright 2011 Juniper Networks, Inc. B Frees us from the tyranny of equal-cost shortest path routing Can use all possible paths between any two QFNodes within the fabric Traffic on each path proportional to the cost (or inverse) of each path Faster convergence times Less computation-intensive

14 FABRIC CONTROL PROTOCOL WHY BGP? Let s consider the desirable attributes of any such protocol Should have in-built scaling model Should have multi-version support needed for Partitions Should be extensible needs to carry both Mac and IP routes Should have overlapping address space support Should be hardened it is the heart of the system after all BGP fits the fill perfectly Route Reflector mechanism Standard open protocol TLV mechanism Route Distinguisher, Route Target constructs Field Proven 17 Copyright 2011 Juniper Networks, Inc.

15 ACTUAL L3-VPN APPROACH 18 Copyright 2011 Juniper Networks, Inc.

16 LEVERAGING L3-VPN APPROACH IN DCF 19 Copyright 2011 Juniper Networks, Inc.

17 FABRIC DATA PLANE 20 Copyright 2011 Juniper Networks, Inc.

18 DIFFERENT DC FABRIC ARCHITECTURES #1 Architecture L2 at access L3, ACLs, buffering in core #2 Architecture L2, FC, ACLs in access L3, FC, ACLs, buffering in core Feature Rich Core Cost Feature Rich Core Cost Minimal Edge Rich Edge Scale Scale #3 Architecture L2, L3, FC, ACLs in access No features in core Minimal Core Cost Rich Edge Scale 21 Copyright 2011 Juniper Networks, Inc.

19 KNOWING FABRIC IN BRIEF LARGE scale distributed system which acts as a single logical L2/ L3 switch Physically consists of multiple chassis Composed of An intelligent edge which makes complex forwarding decisions - TOR/ LCC A high speed but dumb core which transfers packets across the intelligent edge - Interconnect chassis 22 Copyright 2011 Juniper Networks, Inc.

20 CLOS TOPOLOGY Topology developed in 1950s for telephone switching equipment First Stage Second Stage Third Stage I/p 0 I/p 3 F1 F2 F3 O/p 0 O/p 3 I/p 4 O/p 4 I/p 7 F1 F2 F3 O/p 7 I/p 8 O/p 8 I/p 11 F1 F2 F3 O/p 11 I/p 12 O/p 12 F1 F2 F3 I/p 15 O/p Copyright 2011 Juniper Networks, Inc.

21 CLOS SWITCH OPERATION First stage: spray cells evenly Second: Provides All-to-All Connectivity Third stages: Provides non-blocking property 1st Stage F1 2nd Stage F2 3rd Stage F2 24 Copyright 2011 Juniper Networks, Inc.

22 CLOS PROPERTIES Multiple redundant paths per source destination pair. Topology is rearrangably nonblocking In a circuit-switched Clos network, a new connection can always be made if existing connections can be moved to different paths Spraying cells achieves the same effect as moving circuits Each cell may have different transition time through fabric Reorder buffer in Egress PFE to put cells back into sequence before forwarding packet Multiple links between each stage of the fabric. Graceful bandwidth degradation when a component fails 28 Copyright 2011 Juniper Networks, Inc.

23 FABRIC TOPOLOGY EXAMPLE 128 x TOR TOR # DCF #0 1 TOR # DCF #1 29 Copyright 2011 Juniper Networks, Inc.

24 YOUR DINNER 33 Copyright 2011 Juniper Networks, Inc.

25 3 years in development 1 million man hours $100s of millions invested Over 125 patents pending 34 Copyright 2011 Juniper Networks, Inc.

26