Advanced Computer Networks Data Center Architecture. Patrick Stuedi, Qin Yin, Timothy Roscoe Spring Semester 2015

Transcription

1 Advanced Computer Networks Data Center Architecture Patrick Stuedi, Qin Yin, Timothy Roscoe Spring Semester

2 MORE ABOUT TOPOLOGIES 2

3 Bisection Bandwidth Bisection bandwidth: Sum of the bandwidths of the minimum set of channels which, if removed, partition the network into two equal sets of nodes Bandwidth across narrowest part of the network Bisection bandwidth is important for algorithms in which all processors need to communicate with all others bisection cut not a bisection cut bisection bw= link bw bisection bw = sqrt(n) * link bw 3

4 Trees Diameter and average distance logarithmic k-ary tree, height d = log k N address specified d-vector of radix k coordinates describing path down from root Route up to common ancestor and down R = B xor A let i be position of most significant 1 in R, route up i+1 levels down in direction given by low i+1 bits of B H-tree space is O(N) with O( N) long wires Bisection BW? 4

5 Fat-Trees Fatter links (really more of them) as you go up, so bisection BW scales with N 5

6 Butterflies 16 node butterfly building block Tree with lots of roots! Switch nodes: N/2 x logn (logn levels) R = A xor B, at level i use straight edge if r i =0, otherwise cross edge Exactly one route from any source to any dest - no fault tolerance Diameter logn Bisection N/2 vs tree 1 vs mesh k (d-1) 6

7 k-ary d-cubes vs d-ary k-flies degree d N switches vs N log N switches diminishing BW per node vs constant requires locality vs little benefit to locality Can you route all permutations? 7

8 Benes network and Fat Tree 16-node Benes Network (Unidirectional) 16-node 2-ary Fat-Tree (Bidirectional) Back-to-back butterfly can route all permutations A choice of intermediate positions allowing a conflict-free routing of the permutation What if you just pick a random mid point? Any permutation can be routed with very few conflicts with high probability 8

9 k-ary n-tree (n,k) 9

10 Hypercubes Also called binary n-cubes. # of nodes = N = 2 n. O(logN) hops / diameter Good bisection BW N/2 Complexity Out degree is n = logn 0-D 1-D 2-D 3-D 4-D 5-D! 10

11 Relationship Butterflies to Hypercubes Wiring is isomorphic Except that Butterfly always takes log n steps 3/19/99 11

12 Topology Summary Topology Degree Diameter Ave Dist Bisection (D P=1024 1D Array 2 N-1 N/3 1 huge 1D Ring 2 2 N/2 N/4 2 2D Mesh (N 1/2-1) 2/3 N 1/2 N 1/2 63 (21) 2D Torus 4 N 1/2 1/2 N 1/2 2N 1/2 32 (16) k-ary n-cube 2n nk/2 nk/4 nk/4 15 Hypercube n =log N n n/2 N/2 10 (5) All have some bad permutations many popular permutations are very bad for meshes (transpose) randomness in wiring or routing makes it hard to find a bad one! 12

13 LAYERED DATA CENTER ARCHITECTURE 13

14 Network Design Transition 14

15 Data Center Networks servers per rack Each server connected to 2 access switches with 1 Gbps (10 Gbps becoming common) Access switches connect to 2 aggregation switches Aggregation switches connect to 2 core routers Core routers connect to edge routers Aggregation layer is the transition point between L2- switched access layer and l3-routed core layer Low Latency: In high-frequency trading market, a few microseconds make a big difference. Cut-through switching and low-latency specifications. 15

16 Data Center Networks Core routers manage traffic between aggregation routers and in/out of data center All switches below each pair of aggregation switches form a single layer-2 domain Each Layer 2 domain typically limited to a few hundred servers to limit broadcast Most traffic is internal to the data center. Network is the bottleneck. Uplinks utilization of 80% is common. Most of the flows are small. Mode = 100 MB. DFS uses 100 MB chunks. 16

17 Layered Data Center Architecture Core-aggregation-access layered data center architecture improves Network modularity Flexibility Resilience 17

18 Data Center Network Layers Hierarchical network design 1+1 redundancy Equipment higher in the hierarchy handles more traffic, is more expensive (scale-up design) 18

19 Core Layer Provides the high-speed packet switching backplane for all flows going in and out of the data center Provides connectivity to multiple aggregation modules provides a resilient Layer 3 routed fabric with no single point of failure. Runs an interior routing protocol, such as OSPF or EIGRP Load balances traffic between the campus core and aggregation layers 19

20 Aggregation Layer Meeting point for server IP subnets Default gateway Forwarding server-to-server traffic between multiple pairs of access switches Aggregation layer modules provide important functions: Layer 2 domain definitions Spanning tree processing Service module integration stateful network services Content switching load balancing Firewall SSL offload Intrusion detection Network analysis, and more. 20

21 Aggregation Layer All servers require application delivery services for security (VPN, Intrusion detection, firewall), performance (load balancer), networking (DNS, DHCP, NTP, FTP, RADIUS), Database services (SQL) Application Delivery controllers are located between the aggregation and core routers and are shared by all servers Stateful devices (firewalls) on Aggregation layer 21

22 Access Layer Where the servers physically attach to the network Posses highest number of ports Configured for simplicity to improve management Focusing on the communication between servers that are on the same IP subnet Facilitating exchange of unicast, multicast or broadcast between servers 22

23 Design Factors of Data Center Networks Application bandwidth demand Oversubscription: how much bandwidth switches at each layer can effectively offer to downstream devices Failure domain sizing The number of servers per IP subnet, access switch, or aggregation switches Application resilience Server-redundant Ethernet interfaces should be connected to different access switches The network must be able to react faster to a connection failure when compared to the application server. All the factors should be prioritized 23

24 Access-aggregation Connection Options: Looped Triangle Topology Most widely deployed in data centers Deterministic characteristics and flexibility Access-to-aggregation oversubscription remains constant in the case of Uplink failure Aggregation switch failure STP does not allow all deployed uplinks to be used 24

25 Access-aggregation Connection Options: Looped Square Topology Increases the access layer switch density Each access switch demands only one connection to the aggregation layer Traffic oversubscription to the aggregation layer doubles if An aggregation switch fails Uplink fails 25

26 Access-aggregation Connection Options: Loop-free U Topology No blocked paths because A loop cannot be formed STP is still recommended Like looped square Allows a higher number of access switches per aggregation pair Optimized use of uplinks But Allow one pair of access switches per L2 domain Any switch connection failure will stop all L2 communication 26

27 Access-aggregation Connection Options: Loop-Free Inverted U Shares all advantages from U topologies Allows more than one pair of access switches on a single L2 domain Uplink or aggregation failures are tricky "black-hole" the server traffic: an active server is connected to an isolated network device. 27

28 Switch Locations - TOR Intra-rack cabling between Servers Small switches Smaller cable between servers and switches Network team has to manage switches on all racks 28

29 Switch Locations - EOR Inter-rack cabling between Servers High-density switches All network switches in one rack 29

30 ToR vs EoR ToR: Easier cabling If rack is not fully populated -> unused ToR ports If rack traffic demand is high, difficult to add more ports Upgrading (1G to 10G) requires complete Rack upgrade EoR: Longer cables Severs can be place in any rack Ports can easily added, upgraded 30

31 Network Logical Partition Consolidation is a definitive trend Network partitioning to address Traffic isolation for groups of hosts Distinct security areas Different path behavior Shared failure domains Virtualization 31

32 Defining VLANs A VLAN can be defined as a broadcast domain in a single Ethernet switch or shared among connected switches. Within each VLAN A switch emulates an Ethernet bridge Forward Ethernet frames based on their destination MAC address Each port of a VLAN defines a collision domain 32

33 VLAN trunks Use access ports to connect VLANs Need as many connections as the number of VLANs VLAN trunks Transport multiple VLANs over a single Ethernet interface Each frame has a tag that contains a VLAN identifier 33

34 IEEE 802.1Q VLAN Tagging 34

35 Two IP Subnets Sharing a VLAN Direct IP communication occurs among hosts that belong to each IP subnet Every host receives all broadcast and flooded frames from both subnets 35

36 Two VLANs Sharing a Subnet Layer-2 device Used to bridge both VLANs in a single broadcast domain Traffic analysis Acceleration Content security Load balancing Advantage Traffic manipulation without additional switch deployment or recabling 36

37 Inter-VLAN communication Router Delicate one interface (0/0) connected to a switch trunk port Two sub-interfaces 0/ /0.201 Each sub-interface has an IP configured as the default gateway on the servers Router-on-a-Stick: VLAN-aware router can route IP packets between host located in different VLANs through a single Ethernet connection 37

38 Layer-3 Switches Switch Virtual Interface (SVI) Logical virtual interface Used to route IP packets from its associated VLAN Assign IP address to an SVI Use it as the default gateway for the servers belonging to the VLAN No need for an external router Misconception: Layer-3 VLAN Layer-3 switches: Network equipment that can implement hardware-based L2 switching and L3 forwarding 38

39 Spanning Tree Protocol Recap STP algoryhme poem: I think that I shall never see A graph more lovely than a tree. A tree whose crucial property Is a loop-free connectivity. A tree that must be sure to span So packets can reach every LAN. First, the root must be selected. By ID, it is selected. Least-cost paths from root are traced. In the tree, these paths are placed. A mesh is made by folks like me, Then bridges find a spanning tree. Problem: loops Reason: Always forward a broadcast frame to every Ethernet interface except the one that received it Solution: spanning tree protocol Benefits: loopless topologies & path availability 39

40 Spanning Tree Protocol and VLANs Two solutions A single STP instance for all VLANs (CST) Different STP instances per VLAN (or group of VLANs) Benefits multiple instances Traffic from and to C can be statically load balanced A failure in segment A-C A failure in switch A With ST instances, VLANs can achieve virtualization in the control plan All STP instances share the same software processes 40

41 Private VLAN Three types of interfaces with a VLAN Promiscuous ports Isolated ports Community ports Two types of VLANs Primary VLAN Secondary VLAN Benefits Broadcast subdomains within a VLAN Additional security at Layer 2 Improve partitioning scalability 41

42 VLAN Summary VLANS are Ethernet broadcast domains Connecting VLANs Access ports: interfaces whose transmitted and received frames belong to a single VLAN VLAN trunks: transport multiple VLANs over a single Ethernet interface (VLAN tag) Inter-VLAN communication Router-on-a-Stick design Layer-3 switches Spanning tree protocol and VLANs Private VLAN 42

43 Case study Internet Data Center Layer 3 Internet CR CR AR AR AR AR Layer 2 LB S S LB S S S S Key: CR = L3 Core Router AR = L3 Aggregate Router S = L2 Switch LB = Load Balancer A = Rack of 20 servers with Top of Rack switch 43

44 Internal Fragmentation VIP: the IP to which requests are sent DIP: the IP of the server over which the request is spread Popular load balancing techniques (destination NAT) require all DIPs in a VIP s pool be in the same L2 domain Fragmentation and under-utilization of resources 44

45 No Performance Isolation VLANs used for: security, service isolation, traffic management, etc. One service sending/receiving too much traffic hurts all services sharing its subtree Reconfiguration of VLAN trunks painful, error-prone, slow, often manual 45

46 Limited Server-to-Server Capacity Data center run two kinds of application: Outward facing (serving web pages to users) Internal computation (computing search index like HPC) Comm. between servers in different L2 domains must go through L3 network bandwidth bottleneck 46

47 Data Center Networking Issues Higher layers oversubscribed: Other servers in the same rack 1:1 Uplinks from ToR: 1:2 to 1:20 (e.g., 32x10Gb down, 8X10Gb up -> 4:1) Core Routers: 1:240 Generally keep services in one tree Can't arbitrarily move servers Moving across Subnets is painful Requires reconfiguration of IP addresses and VLAN trunks 47

48 Data Center Networking Issues (Cont.) Service trample on each-other. Overuse by one service affects others Poor reliability. One access switch failure doubles the load on the other. Under-utilization: Even when multiple paths exist only one is used. ECMP (Equal Cost Multipath) is used by routers to spread traffic to next hops using a hash function. However, only 2 paths exist. 48

49 Reference G.A.A. Santana, "Data Center Virtualization Fundamentals," Cisco Press, 2013, ISBN: Albert Greenberg, James Hamilton, David A. Maltz, and Parveen Patel The cost of a cloud: research problems in data center networks. SIGCOMM Comput. Commun. Rev. 39, 1 (December 2008),

50 BACKUP 50

51 Virtual LAN 51

52 DCN Requirements Needs to be Scalable, Secure, Shared, Standardized, and Simplified (5 S's) Converged Infrastructure: Servers, storage, and network have to work together Workload Mobility: Large L2 domains required for VM mobility East-West Traffic: Significant server-to-server traffic as compared to server to user. One Facebook request required 88 cache looks, 35 database lookups, 392 backend RPC calls. Internet traffic 935X the http request/response [Farrington] Storage traffic on Ethernet: Congestion management on Ethernet 52

53 How Many Dimensions? n = 2 or n = 3 Short wires, easy to build Many hops, low bisection bandwidth Requires traffic locality n >= 4 Harder to build, more wires, longer average length Fewer hops, better bisection bandwidth Can handle non-local traffic k-ary d-cubes provide a consistent framework for comparison N = k d scale dimension (d) or nodes per dimension (k) assume cut-through 3/19/99 CS258 S99 53

54 Hierarchical Network Design All servers require application delivery services for security (VPN, Intrusion detection, firewall), performance (load balancer), networking (DNS, DHCP, NTP, FTP, RADIUS), Database services (SQL) Application Delivery controllers are located between the aggregation and core routers and are shared by all servers Stateful devices (firewalls) on Aggregation layer 54