Cloud Networking with OpenStack. 马力 海云捷迅 (AWcloud)

Transcription

1 Cloud Networking with OpenStack 马力 海云捷迅 (AWcloud)

2 About Me Li Ma (Nick) SDN Developer Cloud/Network Architect OpenStack Contributer Neutron COE Volunteer

3 About AWcloud AWcloud Pure OpenStack player from 2012 China s leading enterprise cloud service provider Broad deployment of production clouds Large-scale infra operations background Highly diverse set of workloads and use cases

4 Data Center Networking Review Internet Intranet Service Zone GSLB/DNS/ Cache/NAT CA/AAA External Gateway Security Zone FW/IPS/ Anti-DDOS Ops Zone KVM/EOB Fabric Dev/Test Zone R&D Zone Computing Resource Pool Storage Resource Pool

5 DC Technologies is Evolving

6 How to be Cloud-Resiliant Public Clouds Many VMs Many Servers Many Switches MAC & ARP Network Isolation VM Migration Network Bandwidth Architecture Virtualization VDC/VPN Dedicated Devices

7 Traditional Architecture

8 Traditional Architecture Disadvantages Spanning Tree Protocol Bandwidth Convergence: seconds Over-subscription Ratio Downlinks : Uplinks

9 Fabric Architecture

10 Fabric Architecture Advantages MLAG Load Balancing Convergence: milliseconds Over-subscription Ratio Easy to adjust 1:1-3:1

11 Fabric Architecture 160Gb * 2 ECMP 20Gb * 40 Servers

12 OpenStack Architecture

13 L2/L3 Fabric L2/L3 Fabric OSPF + ECMP + Overlay IS-IS based Ethernet Routing TRILL/SPB/Cisco FabricPath Switch Virtualization Cisco VSS, H3C IRF, Brocade VCS Ruijie VSU, DCN VSF, Dell VLT

14 L2/L3 Fabric L2/L3 Fabric Consideration Cost (CapEx, OpEx) Operation Techniques Vender Lock-in

15 L2/L3 Fabric L2/L3 Fabric Consideration Multi-Path Hashing: VxLAN: SRC UDP PORT

16 Tolerant of Rack Level Failure Tolerant of Rack Level Failure xstp, Stacking, MLAG Consideration Cost (CapEx, OpEx) Convergence Need unified control plane? # of participating switches?

17 NIC Settings NIC Bonding balance-rr active-backup balance-xor broadcast 802.3ad (LACP) balance-tlb balance-alb

18 NIC Settings NIC Bonding Ethtool Support: speed, duplex Dynamic Modification: mac-addr Switch Configuration: 802.3ad 802.3ad is more secure. ARP monitor is incompatible with bridging. Update Kernel Routing Table

19 NIC Settings NIC Bonding use_carrier not supported? Cause failure of link detection Not real-time? Cause delay of detection. use_carrier = 0

20 NIC Settings NIC RSS (multi-queue) Support UDP? For Intel 82599, it's disabled that causes fragmented UDP frames go to CPU/Queue 0. Slow performance/utilization for VxLAN.

21 NIC Settings NIC HW Offload Stateless RX/TX Checksum ethtool -K tx/rx on Scatter & Gather ethtool -K ethx sg on

22 NIC Settings NIC HW Offload Stateless TCP Segmentation Offload ethtool -K ethx tso on Large Receive Offload ethtool -K ethx lro on Generic Segmentation Offload ethtool -K ethx gso on

23 NIC Settings NIC HW Offload Stateful TCP Offload Engine Identify TCP states Help kernel do fragmentation/acknowledgement Depend on vendor's implementation

24 NIC Settings NIC HW Offload VxLAN Offload Sender: Inner/Outer checksum computation Large Segmentation Offload Receiver: Receive Side Scaling (multi-queue) Inner/Outer checksum validation

25 NIC Settings NIC HW Offload

26 Hypervisor Jumbo Frame Terminate at the Internet edge ifconfig ethx mtu {mtu} Transmission Queue Size ifconfig ethx txqueuelen {size}

27 Hypervisor Kernel Tuning net.ipv4.tcp_window_scaling net.ipv4.tcp_timestamps net.ipv4.tcp_congestion_control: cubic/hybla net.ipv4.tcp_fin_timeout net.ipv4.tcp_sack net.ipv4.tcp_moderate_rcvbuf net.ipv4.tcp_rmem, tcp_wmem, tcp_mem

28 Hypervisor Kernel Tuning net.core.rmem_max, wmem_max, etc. net.core.netdev_max_backlog

29 Hypervisor

30 Hypervisor Paravirtualization NIC for KVM vhost_net + virtio guest driver

31 Hypervisor Paravirtualization NIC for KVM VhostNet provides better latency (10% less than e1000 on my system) and greater throughput (8x the normal virtio, around 7~8 Gigabits/sec here) for network.

32 Thanks