UCS Nirvana. Lee Burleson Network and Systems Engineer Iowa National Guard. Cisco ARNG Training Event March 2011

Transcription

1 UCS Nirvana I A N G S JOURNEY FROM LEGACY BLADE S E R V E R S T O C I S C O U N I F I E D C O M P U T I N G S Y S T E M, W I T H N O ( N O T I C E A B L E ) V M W A R E D O W N T I M E Lee Burleson Network and Systems Engineer Iowa National Guard Cisco ARNG Training Event March 2011

2 Disclaimer This presentation contains information on Iowa National Guard migration from a legacy blade server environment to a Cisco UCS system. The content was created for the audience attending the 5 th Annual Training Event and not for distribution beyond the attendees without the express permission of the Iowa National Guard.

3 Introduction whoami My credentials Outline Can t do a UCS AO class in 45m

4 Legacy Environment 2 Dell 1855 chassis 18 ea 1955 Blade servers Great for IANG when purchased

5 Legacy Environment, cont. The aura wears off Less-than-ideal chassis management interface Cabling: power, FC, Ethernet, management, KVM Design limitations, especially when virtualizing Module management Eth, FC Points of management

6 Photos

7 Photos, cont.

8 Overview and Background Project goals Modernize Improve management Boot from SAN Reduce complexity Increase performance Blade Chassis Research Read both vendor-produced AND independent literature Drank Cisco Minneapolis Wrote specs (anyone need those?) Received 2 ea chassis & 8 ea B200M2 blades

9 UCS Basics Fabric Interconnects are heart of system (2) 40c x 8b = 320b* Not just for virtualization, but does it very well Single and well-done mgmt UI All config kept in FI hardware independence Built-in, transparent fabric redundancy

10 UCS Photo Front view

11 UCS Photo Rear view

12

13 Hardware Installation Chassis are heavy Very clean cabling Power cable challenges with our PDUs Ordered C19-C14 cables Per chassis: 2 ea 1M cords 2 ea 2M cords

14 Hardware Installation, cont. Ordered reduced-length IOM interconnects (SFP- H10GB-CU1M) Remember: only 1,2,or 4 IOM connections are allowed!

15 Management Baby steps Connected only mgmt ports of both Fabric Interconnects Must reserve IP space in same subnet for KVM access to blades Used management interface (Java) to explore and design system without external impact

16 Management, cont. Able to create entire system without upstream connectivity or even the blades themselves Resource pools Server pools MAC address pools WWNN pools KVM IP address pools vnic templates Etc Service profile templates Service profiles (7b+1ub)

17 Storage Configure switches based on UCS config WWNN Aliases + Zoning = Port independence Enable NPIV (globally + UCS ports) Connected FIs to FC fabric; verified config May want to fully alias and zone in both fabrics; depends on your design No production VMFS presentation yet! Built initial test ESXi server as SAN-boot POC

18 Network Single 1GbE link from FI pair connected upstream - helped prove failover and identify issues Spent a while in this state 4 x 10Gb transition was later seamless Ease of adding VLANs

19 VMware Integration Wanted to improve VM I/O, even past basic host improvements gained with upgrade Hypervisor bypass/pci pass-through/vmdirectpath I thought this was what I wanted, until I tried it Various painful consequences Recommendation: don t do it! (until.)

20 VMware Integration, cont. Our host design: 4 network interfaces (vnics) + 2 FC vhbas vnic0 - vswitch0 mgmt - fabric A/B vnic1 - vswitch1 vmotion - fabric B/A vnic2 - FI dvs vm_uplink01 - fabric A only vnic3 - FI dvs vm_uplink02 - fabric B only No trunking! Pass-through switching - VN-Link vmotion works great Reduces host CPU, reduces latency, improves I/O Creates dynamic vnic## vm int xx - FI dvs balanced fabric connections!

21 VMware Integration, cont. VMware host interfaces, UCS screenshot

22 VMware Integration, cont. VMware host interfaces, VMware screenshot

23 VMware Integration, cont. Created addl cluster for isolation Joining the datacenter DVS connection Different DVSs available: VMware, Nexus 1000V, Nexus 1010, 6100 FI

24 VM Migration The culmination of all our work Recommend temporarily setting DRS to Partial so you don't fight it Ran into first problem

25 VM Migration, cont. Changing the VM Network Connection Second problem: destination host didn t have a Port Group for the VM to connect to

26 VM Migration, cont. Changing the VM Network Connection

27 VM Migration, cont. Moving vcenter itself The last challenge, so I thought Catch 22? Procedure used: [Bad memory block] Did something bad; I lost my vcenter connection Direct-connect to vcenter-owning host Power down vcenter VM Remove from Inventory Direct-connect to UCS crossover host Import VM, verify network, and power it on Connect to vcenter and change adapter over to dvs (as in standard procedure)

28 VM Migration, cont. Moving vcenter itself (also cont.) Procedure that I should have used: Migrate vcenter VM to crossover host Connect to vcenter and change adapter over to dvs (as in standard procedure) Still, be careful

29 VMware Integration, cont. Balanced fabric screenshot

30 Goals Met? Definitely modernized Vastly improved management 100% boot from SAN (even the MSSQL blade) Reduced complexity Improved performance: vmotion SharePoint VM with 16GB RAM Before: ~3 minutes, After: 40 seconds Can take down a host running a dozen VMs within 5 minutes

31 Future Nexus 5548 Topology considerations - are the 5548s the new de facto datacenter core with L3? HP iscsi gateway for EVA 10Gb iscsi for VMs The Nexus 1000V is no longer needed with VMWare on UCS (or is it?)

32 Acknowledgements/Resources blog.scottlowe.com bladesmadesimple.com