UCS Nirvana I A N G S JOURNEY FROM LEGACY BLADE S E R V E R S T O C I S C O U N I F I E D C O M P U T I N G S Y S T E M, W I T H N O ( N O T I C E A B L E ) V M W A R E D O W N T I M E Lee Burleson Network and Systems Engineer Iowa National Guard Cisco ARNG Training Event March 2011
Disclaimer This presentation contains information on Iowa National Guard migration from a legacy blade server environment to a Cisco UCS system. The content was created for the audience attending the 5 th Annual Training Event and not for distribution beyond the attendees without the express permission of the Iowa National Guard.
Introduction whoami My credentials Outline Can t do a UCS AO class in 45m
Legacy Environment 2 Dell 1855 chassis 18 ea 1955 Blade servers Great for IANG when purchased
Legacy Environment, cont. The aura wears off Less-than-ideal chassis management interface Cabling: power, FC, Ethernet, management, KVM Design limitations, especially when virtualizing Module management Eth, FC Points of management
Photos
Photos, cont.
Overview and Background Project goals Modernize Improve management Boot from SAN Reduce complexity Increase performance Blade Chassis Research Read both vendor-produced AND independent literature Drank Cisco Kool-Aid @ Minneapolis Wrote specs (anyone need those?) Received 2 ea chassis & 8 ea B200M2 blades
UCS Basics Fabric Interconnects are heart of system (2) 40c x 8b = 320b* Not just for virtualization, but does it very well Single and well-done mgmt UI All config kept in FI hardware independence Built-in, transparent fabric redundancy
UCS Photo Front view
UCS Photo Rear view
Hardware Installation Chassis are heavy Very clean cabling Power cable challenges with our PDUs Ordered C19-C14 cables Per chassis: 2 ea 1M cords 2 ea 2M cords
Hardware Installation, cont. Ordered reduced-length IOM interconnects (SFP- H10GB-CU1M) Remember: only 1,2,or 4 IOM connections are allowed!
Management Baby steps Connected only mgmt ports of both Fabric Interconnects Must reserve IP space in same subnet for KVM access to blades Used management interface (Java) to explore and design system without external impact
Management, cont. Able to create entire system without upstream connectivity or even the blades themselves Resource pools Server pools MAC address pools WWNN pools KVM IP address pools vnic templates Etc Service profile templates Service profiles (7b+1ub)
Storage Configure switches based on UCS config WWNN Aliases + Zoning = Port independence Enable NPIV (globally + UCS ports) Connected FIs to FC fabric; verified config May want to fully alias and zone in both fabrics; depends on your design No production VMFS presentation yet! Built initial test ESXi server as SAN-boot POC
Network Single 1GbE link from FI pair connected upstream - helped prove failover and identify issues Spent a while in this state 4 x 10Gb transition was later seamless Ease of adding VLANs
VMware Integration Wanted to improve VM I/O, even past basic host improvements gained with upgrade Hypervisor bypass/pci pass-through/vmdirectpath I thought this was what I wanted, until I tried it Various painful consequences Recommendation: don t do it! (until.)
VMware Integration, cont. Our host design: 4 network interfaces (vnics) + 2 FC vhbas vnic0 - vswitch0 mgmt - fabric A/B vnic1 - vswitch1 vmotion - fabric B/A vnic2 - FI dvs vm_uplink01 - fabric A only vnic3 - FI dvs vm_uplink02 - fabric B only No trunking! Pass-through switching - VN-Link vmotion works great Reduces host CPU, reduces latency, improves I/O Creates dynamic vnic## vm int xx - FI dvs balanced fabric connections!
VMware Integration, cont. VMware host interfaces, UCS screenshot
VMware Integration, cont. VMware host interfaces, VMware screenshot
VMware Integration, cont. Created addl cluster for isolation Joining the datacenter DVS connection Different DVSs available: VMware, Nexus 1000V, Nexus 1010, 6100 FI
VM Migration The culmination of all our work Recommend temporarily setting DRS to Partial so you don't fight it Ran into first problem
VM Migration, cont. Changing the VM Network Connection Second problem: destination host didn t have a Port Group for the VM to connect to
VM Migration, cont. Changing the VM Network Connection
VM Migration, cont. Moving vcenter itself The last challenge, so I thought Catch 22? Procedure used: [Bad memory block] Did something bad; I lost my vcenter connection Direct-connect to vcenter-owning host Power down vcenter VM Remove from Inventory Direct-connect to UCS crossover host Import VM, verify network, and power it on Connect to vcenter and change adapter over to dvs (as in standard procedure)
VM Migration, cont. Moving vcenter itself (also cont.) Procedure that I should have used: Migrate vcenter VM to crossover host Connect to vcenter and change adapter over to dvs (as in standard procedure) Still, be careful
VMware Integration, cont. Balanced fabric screenshot
Goals Met? Definitely modernized Vastly improved management 100% boot from SAN (even the MSSQL blade) Reduced complexity Improved performance: vmotion SharePoint VM with 16GB RAM Before: ~3 minutes, After: 40 seconds Can take down a host running a dozen VMs within 5 minutes
Future Nexus 5548 Topology considerations - are the 5548s the new de facto datacenter core with L3? HP iscsi gateway for EVA 10Gb iscsi for VMs The Nexus 1000V is no longer needed with VMWare on UCS (or is it?)
Acknowledgements/Resources www.bradhedlund.com blog.scottlowe.com www.cisco.com/go/ucs bladesmadesimple.com