UCS 4+ Years of Lessons Learned by the TAC

Size: px

Start display at page:

Download "UCS 4+ Years of Lessons Learned by the TAC"

Melissa Byrd
5 years ago
Views:

2 UCS 4+ Years of Lessons Learned by the TAC Ken Krzyzewski - Technical Leader Cisco TAC Data Center & Virtualization Technical Leadership Team

3 Agenda Introduction What went wrong? interactive scenarios UCS Admin. Best Practices Overview Avoiding the Avoidable Maximizing UCS TAC interactions Conclusion 3

4 Session Goals Learn how to avoid the avoidable Leverage UCS best practices Reinforce good habits (for those already following best practices) Learn what can happen when you take shortcuts Manage UCS TAC Cases more effectively Insider tips for quicker issue resolution 4

5 A real UCS customer saga 5

6 6 Release Notes

7 Release Notes Unofficial Survey Do you read the Release Notes? Customers Response TAC's Perception 7 20 Yes No Yes No

8 Release Notes Pay particular attention to: Open Caveats Resolved Caveats are the typical reason for an upgrade New Features Internal Dependencies 8

9 Release Notes Mixed Cisco UCS Release Support Supported in release 2.1(x) and above Allows for independent infrastructure upgrades Consult the release notes for details 9

10 Release Notes Mixed Cisco UCS Release Support matrix example 10

11 Release Notes Mixed Release Refer to the Minimum B/C Bundle Features section New features may require B-bundle upgrade 11

12 Release Notes Caveat details if running in a mixed firmware environment Resolved caveats may require B-bundle upgrade 12

13 13 UCS Firmware Upgrades

14 UCS Firmware Upgrades Treat them like elective surgery Pre-op check-up The operation Recovery Room Released from surgical center Pro-active TAC SR The upgrade Verify functionality Resume production 14

15 UCS Firmware Upgrades Pre-Upgrade Check list Consult Release Notes, work with your account team Back-up your system Review Compatibility Matrices Eliminate Critical/Major Faults Watch our Video Upgrade Guides Check Cisco s online community and support forums 15

16 UCSM Firmware Upgrades Frequently forgotten or missed items Updating OS drivers to meet the compatibility matrix Backing up the system prior to upgrade Upgrade of blade BIOS & Board Controller 16

17 UCSM Firmware Upgrade UCS HW and SW Interoperability example 17

18 UCSM Firmware Upgrade (cont.) Results (continued) Adapter Driver = (FNIC) / (ENIC) Adapter Firmware = 2.1(3) Boot Code / BIOS = 18

19 UCSM Upgrades Host Firmware Package Simple option 19

20 UCS Firmware Upgrade New features in 2.2(2) During the Auto-Upgrade process: System back-up reminder Critical/Major fault presence alerts 20

21 21 Maintenance Windows

22 Maintenance Windows TAC customer example Customer opens a case regarding a critical fault We explain how to resolve it, and due to the potential for it to be service impacting, strongly suggest a maintenance window 1 Hour later, the same customer is back with a P1 SR! 22

23 Maintenance Windows Better safe than sorry An Industry standard best practice for Data Centers Especially critical for Fabric Interconnect changes 23

24 24 System Back Up

25 System Backups TAC Case example During an attempted upgrade, configuration database corrupted UCSM was in a degraded state, not allowing back-ups nor show tech 11 month old show tech ucsm found in our TAC SR database Painful, but successful, reconfiguration (4+ hour effort) of entire UCS domain 25

26 UCSM Back-ups UCS Back-up Types 1. Full State 2. System Configuration 3. Logical Configuration 4. All Configuration 26

27 Backup Types Full State Binary file Encrypted (passwords and sensitive data not in clear text) Intended for Disaster Recovery Ideal for pre-upgrade 27

28 Backup Types System Configuration XML file System configuration such as username/roles Exportable to external Fabric Interconnects 28

29 Backup Types Logical Configuration XML file Service Profiles, VLANs, VSANs, pools & policies Exportable to external Fabric Interconnects 29

30 Backup Types All Configuration XML file Includes System & Logical configuration settings Exportable to external Fabric Interconnects 30

31 31 Fibre-Channel Port Channels

32 FC Port Channels TAC Case Example Hosts reporting high storage latency They added 3 additional FC uplinks (doubling bandwidth) No change in latency! What happened? 32

33 FC Port Channels Individual FC Uplink behavior FC Switch Fabric Int. FDISC fcid1 fcid4 LOGO FDISC FDISC fcid2 fcid3 fcid5 fcid6 fcid1 fcid4 Individual FC Uplinks 33

34 FC Port Channels The power of the bundle On logical link comprised of the individual physical links Frames sent round-robin, per link, per Src/Dst/OxID FC Switch Fabric Interconnect Port-channel bundle 34

35 FC Port Channels Details Requires MDS or Nexus upstream switch Dynamically modify Port Channel link membership Load Balancing amongst member links is inherent No need to be concerned about multiple high b/w hosts pinned to same link 35

FC Port Channels Back to our TAC case 25000 No PC 20000 15000

36 FC Port Channels Back to our TAC case No PC No PC Port 1 Port 2 Port 3 Port 4 Port 5 Port 6 36

37 FC Port Channels Port Channel vs. individual FC uplinks PC No PC Port 1 Port 2 Port 3 Port 4 Port 5 Port 6 37

38 FC Port Channels Conclusion Port Channels (if possible) are preferred Provides optimal traffic distribution Dynamic PC membership changes No known down-side 38

39 FC Topologies

40 FC Topologies Best Practice Model FC Switches A-Side B-Side <- Physical Separation -> Fabric Interconnects 40

41 FC Topologies Common Mistake 1 (ISL) FC Switches A-Side B-Side Fabric Interconnects 41

42 FC Topologies Common Mistake 2 (crossing) FC Switches A-Side B-Side Fabric Interconnects 42

43 FC Topologies Rare Mistake (ISL + Cross) FC Switches A-Side B-Side Fabric Interconnects 43

44 44 3 rd Party Transceivers

45 3 rd Party Transceivers TAC Case example We recently encountered Cisco Compatible non-cisco twin-ax installed in a relatively large UCS B-Series deployment What we found: Cisco PID spoofed, prevented unsupported transceiver fault Low percentage of frames with CRC errors Fibre Channel performance severely impacted due to the dropped frames Please be aware of our experience when choosing 3 rd Party transceivers 45

46 DIMM Faults

47 Degraded DIMM alerts Background Probability of ECC increases as DIMM geometries shrink ECC threshold monitoring can lead to Degraded DIMM marked faults Not to be confused with DIMMs marked Inoperable 47

48 Degraded DIMMs Impact Per UCS Engineering studies: UCS servers handle ECC errors without impact to server No Performance impact with DIMM s in degraded state Our thresholds for marking a DIMM degraded deemed too conservative 48

49 Degraded DIMMs Resolution New Firmware will change thresholds to practical values Fixed in 2.2(1b) and 2.1(3c) Workaround: You can safely ignore the degraded DIMM faults until you upgrade, or RMA the degraded DIMM 49

50 DIMM Blacklisting New in 2.2(1b) With DIMM Blacklisting feature enabled, if uncorrectable DIMM errors are encountered: CIMC records location of faulty DIMM During next boot sequence, the faulty DIMM gets mapped out Benefits Allows server to safely remain in production Allows for RMA of faulty DIMM when convenient Please note that the feature is disabled by default 50

51 C-Series

52 C-Series CIMC configuration Please configure your CIMC! Troubleshooting is nearly impossible without it 52

53 C-Series TAC Case example Large C-series standalone deployment Disk performance degraded on a number of servers TAC found failed BBU s Servers had moved from write-back to write-through Recommendation: Employ SNMP or IPMI monitoring, especially in large deployments 53

54 C-Series Monitoring feature We expect to have a UCSM-like feature to facilitate standalone C-Series monitoring available in a future release. 54

55 C-Series C-Series Integration Please ensure to update the Rack Servers Bundle C is used to upgrade integrated racks servers Caution: Re-ack of a FEX will require a re-ack of all associated rack servers 55

56 Working with the TAC

57 Working with the TAC General Best Practices Generate appropriate show tech dumps ASAP Have a complete Topology Diagram available Invite TAC SR owner to 3 rd Party vendor calls when appropriate Always include attach@cisco.com in all s 57

58 Working with the TAC Why a great Problem Description is worth the effort Speeds up the problem resolution process An engineer familiar with the symptoms is more likely to grab it Helps the SR reviewer ensure that the case is progressing as expected 58

59 Working with the TAC What to include in your Problem Descriptions Firmware Version & type(s) of equipment Error messages Clear and concise explanation of the problem Pertinent details, such as: New installation vs. production environment Any changes that may have led to the problem Impact to your business 59

60 Working with the TAC Upgrades Engage the TAC if you have any questions prior to an upgrade Engage the TAC if you encounter problems during an upgrade The sooner, the better Please note that TAC engineers are unable to: Perform upgrades Recommend Firmware versions Perform bug scrubs 60

61 Working with the TAC Upgrades (continued) Cisco Services are available to assist with upgrades and bug scrubs Consult your Cisco account team for details 61

62 Working with the TAC Hardware Failures & RMA s Expect requests for logs, and allow time for analysis Hardware components should only be replaced when warranted Incompatible hardware and software can appear as failed hardware Examples: B200-M3 with Ivy Bridge processer requires 2.1(3) or higher Inoperable DIMM s may be due to missing Voltage Regulator update 62

63 Working with TAC Hardware Failures & RMA s (continued) Fully assembled blades are an option Typically available Next Business Day (NBD) Subject to part(s) availability at the assembly depot 63

64 Working with the TAC SR Ownership Only re-queue when immediate assistance is required Re-queuing for a status update can be counter-productive Behind every UCS TAC Engineer: Colleagues & Mentors Subject Matter Experts Team Leads Managers Technical Leaders Escalation engineers 64

65 65 Conclusion

66 Wrapping it up, the rest of the story 66

67 Wrap Up Release Notes What was missed (from the 2.0 Release Notes) After an upgrade from a prior release to 2.0(1), a critical fault may be raised about an overlapping or matching FCoE VLAN ID used for a vsan and an Ethernet VLAN ID under the same fabric as the FCoE VLAN. The fault can be avoided by changing either the FCoE VLAN ID or the Ethernet VLAN ID so that they have two different IDs prior to the upgrade. Resolving the problem after the upgrade may lead to down time for the system. 67

68 Wrap Up The rest of the story His boss joins the conference bridge and asks: Did we have a maintenance window in place for the change? Awkward silence after he admitted that there was no maintenance window in place. This didn t have to happen, please don t let it happen to you. 68

69 Key Takeaways Utilize Maintenance Windows Read and understand Release Notes Maintain good Compatibility Matrix hygiene Leverage the TAC efficiently & effectively When in doubt, call the TAC 69

70 Complete Your Online Session Evaluation Give us your feedback and you could win fabulous prizes. Winners announced daily. Complete your session evaluation through the Cisco Live mobile app or visit one of the interactive kiosks located throughout the convention center. Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online 70

71 Continue Your Education Demos in the Cisco Campus Walk-in Self-Paced Labs Table Topics Meet the Engineer 1:1 meetings 71

73 73 I2C A brief history

74 I2C What is I2C? I2C = Inter-Integrated Circuit, around for many decades Master/Slave bus technology Employed by UCS to facilitate IO Module communication with chassis components: Fan and PSU readings Chassis SEEPROM (a.k.a Shared Storage) access Blade readings 74

75 I2C Bus in the UCS Chassis 75

76 I2C Typical Defect Symptoms Fan faults Shared Storage faults Temperature warnings Fans spinning at 100% 76

77 I2C Improving over time I2C issues improve as firmware progresses: CSCue49366 : Midplane I2C norxack, c.ms errors cause budget-mc Error(-5) CSCtx52556 : No UCSM fault yet one or more Fan LEDs indicate a fault CSCtx49686 : Chassis thermal faults do not clear Fan related CSCua10675 : FAN kernel driver bug fix CSCtl43716 : 9541 device error. Fan Modules reported inoperable, running 100% 77

UCS Technical Deep Dive: Getting to the Heart of the Matter

UCS Technical Deep Dive: Getting to the Heart of the Matter Session ID Agenda Introductions UCS Architecture, Innovations, Topology Physical Building Blocks Logical Building Blocks Typical Use Cases (Live