HP CloudSystem Matrix 7.5 Troubleshooting Guide

Transcription

1 HP CloudSystem Matrix 7.5 Troubleshooting Guide Abstract This document is intended for HP CloudSystem Matrix administrators. It describes the major troubleshooting issues for CloudSystem Matrix, particularly for HP Insight Management. HP Part Number: Published: August 2015 Published: Edition: 1

2 Copyright 2015 Hewlett-Packard Development Company, L.P. Notices Confidential computer software. Valid license from HP required for possession, use or copying. Consistent with FAR and , Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor's standard commercial license. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Acknowledgments Java is a registered trademark of Oracle and/or its affiliates. Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation. UNIX is a registered trademark of The Open Group. Intel is a trademark of Intel Corporation in the U.S. and other countries.

3 Contents 1 About this document...6 Intended audience...6 Using this document...6 Using or configuring the CMS...6 Managing CMS crash recovery...6 Using or configuring managed systems...7 Using VM hosts and guests...7 Using logical servers...8 Managing storage...9 Using Matrix recovery management...9 Using deployment and provisioning...9 Using Global Workload Manager...10 HP Matrix KVM Private Cloud troubleshooting Troubleshooting tools...12 Backing up your environment...12 Using the Insight managed system setup wizard...13 Before running the Insight managed system setup wizard...13 Running the Insight managed system setup wizard...13 Using vseassist...14 Use cases...14 Running the vseassist tool...15 Diagnostics in vseassist...17 Using System Management Homepage troubleshooting tools...20 Running Configure or Repair Agents...20 Using Manage Communications...21 Checking log files Troubleshooting procedures...28 Agent and communication troubleshooting...28 Install OpenSSH on Windows 2008 managed node through Configure or Repair Agents fail...28 A local user or domain account becomes locked out...28 WBEM/WMI authentication and communication issues...28 Communication with HP SIM server has been lost...30 No buffer space available, maximum connections reached...31 Server blade serial number is not added on ESX or Hyper-V managed nodes...32 Insight Control virtual machine management registration for ESX or Hyper-V nodes fail...32 Hyper-V VM hosts disappear from the IO server pool in large scale environments...32 Matrix infrastructure orchestration troubleshooting...33 IDE disk on a VMware ESX VM fails to reactivate...33 Unable to add servers in a SCVMM service...33 HP Matrix infrastructure provisioning and linked clone provisioning fails when using a Windows 2012 OS template on ESXi 5.0 Update 1 host...34 Password is blank on provisioned VMs with Windows 2008 R2 and above provisioned VMs...35 Microsoft SCVMM templates do not appear in the Matrix infrastructure orchestration software tab...35 Incorrect hostname displays during validation for deployed Hyper-V VM...36 Delete operation of a VM entry from Microsoft SCVMM fails...36 Matrix infrastructure orchestration service creation fails in the target reconfiguration phase...36 Matrix infrastructure orchestration service creation fails with Error powering on logical server...37 Matrix infrastructure orchestration service creation fails in the VM customization phase...37 Service request fails when using a template configured with custom host names...39 Contents 3

4 Physical server provisioning using Insight Control server deployment fails during the final personalization step...40 Matrix infrastructure orchestration service creation fails for physical requests where Windows Server OS is selected...40 Provisioned VMs fail due to improper VM host hardware configuration...41 Unable to allocate servers and networks in the same VCDG...42 Matrix infrastructure orchestration request fails due to server use outside of infrastructure orchestration...43 Automated OS provisioning to a physical server fails to successfully PXE boot...43 Request History shows deployment server does not have a record of the target system(s)...43 Matrix infrastructure orchestration Networks tab does not retrieve network information...44 Matrix infrastructure orchestration deployment task fails, exceeding its timeout value...44 Matrix infrastructure orchestration cannot deploy Integrity VM guests when gwlm is controlling the host...45 Manually cleaning up Clean-me logical servers (physical)...46 Manually cleaning up Clean-me logical servers (virtual)...47 Matrix infrastructure orchestration users cannot view or receive HP Operations Orchestration Defined SPM storage template does not appear in infrastructure orchestration designer...49 VC domain error after CMS crash recovery...49 Unassign profile request on a VC logical server fails during deactivate after a VC failover...50 Matrix infrastructure orchestration Create Service request does not complete successfully after CMS crash recovery...50 Matrix infrastructure orchestration Create Virtual Servers Request does not complete successfully after a CMS crash recovery...51 Matrix infrastructure orchestration Deactivate physical server does not complete successfully after CMS crash recovery...51 Matrix infrastructure orchestration Deactivate Virtual Servers Request does not complete successfully after a CMS crash recovery...52 Matrix infrastructure orchestration Add Disk request for an ESX virtual server does not complete successfully after CMS crash recovery...52 mxagentconfig command fails with SSH protocol error on Integrity managed node...52 Unable to activate an Integrity VM logical server due to a disk creation failure...53 Insight Control server deployment software is not listed in infrastructure orchestration...55 VM snapshot creation fails...56 Linked clone provisioning of a VM template containing shared disks fails...56 VM instances not deleted after Matrix infrastructure orchestration fails...56 Add disk operation is not working with SPM in VM cluster environment...57 Ignite-UX troubleshooting...57 Matrix recovery management troubleshooting...60 Logical server troubleshooting...63 Integrity Virtual Server Manager troubleshooting...79 gwlm troubleshooting...81 Insight Control troubleshooting...84 Insight Control virtual machine management troubleshooting...84 Insight Control server deployment troubleshooting...89 Server Automation or Insight Control server provisioning troubleshooting...94 Server Automation or Insight Control server provisioning fails due to boot order changes...94 Provisioning of Matrix service fails in customization...95 Failure when creating or deactivating a High Availability Hyper-V virtual machine with RDM/pass-through cluster disks...95 Storage troubleshooting...96 Storage Provisioning Manager issues...96 Storage pool issues...97 Other storage issues Contents

5 HP Matrix KVM Private Cloud troubleshooting Time and date on the management appliance do not match time and date on the NTP server VM guests cannot be provisioned on an activated host Support and other resources Contacting HP Information to collect before contacting HP How to contact HP Warranty information HP authorized resellers Software technical support and update service Registering for software technical support and update service How to use your software technical support and update service Customer advisories Related information Typographic conventions Documentation feedback Acronyms & Abbreviations Index Contents 5

6 1 About this document This document provides troubleshooting information for the HP CloudSystem Matrix 7.5. Intended audience This document is intended for users who are: Knowledgeable about HP CloudSystem Matrix. Experienced with HP Systems Insight Manager, Matrix Operating Environment, HP Insight Control, and HP Virtual Connect Enterprise Manager. Experienced with installing and configuring Windows Server operating systems and the required hardware, firmware, and software drivers, including: Familiarity with the HP Service Pack for ProLiant and the associated Windows Services, such as SNMP. Knowledge of Windows user account and group creation, modification, and membership. Familiar with installation of VMware Infrastructure 4.x or 5.x Enterprise Edition, specifically VMware ESX Hypervisor and vcenter. Qualified to configure an HP BladeSystem VC environment, including VC Ethernet modules, VC Fibre Channel modules, and server blades. Using this document This document describes the major troubleshooting issues for HP Insight Management. Chapter 2 describes the major troubleshooting tools. You will find the major issues and resolutions listed by software in Chapter 3. To further help you locate an issue, the following categories cross-reference the issues by symptom or error message. Using or configuring the CMS Communication with HP SIM has been lost (page 30) No buffer space available, maximum connections reached (page 31) Matrix infrastructure orchestration Networks tab does not retrieve network information (page 44) Matrix infrastructure orchestration users cannot view or receive HP Operations Orchestration (page 49) Logical Server Automation service fails to start if TCP layer ports are in use (page 69) Detecting and repairing database connectivity problems (page 86) Managing CMS crash recovery Unassign profile request on a VC logical server fails during deactivate after a VC failover (page 50) VC domain error after CMS crash recovery (page 49) Matrix infrastructure orchestration Create Service request does not complete successfully after CMS crash recovery (page 50) Matrix infrastructure orchestration Create Virtual Servers Request does not complete successfully after a CMS crash recovery (page 51) 6 About this document

7 Matrix infrastructure orchestration Deactivate physical server does not complete successfully after CMS crash recovery (page 51) Matrix infrastructure orchestration Deactivate Virtual Servers Request does not complete successfully after a CMS crash recovery (page 52) Matrix infrastructure orchestration Add Disk request for an ESX virtual server does not complete successfully after CMS crash recovery (page 52) VM logical server is incorrectly activated after a CMS crash recovery (page 70) mxagentconfig command fails with SSH protocol error on Integrity managed node (page 52) Using or configuring managed systems A local user or domain account becomes locked out (page 28) Install OpenSSH on Windows 2008 managed node through Configure or Repair Agents fails (page 28) WBEM/WMI authentication and communication issues (page 28) Server blade serial number is not added on ESX or Hyper-V managed nodes (page 32) Insight Control virtual machine management registration for ESX or Hyper-V nodes fail (page 32) Using multipath I/O with QLogic QMH2462 4Gbps Fibre Channel HBA and Intel-based HP G6 BLc servers (page 92) Using multipath I/O with Emulex Fibre Channel HBA and HP servers (page 93) Servers configured with system ROM version may not install the operating system to the SAN (page 94) Using VM hosts and guests IDE disk on a VMware ESX VM fails to reactivate (page 33) Matrix infrastructure orchestration service creation fails in the target reconfiguration phase (page 36) Matrix infrastructure orchestration service creation fails with Error powering on logical server (page 37) Matrix infrastructure orchestration service creation fails in the VM customization phase (page 37) Synchronizing the clocks on the hosts and VMs (page 67) VM logical server is incorrectly reflected in the Logical Server Status (page 69) VM logical server is reflected as Active on unknown host while performing an activate operation (page 69) VM logical server is reflected as Active on unknown host while performing a move operation (page 70) VM guests appear without system addresses in HP SIM (page 85) Using HP Virtual Connect Flex 10 with VMware ESX (page 85) Operations related to Hyper-V guests (page 87) Linked clone provisioning fails with concurrency set to 10 on Hyper-V (page 87) Hyper-V R2 hosts may hang while undergoing rapid VM creation/deletion operation (page 87) If the IP address of the host changes, virtual machine management cannot manage the host (page 88) Using this document 7

8 Cannot register vcenter in HP SIM virtual machine management (page 88) VM names do not display correctly in the Insight Control virtual machine management user interface (page 88) Host/Guest control buttons disabled (page 89) Additional troubleshooting for Integrity VM hosts and guests Unable to activate an Integrity VM logical server due to a disk creation failure (page 53) Unable to import an Integrity VM logical server (page 71) Unable to create an Integrity VM logical server (page 71) Logical server boot errors when booting as an Integrity VM guest (page 73) Editing an Integrity VM outside of logical server management results in an error with the storage port (page 74) Networking error occurs when attempting to activate a logical server on Integrity VM (page 74) An Integrity VM target host is rejected when performing a P2P move of a VM host (page 75) Importing an Integrity VM guest fails with no boot path (page 75) EFI boot paths are empty after adding an LFBP and then performing a V2P move (page 76) Access failure occurs when attempting to invoke Integrity Virtual Server Manager (page 79) Failure starting an Integrity VM (page 79) Failure to modify the Integrity VM vcpu entitlement (page 80) Integrity VM information is out of date, incomplete, or missing (page 80) Deleted Integrity VM still appears in Matrix OE visualization (page 80) Using logical servers VM snapshot creation fails (page 56) Portable logical server must not include local storage (page 63) Physical disks with multiple partitions are unsupported unless all but the first partition is excluded (page 63) Local disk support is only available through Matrix infrastructure orchestration (page 63) Do not unmanage a logical server that has been previously activated (page 63) Unmanaging a logical server leaves WWNs used by VC profile unavailable (page 64) Storage pool entries marked Inoperable or available storage is not available (page 64) Storage pool entry status is Presentation Completed with Errors Inoperable or Presentation Completed Inoperable (page 65) Matrix infrastructure orchestration pauses for manual zoning information on auto-generation storage (page 65) Unable to update storage pool entry after deleting service request in SPM (page 66) Recovery after logical server operation failures (page 66) Recovering an inoperable logical server (page 67) Logical server fails to power on when logical server is activated with power on (page 69) VM host does not appear on move logical server target screen (page 70) Failure during VM deletion leaves active but unpowered logical server (page 76) 8 About this document

9 Deactivating or powering off logical server causes license error check (page 77) LSA service may run out of memory on large scale systems (page 77) Profile is created but not associated with logical server when service is stopped (page 78) Logical server is missing from Matrix OE visualization (page 78) VM logical server displayed with incorrect association (page 84) Managing storage Communication errors occurred between SPM and Matrix (page 96) Cannot import 3PAR StoreServ Systems (page 97) Matrix infrastructure orchestration service fails to auto-generate multi-path storage (page 97) No matching candidates found for this volume (page 98) Port conflict when HP P6000 Command View Software runs on Matrix OE CMS (page 99) NPIV/vport storage not available with FlexFabric Converged Network Adapters (page 100) Using Matrix recovery management Matrix recovery management Activate or Deactivate job fails (page 60) Matrix recovery management change local site job failed (page 61) Failure in storage replication failover (page 62) Logical server not located during failover job (page 62) Failover job succeeds, but logical servers are not activated (page 62) Failover job fails because of insufficient servers (page 62) Using deployment and provisioning Unable to add servers in a SCVMM service (page 33) HP Matrix infrastructure provisioning and linked clone provisioning fails when using a Windows 2012 OS template on ESXi 5.0 Update 1 (page 34) Password is blank for Windows 2008 R2 and above provisioned VMs (page 35) Microsoft SCVMM templates do not appear in the infrastructure orchestration software tab (page 35) Deleting VM entry from Microsoft SCVMM fails (page 36) Physical server provisioning using Insight Control server deployment fails during the final personalization step (page 40) Matrix infrastructure orchestration service creation fails for physical requests where Windows Server OS is selected (page 40) Provisioned VMs fail due to improper VM host hardware configuration (page 41) Unable to allocate servers and networks in the same VCDG (page 42) Matrix infrastructure orchestration request fails due to server use outside of infrastructure orchestration (page 43) Automated OS provisioning to a physical server fails to successfully PXE boot (page 43) Request History shows deployment server does not have a record of the target system(s) (page 43) Matrix infrastructure orchestration deployment task fails, exceeding its timeout value (page 44) Using this document 9

10 Matrix infrastructure orchestration cannot deploy Integrity VM guests when gwlm is controlling the host (page 45) Manually cleaning up Clean-me logical servers (physical) (page 46) Manually cleaning up Clean-me logical servers (virtual) (page 47) Defined SPM storage template does not appear in infrastructure orchestration designer (page 49) Insight Control server deployment software is not listed in infrastructure orchestration (page 55) PXE boot to Insight Control server deployment fails (page 89) PXE error: No boot filename received (page 90) Install MPIO EVA DSM or Install MPIO XP DSM job hangs (page 92) Server deployment failure in a multipath I/O environment (page 92) Server Automation fails due to boot order changes (page 94) Linked clone provisioning of a VM template containing shared disks fails (page 56) VM instances not deleted after Matrix infrastructure orchestration fails (page 56) Add disk operation is not working with SPM in VM cluster environment (page 57) Provisioning of Matrix service fails in customization (page 95) Failure when creating or deactivating a High Availability Hyper-V virtual machine with RDM/pass-through cluster disks (page 95) Hyper-V provisioning fails on Hyper-V 2012 and Hyper-V 2012 R2 with timed out errors (page 89) Additional troubleshooting for Ignite-UX Ignite-UX is missing as a software source (page 57) Ignite-UX deployments do not begin (page 57) Ignite-UX deployments fails with PXE error (page 58) Integrity blade provisioning fails during the Ignite-UX installation phase (page 58) Request History shows an unexpected error has occurred on an Integrity blade (page 59) Error in Request History when erasing Integrity logical server disks (page 59) Unable to power off device bay occupied by an Integrity blade because the operation timed out (page 60) Using Global Workload Manager Unable to build a single SRD (page 81) Old host name not changed (page 81) Unable to determine the network address and/or hostname of the current host (page 81) Cannot create an SRD due to a firewall between managed nodes (page 82) Managed system unable to rejoin an existing SRD (page 83) Agent will not restart on reboot (page 84) HP Matrix KVM Private Cloud troubleshooting VM guests cannot be provisioned on an activated host (page 101) 10 About this document

11 Time and date on the management appliance do not match time and date on the NTP server (page 101) Using this document 11

12 2 Troubleshooting tools To begin troubleshooting, perform the following tasks: Verify that you have the supported versions of hardware, software, and firmware. For additional information, see the HP CloudSystem Matrix Compatibility Chart at matrixcompatibility. Review known issues. For information, see the HP CloudSystem Matrix Release Notes at Check for any existing CloudSystem Matrix Customer Advisories that may pertain to your environment. Sign up to receive advisories at Select Solutions as the Product Category and CloudSystems as the Product Family and then choose the specific versions for which you want to receive advisories (such as CloudSystem Matrix 7.x ). It is important that you only follow or apply the recommendations of a CloudSystem Matrix advisory for the version of CloudSystem Matrix that you are running so that your system does not get out of sync with the supported software/firmware recipe. Ensure that DNS is configured for all managed systems. Without DNS configuration, issues can occur, such as Matrix OE logical server management failure and Matrix infrastructure orchestration deployment failure. Ensure time synchronization was performed among the CMS, managed systems (including ESX hosts), VMware vcenter Server, and System Center Virtual Machine Manager. Failure to do so results in errors in both vcenter and Insight Management. Ensure that you have configured VMware vcenter Server and System Center Virtual Machine Manager settings in HP SIM, including the credentials and URL for vcenter. Without this configuration, issues can occur, including infrastructure orchestration VM provisioning failure. After these initial steps, use the tools described in the following sections to identify and correct potential issues. Backing up your environment HP recommends that you back up your system before making changes to your environment including: Configuration changes Additions or upgrades to the hardware or system software Troubleshooting one or more problems Being able to go back to a known state minimizes the risks involved with making changes. The CMS software should be backed up on a regular basis following the procedures described in the white paper Backing up and restoring HP Insight Management 7.0 Central Management Server (Windows) at Back up the following components of the CloudSystem Matrix solution when making configuration changes or backing up the CMS: OA VC domains SAN switches Matrix infrastructure orchestration networking subnet XML files Matrix infrastructure orchestration provisioning templates Customized OO workflows HP recommends backing up the OA, VC domains, and SAN switches while the CMS is not being used (such as during a full or partial backup window while the CloudSystem Matrix services are 12 Troubleshooting tools

13 shut down). These backup files are valuable if you need to reconstruct the environment. Include the IO networking subnet XML files, the IO provisioning templates, and the customized OO workflows in the CMS partial backup file set. If the environment changes frequently, perform partial backups between CMS backups. Using the Insight managed system setup wizard The managed system setup wizard is useful for troubleshooting problems, including: Licensing issues Agent installation and configuration issues SSH configuration issues The managed system setup wizard automatically determines the configuration needed for selected features. Some of these steps are manual, but using the Wizard, most are performed automatically. The managed system setup wizard eases configuration of systems for many Insight Management products and features. After you successfully complete the managed system setup wizard, the selected systems are configured with the management features as specified. The managed system setup wizard does the following: Configures agents necessary to enable the management features Applies software licenses as needed Completes any additional configuration steps You can run the managed system setup wizard at any time to change the configuration or management features of your systems. Run the wizard to configure a managed system each time its operating system is installed, updated, or changed. Also, use this wizard to diagnose configuration issues with your systems. Before running the Insight managed system setup wizard Before running the managed system setup wizard to configure and manage systems, you must have successfully discovered all systems in HP SIM and have licenses available for all management features. In addition: The target systems must be discovered with valid credentials stored in HP SIM. The target systems must have valid credentials stored within HP SIM Enter credentials during the Set Credentials step of the wizard. You must provide software licenses for all management feature products as required. Licenses may be needed in the form of either 5x5 keys or Software Distributor license bundles (for HP-UX). For 5x5 keys, the managed system setup wizard presents a dialog for entering license keys. You also can add 5x5 keys using the License Manager (in HP SIM, select the Deploy License Manager +Add Licenses). NOTE: You must install required HP-UX Software Distributor license bundles on HP-UX managed nodes. Insight managed system setup wizard scans for licenses installed on the selected HP-UX target systems. Some products also provide trial licenses that enable using the product for a short period of time before purchasing a license. Running the Insight managed system setup wizard You must authorize the System Setup toolbox authorization in HP SIM on each managed system being configured using the managed system setup wizard. Using the Insight managed system setup wizard 13

14 To add licenses using the managed system setup wizard, you must select Options Security Users & Authorizations, and then enable the option User can configure CMS security access such as creating, modifying or removing other users. Only one instance of the managed system setup wizard can run at a time in HP SIM. To start the managed system setup wizard, select Configure Managed System Setup Wizard... in HP SIM as shown in Figure 1 (page 14). If target systems are selected in HP SIM, you are prompted to verify the target systems before the wizard begins. Otherwise, you are prompted to select the target systems. Figure 1 Starting the managed system setup wizard The managed system setup wizard introduction page gives an overview of the wizard. Using vseassist Use cases The software tool vseassist, available from the HP SIM GUI or the vseassist command on the command line, verifies that Matrix Operating Environment components are installed, configured, and operating correctly on the CMS. This reduces the time to fix the most common problems. Also, vseassist verifies communications between the CMS and selected managed nodes to: Ensure that communications paths are available. Exercise select providers and agents. Ensure that providers and agents produce expected results to selected tests. In addition, vseassist performs basic network configuration checks to ensure that there are no issues resolving the CMS host name or IP address. Finally, the vseassist tool verifies the configuration and operation of managed systems by executing vseassist on the managed node. Execute vseassist from the HP SIM graphical user interface, from the command line on the CMS, or from the command line on a managed node. You can use the vseassist tool to check the configuration and operation of the software in three ways: Verify the CMS configuration. Verify the communication between the CMS and selected managed nodes. Verify the configuration of selected managed nodes. This can be performed remotely from the CMS or directly on a managed node, as shown in Figure 2 (page 15). 14 Troubleshooting tools

15 Figure 2 CMS to managed node communication Running the vseassist tool HP recommends the following diagnostic procedure to troubleshoot Matrix OE. 1. If you have administrative access to the CMS, execute the following from the command line: <VSEINSTALLDIR>\bin\vseassist a where <VSEINSTALLDIR> is the location of the installation. The default install location is C:\Program Files\HP\Virtual Server Environment. As an alternative to running the vseassist a command which runs all verifications, you can select vseassist from the graphical user interface. This command verifies the following information: Matrix OE management software configuration on the CMS Communication with all systems in the All Matrix OE Resources collection Matrix OE managed node software configuration on managed nodes The vseassist -a command may take multiple hours to complete. The time consumed is dictated by the number of managed nodes and their state. Improperly configured systems will take longer. 2. After running this command, see the summary output and examine the log file for any issues. The log file is located at: C:\Program Files\HP\Virtual Server Environment\logs Follow the resolution steps in the log to correct any issues. Check the configuration of the CMS, managed nodes, and CMS to managed node communication using the following steps: 1. Resolve all issues with the configuration of the CMS by performing one of the following steps: From the HP SIM graphical user interface, select Diagnose Troubleshoot Matrix Operating Environment Check CMS Configuration Using vseassist 15

16 Alternatively, on the CMS command line enter: vseassist -c As the vseassist diagnostic tool is running, the results of each check will be displayed on the screen. Figure 3 (page 16) shows a CMS configuration verification. Figure 3 CMS configuration verification example 2. After you have resolved all issues with the CMS configuration, find any communication issues between the CMS and managed nodes by performing one of the following steps: From the HP SIM graphical user interface, select the managed nodes, then from the Diagnose menu, select Troubleshoot Matrix Operating Environment Check CMS to Managed Node Communication. Alternatively, from the CMS command line, specify the HP SIM managed node names or system collection names to be tested using the following command: vseassist -l { -n <nodename> -g <collectionname>...} 3. After you have resolved the communication issues between the CMS and managed nodes, verify the Matrix OE software configuration on managed nodes by performing one of the following steps: From the HP SIM graphical user interface, select the managed nodes, then from the Diagnose menu, select TroubleshootMatrix Operating Environment Check Managed Node Configuration Alternatively, from the CMS command line, specify the HP SIM managed node names or system collection names to be tested using the following command: # vseassist -r { -n nodename -g collectionname...} 16 Troubleshooting tools

17 Diagnostics in vseassist This section describes the types of diagnostics performed by the vseassist tool. Windows CMS checks You can perform checks of the CMS configuration in two ways: Select Diagnose Troubleshoot Matrix Operating Environment Check CMS Configuration from the menu. Enter vseassist -c on the command line. The vseassist tool performs the following checks of the CMS configuration when invoked using the Check CMS Configuration menu or the vseassist c command: HP SIM running HP SIM memory configuration Matrix OE running Local host name resolution Application Discovery server running File system free space Java JRE installed HP Insight Control virtual machine management running HP Insight Control virtual machine management responsive Virtual Connect Enterprise Manager responsive If Matrix infrastructure orchestration is installed on the CMS, the vseassist tool automatically invokes an additional tool called ioassist to check the configuration of Matrix infrastructure orchestration. Using ioassist, these additional checks are performed: Sysprep files for virtual software customization exist HP SIM running and responding VM hosts and physical servers available and visible to infrastructure orchestration Deployable software available and visible to infrastructure orchestration Deployment servers available and visible to infrastructure orchestration Virtual Infrastructure servers available and visible to infrastructure orchestration Database running and responding Matrix infrastructure orchestration Windows groups exist The Matrix infrastructure orchestration Administrators user is authorized for infrastructure orchestration toolbox in HP SIM Networks available and visible to infrastructure orchestration Networks configured and ready for infrastructure orchestration usage Matrix infrastructure orchestration logical server management running and responding HP Operations Orchestration installed and configured for Matrix infrastructure orchestration Matrix infrastructure orchestration workflows imported into Operations Orchestration Insight Control virtual machine management running and responding Storage pools available for each existent VCDG Matrix infrastructure orchestration user pools exist and available as resources Using vseassist 17

18 Matrix infrastructure orchestration user access to infrastructure orchestration user pools Matrix infrastructure orchestration templates available and published for infrastructure orchestration users Managed node checks When you invoke the vseassist tool by selecting the Check Managed Node Configuration menu item or entering vseassist r on the command line, vseassist performs the following checks of managed node configurations: Windows managed nodes CMS host name resolution WMI running WMI responsive System type matches HP SIM system type Compare CMS and managed system nslookup SSH configuration validation File system free space Windows 2003 SP2 Installed Compare CMS and managed node time Utilization Provider validation Linux managed nodes CMS host name resolution HP Web-Based Enterprise Management running WBEM responsive System type matches HP SIM system type Compare CMS and managed system nslookup SSH configuration validation File system free space Check VMware ESX Version Compare CMS and managed node time HP-UX managed nodes CMS hostname resolution System type matches HP SIM system type Virtual machine WBEM provider validation npartition WBEM provider validation npartition name valid 18 Troubleshooting tools

19 Virtual partition WBEM provider validation Complex name valid Virtual partition naming conflicts LDAP/WBEM compatibility Compare CMS and managed system nslookup SSH configuration validation Application Discovery agent installed Managed system required patches installed File system free space Java JRE bundle installed Fair-Share Scheduler installed Processor Sets installed HP gwlm agent configuration icap status Communication checks Check Integrity Virtual Server Manager and VM Provider conflicts Compare CMS and managed node time HP WBEM status checks: Checking WBEM Services installation Checking CIM server status Checking WBEM ports Checking WBEM cimsrvr UNIX user Checking WBEM cimsrvr UNIX group Checking SSL certificate permissions Checking SSL certificates Checking CIM Server response Checking WBEM files and directories When you invoke the vseassist tool by selecting the Check CMS to Managed Node Communication menu item or typing vseassist l on the command line, vseassist performs the following checks of communication between the CMS and managed nodes: Managed system host name resolution WBEM port open WBEM authentication SSH port open SSH authentication Using vseassist 19

20 SNMP port open Trial or permanent license applied Insight Control virtual machine management accessibility Matrix OE Capacity Advisor data collection Check VMware ESX version Using System Management Homepage troubleshooting tools Running Configure or Repair Agents The Configure or Repair Agents feature enables you to configure or repair agents in Windows, Linux, HP-UX, and ESX hosts. The Configure or Repair Agents tool enables you to install agents or repair the communication problems between HP SIM and the systems you are managing. You can start the Configure or Repair Agent by selecting Configure Configure or Repair Agents...from the HP SIM menu as shown in Figure 4 (page 20). Figure 4 Configure or repair agents The Configure or Repair Agents feature adds the security and trap community strings and trust settings to the target systems, but it does not replace existing settings. Use the Replicate Agent Settings feature in HP SIM to replace existing settings and other settings on target systems. Use the Configure or Repair Agents tool to send test SNMP traps from Windows and Linux systems with Insight Management Agents and send WBEM test indications from Windows and HP-UX systems with WBEM providers installed. You can also configure WBEM certificates for HP-UX systems and WBEM/WMI users for Windows systems with WBEM Providers for Windows Server 2003 and Windows Server You can also use the Configure or Repair Agents tool to register a VM host with Insight Control virtual machine management. The Configure or Repair Agents feature on a Windows CMS also enables you to install various agents and providers on an HP ProLiant or HP Integrity system with a Windows operating system, including: HP Web-Based Enterprise Management Providers for Windows Server 2003 and Windows Server 2008 OpenSSH HP VCA for Windows SNMP Insight Management Agentss for Windows 20 Troubleshooting tools

21 To run Configure or Repair Agents against a system, you must have authorization to run the Configure or Repair Agents tool. You must have administrator privileges for Windows operating systems on the target systems to configure or repair the agent settings. You must have root privileges for HP-UX and Linux systems. The SSH protocol is required to install the agent software to support the visualization and configuration features of Matrix OE visualization and to collect utilization data for Matrix OE Capacity Advisor. If your environment precludes installing OpenSSH on the Windows server blades, see the white paper HP Matrix Operating Environment 7.0 software without OpenSSH on HP ProLiantand Integrity Servers Running Microsoft Windows at bc/docs/support/supportmanual/c /c pdf. If you select Configure SSH access, you must select one of the following options: Host based authentication for SSH For this option to work, the user name and password provided must be an administrative level account. For Linux or HP-UX targets, it must be the root account and password. User based authentication on the managed system If you do not want all users that have login access to HP SIM to run the tool and you would like to control which users need to have access, this option is more secure. You can configure SSH only if the OpenSSH service is running on the managed systems. You can install OpenSSH on Windows operating systems by running Install Open SSH from Configure or Repair Agents or by selecting Deploy Deploy Drivers, Firmware and Agents Install Open SSH. Using Manage Communications Sometimes running the Configure or Repair Agents tool is not sufficient. To troubleshoot communication problems between the CMS and targeted systems, use the Manage Communications feature of HP SIM at Configure Manage Communications as shown in Figure 5 (page 21). Figure 5 Manage Communications For each failed communication function, troubleshooting information is available. You can reconfigure certain communication settings, launch agents, and push certificates to target systems. For each selected target system, Manage Communications displays the following information: Communication status Manage Communications table Identification Identification status Using System Management Homepage troubleshooting tools 21

22 Events Run Tools Version Control System type Operating system name Ability to receive events Ability to run tools Availability of software and firmware inventory data Manage Communications main page buttons Advise and Displays diagnostic results and troubleshooting tips Repair Quick Repair Update Print Launches Configure and Repair Agents tool Refreshes communication status Creates printer-friendly version of lists in new window The communication status between the CMS and the targeted systems appears in the table. Place your mouse over a status icon to display available information explaining the status. If you click the status icon, the Advise and Repair section appears with the corresponding tab open. For example, if you click a status icon in the Run Tools column, the Run Tools tab opens in the Advise and Repair section instead of the default Identification tab. System Name This column contains the actual system name of selected systems. When you place the cursor over the system name, the FQDN appears, which helps differentiate between two or more systems that share the same system name. Identification The Identification column includes status information on the state of an identification process. Identification attempts to determine the system type, the management protocols a system supports using the configured credentials, and the operating system and version loaded, and other basic system attributes. The Identification column also indicates if the system is associated with another system. The status in this column is a roll-up of the status of all available protocols. The statuses are analyzed based on the status itself, the system type, the system operating system, and instrumentation options. Status icons do not appear for systems that cannot have a status, such as, complexes, enclosures, and racks. The identification status updates each time an identification task runs. Events The Events column indicates if the CMS can receive events from the target systems. This status reflects the sending of SNMP traps and WBEM indications. Run Tools The Run Tools column indicates if the CMS can run tools locally on target systems, such as HP System Management Homepage. Communication issues in this column usually relate to security and trust relationships. Version Control The Version Control column indicates the availability of the software and firmware inventory data for target systems. The status is collected and stored during data collection. This is only valid for target systems with a version control agent installed. 22 Troubleshooting tools

23 System Type The System Type column displays the system type, for example, Server or Desktop. The Unmanaged system type indicates systems that have no management protocol that HP SIM can detect, for example, systems with no SNMP, WBEM, DMI, or SSH. The Unknown system type indicates that none of the built-in or STM tasks could identify the system. However, some management protocol was detected on the system. Checking log files The following log files can help you understand system activities: Matrix infrastructure orchestration C:\Program Files\HP\Matrix infrastructure orchestration\logs hpio-controller.log Rolling logs Search for WARN, ERROR hpio.log Service startup Shutdown Logins Matrix OE C:\Program Files\HP\Virtual Server Environment\logs hp_lsa_service.log Service starts Stops vselog.lsm.log Matrix recovery management See the following log files for useful recovery management troubleshooting information: lsdt.log and lsdt_web.log in the logs directory where recovery management is installed on the system clxevarun.log in STORAGE/EVA/log where recovery management is installed on the system clx3parrun in Program Files\Hewlett-Packard\Cluster Extension 3PAR\log directory For a list of log files to use for troubleshooting HP P9000 storage failover issues, see the HP P9000 Cluster Extension Software Administrator Guide available at h20000.www2.hp.com. Click Manuals, then go to Storage Storage Software Storage Replication Software HP Cluster Extension Software. For a list of log files to use for troubleshooting HP 3PAR StoreServ Storage failover issues, refer to the HP 3PAR Cluster Extension Software Administrator Guide available at h20000.www2.hp.com. Click Manuals, then go to Storage Storage Software Storage Replication Software HP Cluster Extension Software. HP-UX installation logs Ignite server logs /var/opt/ignite/logs/* Managed system logs /var/adm/sw/*.log Checking log files 23

24 LSM utility logs LSMUTIL is a script that you can run from the command line or from a DOS bat file. LSMUTIL reads information from the logical servers tables in the Matrix OE database. Based on the option you select, the script exports information about logical servers, storage pool entries, or managed resources. For a list of logical server, enter lsmutil list ls xml file ls.log For a list of storage pool entries, enter lsmutil list spe xml file spe.log For a list of managed resources, enter lsmutil list -cr xml file cr.log VMware logs HP recommends checking the VMware logs on the ESX hosts, which are managed by HP SIM. VMware logs are located on the ESX server at /var/log/vmware/hostd.log. For information about uploading the VMware log files, see the VMware Knowledge Base article Collecting diagnostic information for VMware Infrastructure SDK at the following website: To collect VMware log files, execute the vm-support a command. Insight Control virtual machine management Event viewer logs Insight Control virtual machine management provides the following event logs from system drive\windows\system32\winevt\: Logs\Application.evtx Logs\System.evtx Management logs Insight Control virtual machine management provides the following management logs from C:\Program Files\HP\Insight Control virtual machine management\log\: vmm_controller.log hpvmmsvc.log Rolling logs Search for WARN, ERROR hpvmm.log Service startup Shutdown HP SIM C:\Program Files\HP\Systems Insight Manager\logs mxdomainmgr.0.log Mx.log HP P6000 Command View Software Events track the progress of actions, both normal and exceptional, that occur on the array. Examples of normal events are creating virtual disks or initializing an array. Examples of exceptional events are an incomplete function or a reduction in array capabilities. Normal events are more common than exceptional events. 24 Troubleshooting tools

25 You can use HP P6000 Command View Software to configure and view these events. Managing events from the HP P6000 Command View Software user interface is especially useful when you are monitoring multiple EVAs. Events are captured in one or more of the following logs: Management server event log A management server is the installation of HP P6000 Command View Software on a management server. Management server events are triggered by the following: Responses to user actions in the user interface Actions initiated through the HP P6000 Command View Software API Controller event log A controller event is a normal or exceptional action on any hardware component or logical object within the array. Examples of controller events include: Disk or disk enclosure changes Configuration changes (such as creating a disk group) Controller reboots Changes detected by the environmental monitor unit within the array. An example of a normal event would be the controller reporting that creation of a virtual disk has completed. An example of an exceptional event would be a disk enclosure that has lost communication on a Fibre Channel loop. Controller termination event log Controller termination events report that a controller has ceased operation, but do not report on controllers in uninitialized arrays. All event logs are stored in the following locations: C:\Program Files\Hewlett-Packard\Sanworks\Element Manager for StorageWorks HSV\cache\ directory on the management server C:\Program Files\Compaq\Sanworks\Element Manager for StorageWorksHSV\cache\ directory on the SMA. Storage Provisioning Manager SPM maintains an operational log for the storage catalog, listing all operations performed and all error events. All users can review the operational log using the following procedure: 1. Open a browser window and enter: or 2. Under Operational Logs, click the appropriate file name. The file opens. Operational log entries can be copied for pasting content into another window on the desktop. NOTE: The audit logs can also be accessed from the HP Storage Provisioning Manager log and contain all operations and permissions events. SPM logs can also be found in C:\Progarm Files (x86)\hp\storage Provisioning Manager\logs. By default, logging is set to a fairly high level. If there is insufficient detail in the log files and the problem is reproducible, you may wish to enable a lower level of SPM logging. To lower the logging level and include more detailed information, perform the following steps: Checking log files 25

26 1. Navigate to: C:\Program Files (x86)\hp\storage Provisioning Manager\ state. 2. Using your favorite text editor, open the file ApplicationConfig.xml. 3. Find the XML line <LogEntryThreshold>Informational</LogEntryThreshold> 4. Change Informational to Tracing. 5. Save the file and exit the editor. The setting takes effect immediately. Global Workload Manager gwlm logging mechanisms The name of the current log always ends in.log.0. Once this file grows to a certain size, it is moved to a filename ending in.log.1 and a new.log.0 file is started. If a.log.1 file already exists, it is renamed.log.2. If a.log.2 file already exists, it is overwritten. gwlmcmsd daemon or service HP-UX: /var/opt/gwlm/gwlmcmsd.log.0 Windows: C:\Program Files\HP\Virtual Server Environment\logs\ gwlmcmsd.log.0 gwlmagent daemon HP-UX: /var/opt/gwlm/gwlmagent.log.0 Windows: Not applicable gwlm interface in HP SIM HP-UX: /var/opt/gwlm/gwlm.log.0 Windows: C:\Program Files\HP\Virtual Server Environment\logs\ gwlm.log.0 gwlm command HP-UX: /var/opt/gwlm/gwlmcommand.log.0 Windows: C:\Program Files\HP\Virtual Server Environment\logs\ gwlmcommand.log.0 gwlm user-configurable log file size and number of log files Specify the size (in MB) and number of files to use for logging. For a single file of unlimited size, set logfilesize to negative one (logfilesize = -1). Otherwise, total log file size is logfilesize * lognfiles. By default, the log file size is limited to 20 MB and the number of log files is limited to 3. To change the default log file size and the number of log files, change the following properties: com.hp.gwlm.util.log.logfilesize = 20 com.hp.gwlm.util.log.lognfiles = 3 For the gwlmagent log, change the values of these properties in /etc/opt/gwlm/conf/ gwlmagent.properties. For all the other log files, change the values in /etc/opt/ gwlm/conf/gwlmcms.properties on HP-UX and in C:\Program Files\HP\ Virtual Server Environment\conf\gwlmcms.properties on Windows. 26 Troubleshooting tools

27 Useful gwlm commands gwlm list When run on the CMS, this command provides all the object s policies, workloads, and deployed SRDs. gwlm monitor CLI for monitoring; if the GUI is not operating properly, this command shows if the managed nodes and gwlmcmsd are working. gwlm agentinfo Displays an agent's host, gwlm version, SRD, and CMS gwlmstatus This command runs on a managed system and checks the health of the system and SRD. Checking log files 27

28 3 Troubleshooting procedures Agent and communication troubleshooting Install OpenSSH on Windows 2008 managed node through Configure or Repair Agents fail When using Configure or Repair Agents to Install OpenSSH onto a managed node with Windows Server 2008 installed, the installation fails if you enable the UAC setting and you use a domain account for the credentials. The installation works if you use the local admin (root) account for the managed node. Disable UAC on the managed node. For information, see the Microsoft TechNet article User Account Control Step-by-Step Guide available at the following website: Use the local admin account of the managed node for the credentials. A local user or domain account becomes locked out When running discovery or identification tasks in HP SIM, a local user or domain account becomes locked out. The credentials provided to HP SIM are incorrect. If using system credentials, verify that the credentials are correct for the managed nodes (sign-in credentials are used for WBEM/WMI access). 1. Open a command prompt on the CMS, and enter mxnodesecurity -l -p <sign-in> to see the names of the devices that have credentials manually set. If any of the managed node server names or IP addresses appear, then credentials have been manually set on the system and could potentially be incorrect. 2. In the Search box in HP SIM, type the name of the server in question from the command line output, and then access the System page for that device by clicking on the name. 3. Click Tools and Links, and then click System Credentials under the HP SIM Pages section. 4. Select the checkbox in the gray area to select all credentials, and then click Edit system credentials. 5. In the Sign In credentials section, if there is a user name listed, it was manually configured by a user. If there are no credentials listed, either the Discovery Task or Global Credentials are being used. 6. You can manually change the credentials by editing them and then clicking OK, or you can remove the username and password (and confirmation) data, and then click OK. If you remove the system credentials, HP SIM uses the Discovery (recommended) or Global credentials information, if configured. WBEM/WMI authentication and communication issues s 28 Troubleshooting procedures

29 HP SIM cannot connect to the WBEM/WMI data on a managed node. This causes Matrix OE visualization to display warnings on the utilization meters and prevents Matrix OE Capacity Advisor from collecting utilization data from the managed systems. Matrix OE Capacity Advisor data collection events reports a WBEM communication error such as: Utilization data collection failed because of a failure to connect to WBEM/WMI on the managed node. Discovery tasks report invalid WBEM/WMI credentials. Vseassist reports a WBEM authentication failure. WMI Mapper settings are incorrect. The WBEM provider is not properly installed or configured on the managed node. The credentials provided to HP SIM are incorrect. 1. Verify that the WMI Mapper settings are correct: a. In HP SIM, verify that the WMI Mapper IP address is configured correctly in Options Protocol Settings WMI Mapper Proxy. The WMI Mapper IP address is the IP address of the CMS and must reside on a network subnet that has connectivity to all the managed systems. The host name shown must be the FQDN associated with that IP address. If the IP address is incorrect or does not match the host name, delete the WMI Mapper Proxy setting and enter a new one. b. Determine the FQDN by which HP SIM knows the CMS. In HP SIM, use the Search tool to find the CMS and mouse over the System Name to see the FQDN known to HP SIM. If the CMS FQDN does not match the host name shown in the WMI Mapper Proxy configuration screen, re-discover the CMS using the FQDN or IP address of the WMI Mapper: i. Navigate to Options Discovery. ii. Create a new discovery task for the CMS using the FQDN or IP address of the WMI Mapper. iii. Run the task. c. In the HP SIM All Systems view, select the CMS, and then select Options Identify Systems to run the Identify Systems task against the CMS. In the Task output, verify that the following message appears: Running WBEM rules based identification. Got ComputerSystem WBEM/WMI data from the system. This validates that the CMS is communicating to the WMI Mapper. 2. Ensure that the WBEM provider is not missing from the managed node. Run vseassist l n nodename (or select the managed systems from the HP SIM user interface, and then from the Diagnose menu, select Troubleshoot Matrix Operating Environment Check CMS to Managed Node Communication). If vseassist reports a WBEM authentication failure for the CMS, ensure that the FQDN by which HP SIM knows the CMS corresponds to the IP address of the WMI Mapper: Determine the FQDN by which HP SIM knows the CMS: In HP SIM, use the Search tool to find the CMS and mouse over the System Name to see the FQDN known to HP SIM. Determine the WMI Mapper settings: In HP SIM, navigate to Options Protocol Settings WMI Mapper Proxy. Agent and communication troubleshooting 29

30 If a mismatch is determined, either reconfigure the WMI Mapper or re-discover the CMS using the FQDN or IP address of the WMI Mapper. For example, in HP SIM, navigate to Options Discovery. If necessary, create a new discovery task for the CMS using the FQDN or IP address of the WMI Mapper and run that task. 3. If the target system has been identified as WBEM enabled, but the credentials are failing for the target system, perform the following steps: Verify the credentials and identify the target system again. The password might be incorrect on the System Credentials page. To access, select Options Security Credentials System Credentials. Communication with HP SIM server has been lost s The HP SIM portal relies on the CMS to respond quickly to all requests. Browser performance degrades when CMS responses are delayed. The browser appears to lock up completely when CMS responses are delayed. The browser displays the message, Communication with the HP SIM server has been lost, even though the CMS might be responsive to other browser sessions. This could be a temporary condition due to CMS or network congestion, or CMS restart. Retrying after a short period might resolve the issue. On a LAN, the browser receives a CMS response to most requests within 10 to 100 milliseconds. Requests that involve database queries or secondary network communication might take a few seconds to respond. There are situations that might result in particularly slow response times. For example: Viewing large collections of systems or events Specific and custom database queries taking an unexpectedly long time Many users simultaneously accessing a shared resource, such as the database Network delays Pages that retrieve data using WBEM or SNMP Webagent and display the results (for example, System Properties) can take a long time, especially when the requests are destined to time out. By default, web browsers (such as Microsoft Internet Explorer 7 and earlier and Mozilla Firefox 6.x and earlier) limit themselves to two simultaneous connections to a single web server. If two requests are outstanding, any additional requests wait until one of the current connections completes. This is by design and is in agreement with the HTTP 1.1 specification. Increase the maximum number of server connections in both Internet Explorer and Firefox. Though you can increase this number into the hundreds, HP recommends that you do not exceed 10 connections. For more information, see: How do I configure Internet Explorer to download more than two files at one time? available at: For Firefox's Network.http.max-persistent-connections-per-server available at: kb.mozillazine.org/network.http.max-persistent-connections-per-server Restart the HP SIM service by restarting HP SIM in the Windows Services interface on the CMS. Notes: 30 Troubleshooting procedures

31 Other reasons why the browser might lose the connection with the CMS include the following: The CMS is shut down or restarted. The network connection between the browser and CMS is broken. If the system running the browser hibernates (or sleeps), it loses connection with HP SIM. If the browser is going through a web proxy, the proxy may be obtrusively slow to respond at some point over an extended period. For more information, see the following: HP SIM online help (troubleshooting) HP Systems Insight Manager User Guide at sim/docs No buffer space available, maximum connections reached A CMS server running Windows Server 2008 R2 with infrastructure orchestration generates the following error while attempting to open a new socket connection: java.net.socketexception: No buffer space available (maximum connections reached?) This error can occur after approximately 20 days or more of continuous operation. If this error occurs, additional sockets cannot be opened causing failures and preventing the Matrix OE software stack from functioning. NOTE: This is a known issue with certain Microsoft operating systems. Although this has been seen in Matrix OE, it has also been reported in other (non-hp) software programs. As the failure occurs, the following error may be seen in the HP SIM log mxdomainmgr.0.log: 9/13/11 4:26:58 PM An error occurred during a workload repository operation. java.rmi.connectioexception: Exception creating connection to: CPVC-MTRX01.support.local; nested exception is: java.net.socketexception: No buffer space available (maximum connections reached?): connect (4 occurrences) Reboot the CMS. Microsoft has released a Hot Fix that prevents the error and messages from occurring while attempting to open a new socket connection. See the Knowledge Base Article , Kernel sockets leak on a multiprocessor computer that is running Windows Server 2008 R2 or Windows 7 at the following website: The Hot Fix that corrects the issue is included in the article with the process to apply the Hot Fix: For more information, see Customer Advisory c , HP Insight Dynamics - The HP Insight Dynamics Fails Due to Socket Exception No Buffer Space Available Error When Installed on HP ProLiantServers Running Windows Server 2008 R2 or Windows Server 2008 R2 SP1 at the following website: Agent and communication troubleshooting 31

32 Server blade serial number is not added on ESX or Hyper-V managed nodes After HP SIM discovery, the blade's serial number is not added on the VMware ESX or Hyper-V nodes. Possible cause HP SIM does not have the enclosure and blade bay information to associate it to ESX or Hyper-V nodes. Suggested action 1. Discover the enclosure which is supporting the blade. 2. After discovery, go to HP SIM / All systems page and check if the bay was associated correctly. 3. If the problem persists, select the blade which does not have the serial number and start an identify request. From HP SIM, select Options Identify Systems. Insight Control virtual machine management registration for ESX or Hyper-V nodes fail Insight Control virtual machine management registration for ESX or Hyper-V nodes fail. Possible cause For ESX, VMware vcenter settings are not configured in HP SIM. For Hyper-V, WMI proxy services are not running on Hyper-V. For ESX: 1. Start discovery task for the vcenter server using the appropriate credentials. 2. From HP SIM, select Options VME options Add or edit VME Credentials. 3. Attempt Insight Control virtual machine management registration again. For Hyper-V: 1. Go to the Hyper-V server and start the WMI proxy services. a. Select Start WMI Mapper. b. Select Start WMI Performance Adapter. c. Select StartWindows Management Instrumentation. 2. Attempt Insight Control virtual machine management registration again. Hyper-V VM hosts disappear from the IO server pool in large scale environments VM hosts disappear from the server resource pool on the IO Servers tab. Possible cause Slow response from the host to WMI Mapper queries from the CMS cause a timeout. Update the WMI Mapper property: 1. Access <vmm-install>\bin\hpvmm.conf. 2. Change the value of HyperVQueryTimeout: HyperVQueryTimeout= Restart the HP Insight Control virtual machine management service. 4. Rediscover Hyper-V VM hosts in Systems Insight Manager. 32 Troubleshooting procedures

33 5. Verify the device subtype of the VM hosts. If the subtype is not Microsoft Hyper-V Virtual Host, restart the WMI Mapper service and rediscover the VM host. Matrix infrastructure orchestration troubleshooting IDE disk on a VMware ESX VM fails to reactivate When using an ESX VM with an IDE disk, the disk fails to reactivate in Matrix Operating Environment. Suggested action 1. Using VMware vsphere, power off the VM. 2. Add a SCSI driver in the VM (any size). 3. Power on the VM and install the drivers for the controller. 4. Power off the VM. 5. Outside of vsphere, connect on ESX host and use any editor to edit the.vmdk file for the IDE disk. Change ddb.adaptertype from ide to lsilogic. 6. Remove both the IDE and SCSI disks in the editor, but do not physically remove the IDE disk. 7. Close the Edit Settings. 8. Add a SCSI disk pointing to an existing disk (the old IDE disk). 9. Power on the VM. Unable to add servers in a SCVMM service Running an infrastructure orchestration service request that deploys an SCVMM template fails with the following error: VM Deploy Template failed. Caused by: Failed to deploy VM Template for SCVMM: The VM file is already in use by another VM. Error ID 802. Wait for the object to become available, and then try the operation again. Possible cause A VM with the same name already exists in the SCVMM database on the same host as that requested for deployment. : A VM of the same name may have been deployed to the same host as part of a prior service request. A user may have manually created a VM with the same name on the same host. A failure in VM delete or other operation like Deactivate resulted in a VM with status Missing in the SCVMM database. To resolve, choose one of the following three options: Use a different VM name as part of the service request. OR In SCVMM, right-click Delete to remove the VM. OR Deploy the VM from a different host. Matrix infrastructure orchestration troubleshooting 33

34 HP Matrix infrastructure provisioning and linked clone provisioning fails when using a Windows 2012 OS template on ESXi 5.0 Update 1 host When using a Windows 2012 OS template on an ESXi 5.0 Update 1 host during provisioning, the virtual machine stops responding and displays a black screen with a white status ring or displays the stop code Error Code 0xc Restarting the virtual machine does not resolve the issue. Possible cause This is a known issue after installing Windows Server 2012 on an ESXi 5.0 Update 1 host. NOTE: The procedure below to add the vmgencounter.enable parameter to the virtual machine.vmx file may cause the new snapshot protection for domain controllers introduced in Windows8/Windows Server 2012 to stop functioning. For more information, see the Microsoft Technet Article Active Directory Domain Services (AD DS) Virtualization. Perform one of the following procedures. For HP Matrix infrastructure provisioning with a Windows 2012 OS template on ESXi 5.0 Update 1 host 1. Create a Windows 2012 virtual machine in VMware vcenter and add vmgencounter.enable = FALSE to the.vmx file of the virtual machine. a. Right-click the virtual machine and click Power Shut Down Guest. b. Right-click the virtual machine and click Edit Settings. c. Click the Options tab. d. Under Advanced click General. e. Click Configuration Parameters. f. Click Add Row. g. Under name enter vmgencounter.enabled. h. Under value enter False. i. Click OK to save the changes and then close the window. j. Power on the virtual machine. 2. Create the Insight Control virtual machine management template. 3. In IO, create a service and customize the virtual machine. 4. The deployed VM will now boot correctly with all customized parameters set. For linked clone provisioning with a Windows 2012 OS template on ESXi 5.0 Update 1 host 1. Create a Windows 2012 virtual machine in VMware vcenter and add vmgencounter.enable = FALSE to the.vmx file of the virtual machine. a. Right-click the virtual machine and click Power Shut Down Guest. b. Right-click the virtual machine and click Edit Settings. c. Click the Options tab. d. Under Advanced click General. e. Click Configuration Parameters. f. Click Add Row. g. Under name enter vmgencounter.enable. h. Under value enter False. i. Click OK to save the changes and then close the window. j. Power on the virtual machine. 34 Troubleshooting procedures

35 2. Create the Insight Control virtual machine management template. 3. In IO, create a service from a template with Deploy as linked servers checked, and customize the virtual machine. The deployed link clone VM (child VM) may not boot and displays a black screen with a white status ring. In this case, power off the child VM and repeat the procedures in step Customization starts and all the parameters are set on the VM. It takes about minutes to reflect all the customization parameters set on the VM. The VM then successfully boots up with all parameters set correctly. Password is blank on provisioned VMs with Windows 2008 R2 and above provisioned VMs If a VM is provisioned with Windows 2008 R2 or above from infrastructure orchestration without setting any password in the sysprep file, one of the following occurs: VM deployed on ESX/ESXi hosts boot up and users can log in with a blank password. VM deployed on Hyper-V boot up and prompt the user to set a password. Possible cause Microsoft removed the option password never Expires in Windows 2008 R2 and higher (as part of sysprep). VMware overrides the option prompt for a password on the next boot. Do one of the following: Specify an explicit password using either clear text or encrypted format in the Sysprep.ini file for any deployment of a VM that has an operating system Windows 2008 R2 or higher operating system, or Specify an empty string password in the sysprep.ini file, which results in the prompt for a password on the next boot from VMware. Microsoft SCVMM templates do not appear in the Matrix infrastructure orchestration software tab Valid SCVMM templates are missing in the Matrix infrastructure orchestration software tab. Possible cause When creating a SCVMM template from the SCVMM server, the virtualization platform was set to Unknown. These templates will not display in the Matrix infrastructure orchestration software tab. From the SCVMM UI: 1. Identify the template not listed in the Matrix infrastructure orchestration tab: On the SCVMM console, right click the template that is not listed, select Properties and then click the Dependencies tab. 2. Click Virtual Hard Disk to display the Properties window. 3. The Virtualization Platform in the properties window displays the value Unknown, choose Microsoft Hyper-V from the drop down and then click OK to save. 4. Click OK again to close the main window. 5. From the CMS where you are managing the SCVMM, click Start, select Run and type: services.msc Matrix infrastructure orchestration troubleshooting 35

36 6. Find HP Insight Control virtual machine management in the list and restart the service. 7. After several minutes, the missing template appears in the Matrix infrastructure orchestration software tab. Incorrect hostname displays during validation for deployed Hyper-V VM After infrastructure orchestration provisioning and customization on the provisioned VM, in some instances the VM is not updated correctly. Suggested action Before provisioning, add the parameter RetryForVMRefreshCache = 50 in the file (shown in the default location) at C:\Program Files\HP\Insight Control virtual machine management\bin\hpvmm.conf. Restart the Insight Control virtual machine management service and start the provisioning requests. Delete operation of a VM entry from Microsoft SCVMM fails The Remove Server or Delete Service operation in Matrix infrastructure orchestration fails when trying to delete a VM on SCVMM, as seen in the Matrix infrastructure orchestration Request Details screen. Possible cause The SCVMM database has not been updated before the deletion of the VM entry from SCVMM. Suggested action 1. Open the SCVMM user interface. a. Select Start All Programs. b. Open the Microsoft System Center folder and click on Virtual Machine Manager 2008 R2 if you are running SCVMM 2008 (or Virtual Machine Manager 2012 if you are running SCVMM 2012). c. Double-click on Virtual Machine Manager Administrator Console. 2. Select the deleted VM, shown with a Missing status. 3. Right-click and select Delete. The VM is removed from the SCVMM database. Matrix infrastructure orchestration service creation fails in the target reconfiguration phase Service creation intermittently fails when a service is created with Red Hat Enterprise Linux 6 OS. During the service creation process, the target server fails to shut down in the reconfiguration phase due to an issue with the HP Health Agents versions 8.70 and older. 1. Identify the failed target servers in the Maintenance server pool. a. Browse to Tools Infrastructure orchestration from the Matrix Operating Environment browser window. b. Click on the Servers tab. c. Expand the server pool by name Maintenance. d. Note the names of servers in this server pool; these are the serial numbers of the servers. 2. Delete the logical servers from Matrix OE visualization. a. Browse to Tools HP Matrix OE visualization. 36 Troubleshooting procedures

37 b. Select Logical Server from the Perspective drop-down box. c. Identify the logical server entries where the name starts with Clean-me and contains the previously noted serial number. d. Select these logical servers and delete them by navigating to Delete Delete Logical Server from within the Matrix OE visualization screen. 3. After the logical servers are successfully deleted, move the servers from the Maintenance pool to an appropriate server pool. a. Navigate to Tools infrastructure orchestration Servers. b. Select the server pool where the servers from the maintenance pool will be moved. c. Click Modify Pool button. d. Select Maintenance from the drop-down list. e. Select the server that needs to be moved from the list displayed and click the >> button. f. Save the server pool. 4. Retry the service creation. Matrix infrastructure orchestration service creation fails with Error powering on logical server A create service request fails for an ESX VM template with automatic OS deployment. The following error is displayed: Failure: Error powering on logical server. The start function on virtual machine vm-name failed: VMware ESX Server cannot open the virtual disk disk-name for clustering. Please verify that the virtual disk was created using the 'thick' option. Cannot open the disk disk-name or one of the snapshot disks it depends on. Reason: Thin/TBZ disks cannot be opened in multiwriter mode. This error may occur when no virtual machine licenses are available on the target VM Host. Possible cause VMware ESX Server 3.5 is the target VM host, which is no longer supported by infrastructure orchestration. Suggested action Update the VM host to a supported version of ESX, or edit the infrastructure orchestration service and select a different resource pool that contains an ESX VM host with a supported version of ESX. For a list of supported versions, see HP CloudSystem Matrix Compatibility Chart at Matrix infrastructure orchestration service creation fails in the VM customization phase The service creation failure is due to one of the following issues. Service creation fails when a VM is created on a Hyper-V host. Service creation fails during the customization phase of deploying VMs. The VMware fault fault.customizationpending.summary displays in the Insight Control virtual machine management log files. Failure in customizing a Windows Server 2008 VM. VM console displays the following error message: Windows could not parse or process the unattend answer file for pass [specialize]. The settings specified in the answer file cannot be Matrix infrastructure orchestration troubleshooting 37

38 applied. The error was detected while processing settings for component [Microsoft-Windows-Shell-Setup]. The computer restarted unexpectedly or encountered an unexpected error. Windows installation cannot proceed. To install Windows, click OK to restart the computer, and then restart the installation. The environment is not completely configured. The VM used in the Insight Control virtual machine management template was not powered off properly. The vcenter or vsphere is not configured properly for successful customization. The incorrect product key was specified for the operating system in the IO sysprep file. The IO sysprep file contains an invalid parameter. The Hyper-V host is configured with a password that contains or characters. Windows sysprep files are not added or missing. Verify that you are provisioning a guest operating system supported by HP CloudSystem Matrix. For VMware ESX VMs, verify the source VM being used to create the template has the VMware Tools installed and configured. For Linux VM guests, ensure the vmware-config-tools.pl command was executed after installing the tools. Verify the source VM being used to create the template has been stopped or paused (not suspended) before creating the template. Verify the source VM being used to create the template has been properly shut down before creating the template. When guests are not gracefully shutdown or halted it can cause customization phases to fail. Verify that the product key (and any other configuration option) being used in the IO sysprep file functions with the VM by performing the following test. In vcenter or vsphere: Locate the original VM that was used to create the Insight Control virtual machine management template that is experiencing the failure. Right-click the VM, and select Template Clone to Template. This action creates a vsphere template of the VM. Create a new VM from this vsphere template in vsphere. During the VM creation wizard, select the option to customize the VM and input the requested customization parameters. If this VM customization process fails, deployment in infrastructure orchestration will not be successful. Correct the vsphere customization issues, and create a new Insight Control virtual machine management template from the functional VM. Proceed with deployments from infrastructure orchestration. Verify the IO sysprep file does not contain an invalid parameter. Use the sample file as a reference. For VMware VMs, VMware vcenter Server must have Sysprep tools installed on the vcenter management server in the VMware VirtualCenter\sysprep\svrOS folder. For Microsoft Hyper-V, Microsoft Sysprep tools are required on the HP CMS in the Insight Control virtual machine management sysprep folder. Change the password on the Hyper-V host to something that does not contain or characters. 1. Select the Hyper-V host from the system list. 38 Troubleshooting procedures

39 2. Edit the system credentials. Go to Options Security Credentials System Credentials and select the Hyper-V host from the list. 3. Change the sign-in credentials and save the new credentials. Wait a few minutes for the host to complete re-identification. 4. Start the Matrix infrastructure orchestration service request. Verify the Windows sysprep files have been added: For Hyper-V VMs running Windows, go to the CMS and copy the sysprep files to C:\ Program Files\HP\Insight Control virtual machine management\ Sysprep\2003. For VMware ESX VMs running Windows, go to the system which has Virtual Center installed and copy the sysprep files to the following paths: Virtual Center 4.x and 5.x: C:\Documents and Settings\All Users\ Application Data\VMware\VMware VirtualCenter\sysprep\svr2003 For more information about customization or configuration, see the HP Matrix Operating Environment Infrastructure Orchestration User Guide at NOTE: The C:\Program Files\HP\Matrix infrastructure orchestration\conf\ Sysprep.inf file must be edited to specify additional customizations for Windows VMs. Without this edit, a newly provisioned system halts during first boot waiting for a key. For details, see HP Matrix Operating Environment Infrastructure Orchestration User Guide at go/matrixoe/docs. Service request fails when using a template configured with custom host names When a service request using custom host names fails, the resulting error message is incorrect. The message is of the form: The following hostname(s) are invalid. The execution of workflow "/Library/Hewlett-Packard/Infrastructure orchestration/custom Hostnames/samples/hostname pool" has failed with the following error. FailureMessage=;TimedOut=. The current template is not configured correctly to work without custom hostnames enabled. Please use the designer to modify the server group hostname or enable custom hostnames Possible cause This happens when a user disables the custom host name feature after provisioning a service with the custom host name feature enabled. Verify that the Operations Orchestration service is available. If it is not available, restart the OO service and enable the custom host name feature. Retry the operation. Matrix infrastructure orchestration troubleshooting 39

40 Physical server provisioning using Insight Control server deployment fails during the final personalization step Physical server provisioning using Insight Control server deployment fails during the final personalization step when the CMS is configured with an automatic proxy configuration script in IE. Possible cause There is a problem or delay communicating with the automatic configuration script configured in Microsoft Internet Explorer. The problem is in Windows.NET, and the proxy configuration is shared between IE and.net. If the proxy configuration is causing physical deployment failures, the Matrix infrastructure orchestration log file at..\program Files\HP\Insight Orchestration\ logs shows an error similar to the following: ERROR com.hp.hpio.sbapi.tasks.createphysicalservertask.execute Error personalizing OS with host uuid: 38C77BED-9FAE C06 0F536BA46507 com.hp.hpio.sbapi.exception.aresadapterexception: Deployment server did not accept customization data. The log file for the connector software that drives Insight Control server deployment (..\Program Files\HP\Systems Insight Manager\logs\Alc1_0.0.log), shows that the operations which perform the customizations are timing out. Suggested action Do one of the following: Configure Internet Explorer with an explicit proxy server address and appropriate exceptions instead of using an automatic configuration script. 1. From Internet Options, select the Connections tab. Click LAN settings. 2. Check Bypass proxy server for local addresses if the server deployment server is on the CMS. Otherwise, click Advanced. Under Exceptions, enter the IP address of the server deployment server. Provide a.net configuration file for Internet Explorer so it will not use the system default proxy. Create the file..\program Files\HP\Systems Insight Manager\bin\ alc-rdpaccess.config with the following content: <configuration> <system.net> <defaultproxy> <proxy usesystemdefault = "false" /> </defaultproxy> </system.net> </configuration> Matrix infrastructure orchestration service creation fails for physical requests where Windows Server OS is selected Service creation fails with a physical template installing Windows Operating System. On the target server, the log cpqsetup.log shows the job HP Insight Management WBEM Providers for Windows Server 2003/2008 x64 Editions is waiting on Executing MSI. Matrix infrastructure orchestration times out waiting for HP Insight Control server deployment to complete installation of the Install HP Management Agents job, which is hanging on the target server. 40 Troubleshooting procedures

41 If the failure to install the WBEM provider is intermittent, no corrective action may be required because infrastructure orchestration automatically retries the failed request. If the failure is observed frequently, disable the installation of HP Web-Based Enterprise Management providers. 1. Edit the install.cmd file, found under <Insight Management install folder>\ rdp\deployment server\hpfeatures\hpagents-ws\components folder, and remove the following two lines. call :installsc wbem%processor_architecture% if %cpresult% geq 1 set /a cpresults=cpresults+1 2. Save the file. The new service requests will no longer attempt to install Web-Based Enterprise Management providers. SNMP Webagent continues to provide management capabilities when WBEM providers are not installed. Provisioned VMs fail due to improper VM host hardware configuration Microsoft Virtual Network Manager fails to load during rapid provisioning of VMs on ProLiant servers with Hyper-V Windows 2008 R2 SP1/SP2. The following error message displays: Task for logical server has failed, failure virtual machine hardware was not configured correctly. The customization of the provisioned VM's fail and the infrastructure orchestration service request fails. 1. Install the latest version of NCU found at the following website: bizsupport/techsupport/softwaredescription.jsp?lang=en&cc=us&prodtypeid= & prodseriesid= &switem=mtx-5221f1131e1e46a49d795ccded& prodnameid= &swenvoid=4064&swlang=8&taskid=135&mode=3 2. Enable the TCP offload feature in the Hyper-V host. a. Access the Network Adapter Properties window for the VM host. b. Click the Advanced tab. Matrix infrastructure orchestration troubleshooting 41

42 c. Select the TCP offload options in the Property list and select Enable as the Value. d. Click OK. Unable to allocate servers and networks in the same VCDG A Matrix infrastructure orchestration request failed with the following error message: Unable to allocate servers and networks in the same Virtual Connect Domain Group. Either more available servers are required, or could not match existing servers against memory size, disk space and processor count requirements. There were insufficient servers in the server pools referenced in the Create Request. One or more servers considered available for use were actually in the maintenance pool. At least one of the server blades in a referenced user server pool already has a profile assigned. Verify that the infrastructure orchestration server pools referenced in the Create Request contain sufficient servers taking into account all the servers, storage, and networking requirements of the template. Check whether any servers were moved to the maintenance pool due to a previous provisioning failure. For any clean-me logical servers, perform a manual cleanup process. See Manually cleaning up Clean-me logical servers (physical) (page 46). Search for errors in Insight Management log files. 42 Troubleshooting procedures

43 Matrix infrastructure orchestration request fails due to server use outside of infrastructure orchestration A Matrix infrastructure orchestration request fails with the following error: Server must not be in use outside HPIO. Possible cause Target server is powered on (in use). Suggested action Before starting a create request, ensure that the target server is powered off and without profiles assigned. Perform Manually cleaning up Clean-me logical servers (physical) (page 46). Perform refresh. Tools Logical Servers Refresh Search for errors in the Insight Management log files. Automated OS provisioning to a physical server fails to successfully PXE boot A PXE network was assigned to an incorrect FlexNIC physical function causing the physical server to fail a PXE boot. Only the first physical function of each CloudSystem Matrix FlexNIC interface supports PXE booting. Accordingly, CloudSystem Matrix FlexNICs support PXE on only the following ports: LOM:1-a LOM:2-a MEZZ<num>:1-a MEZZ<num>:2-a Attempting to assign a PXE network to any other NIC port will result in the physical server failing to correctly perform a PXE boot. Matrix infrastructure orchestration assigns networks to physical functions on the server blade in a round robin fashion, and a PXE network may have been assigned to a port and physical function not supporting PXE boot. The PXE network must be one of the first N networks assigned where N is the number of physical NIC ports available. In addition, to achieve maximum portability of a logical server, the PXE network should be one of the first 2 networks assigned since every server blade has an integrated dual-ported NIC LOM and the first physical functions of each LOM port will support PXE booting. Request History shows deployment server does not have a record of the target system(s) The following message displays in the infrastructure orchestration request history: Deployment server does not have a record of the target system(s) The installation failed during automatic provisioning. The ProLiant or Integrity blade remote console shows the installation is hung. This impacts ProLiant BL620c, BL680c G7, and Gen 8, and Integrity BL890c/BL890c i2 server blades. The target server cannot boot through the PXE server. Matrix infrastructure orchestration troubleshooting 43

44 Another node is running with the same IP address as the blade being provisioned. The server cannot continue installation with an IP conflict. Occasionally if the server firmware has a problem or was incorrect, the logical UUID cannot be created from the VC profile. During the provisioning phase, infrastructure orchestration periodically polls the status of the blade in the PXE boot process. If the blade being provisioned is a ProLiant G7 or Gen 8, and/ or BL890c/BL890c i2 server blades the PXE boot retry count may not be large enough (default is 30 tries). The server is already registered in the deployment server. Ensure that the PXE server is working properly and the target server is on the same network as the PXE server. Ensure the IP address assigned to the blade for provisioning does not conflict with other nodes in the environment. Ensure the server firmware and all other firmware in the provisioning environment are consistent with the supported recipe for the Integrity blade to be provisioned. Change the default setting for retry.count.ares.pxe = 30 to 50 or larger. Ensure the deployment server does not have the server already registered or a server with the same MAC. Matrix infrastructure orchestration Networks tab does not retrieve network information The network page in infrastructure orchestration does not retrieve any network information, or infrastructure orchestration allocation failure and provision failure occurs as shown in Figure 6 (page 44). Figure 6 Allocation failure example VMware vcenter Server credentials are not configured or are lost during an Insight Control virtual machine management update. VMware vcenter Server is unreachable. Enter the VMware vcenter Server credentials in HP SIM. 1. Go to Options VME options View VME options View VME Settings and check the vcenter status. 2. Follow the steps on that page OR follow the steps below: a. Select Options Security Credentials System Credentials. b. Edit system credentials. c. Click Show Advanced protocol credentials VME tab. d. Enter credentials and click OK. Verify that the configured VMware vcenter Server is reachable. Matrix infrastructure orchestration deployment task fails, exceeding its timeout value 44 Troubleshooting procedures

45 The task for logical server logical_server_id has failed. The job job_id has exceeded its timeout value of nnn minutes, marking the job as failed. A configuration issue with server blade firmware. Insight Control server deployment has been reconfigured in a way that modified the default boot sequence so it no longer works with infrastructure orchestration. Errors in storage presentation, multipath, zoning configuration, or failed components. Timeout value for create logical server request is too short. Check the status of the Insight Control server deployment task: 1. Locate the Provisioning logical server message which identifies the server blade in use. 2. Log into the deployment server and open the Deployment Server Console. 3. Select the computer in the Computers pane. 4. On the right side, double-click Job to open the property window. 5. Click the Status Details button to show the Status Details windows and review the job status. 6. If the in progress icon appears as the last task, right-click Properties to display the Job Schedule Information. To reconfigure deployment server software, see HP Matrix Operating Environment Infrastructure Orchestration User Guide at the following website: Use the HP ilo console to first assess the state of any executing Insight Control server deployment task. Check if the HBA firmware can see the disk on the SAN with a SCAN operation. Contact your storage administrator to ensure that your storage presentation, zoning, operating system distribution, and drivers all support multipath SAN environments. To set the timeout value, see HP Matrix Operating Environment Infrastructure Orchestration User Guide at the following website: Optionally, remove the storage pool entry which will remove the SPM storage service. Whatever the cause, always check for any "clean-me" logical servers. If these are present, perform a manual cleanup process. See Manually cleaning up Clean-me logical servers (physical) (page 46). Matrix infrastructure orchestration cannot deploy Integrity VM guests when gwlm is controlling the host If IO chooses a host controlled by gwlm for deploying an Integrity VM guest service template, the guest cannot start and the following messages appear in the /var/opt/hpvm/common/ command.log: 05/31/12 11:43:45 ERROR Guest_number root Guest Guest_number has been marked as not startable. 05/31/12 11:43:45 ERROR Guest_number root An external manager (gwlm) is managing VMs on this Host. This manager is located on location and can be accessed via Add the VM to the manager to make it available for starting. Suggested action Exclude any Integrity VM hosts and VSPs under the control of gwlm from the IO server pools. Matrix infrastructure orchestration troubleshooting 45

46 Manually cleaning up Clean-me logical servers (physical) Clean-me logical server appears for a physical server. An infrastructure orchestration physical server deployment fails or infrastructure orchestration physical server delete service request fails. With these failures, the server blade is moved to the infrastructure orchestration Maintenance pool, and the Matrix OE logical server is renamed with the prefix Clean-me. Perform a manual cleanup. The server profile and boot disk are preserved, enabling the administrator to determine the cause of the failure and perform any required maintenance before returning the server blade to a user pool. Manual clean-up of the failed server blade depends on the deployment server used to provision it. For Insight Control server deployment: 1. Power on the server blade. (You might need to cycle power if the server blade is stuck.) 2. Make sure that the attached boot LUN erases, by performing one of the following: Run the Erase ProLiant Hardware job from the Insight Control deployment console. This job can be found in Server Deployment Toolbox Array Hardware. HP recommends this option. Power on the blade, press F9, and then select Advanced Options Erase Boot Disk. 3. Ensure the server blade is powered off after the job is complete. Manually power off the server blade, if necessary. 4. Manually delete the server blade and all its jobs from the Insight Control deployment console. For Ignite-UX: 1. Power on the server blade and allow it to boot the operating system. 2. Run HP SIM discovery from the GUI, or on the command line by entering: mxnode -a <target IP or DNS name> -w where: <ip> is the IP address of the newly created HP SIM node, and <password> is the root password of the newly created HP SIM node. NOTE: To use the mxnode command, enter the root login for the provisioned server into the global credentials for HP SIM (Options Security Credentials Global Credentials). 3. From the CMS, run the mxtool Ignite Erase Disks by typing this command into a Windows command prompt: mxexec t Ignite Erase Disks n <ip> where <ip> is the IP address of the server to be cleaned. 4. Once cleanup is complete, remove the global credentials using Options Security Credentials Global Credentials, if you used the mxnode command in step 2. For HP Server Automation: 1. Power on the server blade. 2. Press F9, and select Advanced Options, Erase Boot Disk. 3. Boot again, press F9 again, and select the NIC as the first boot device. 4. Verify that the server blade is powered off after completing the above step. 5. Connect to the SA Core Server. 46 Troubleshooting procedures

47 6. Check to see if the server that failed is in either the All Managed Servers or the Unprovisioned Servers list. If so, delete it. System identification should be done by using the MAC address of the system that failed to deploy (VC assigned MAC). It may be required to change the view in the Properties pull down menu to see the MAC address. For all deployment options: After completing the previous steps based on the deployment server being used, perform the following steps to allow the storage pool entry to be selected in a future provisioning request: 1. Deactivate the associated Matrix OE logical server (if it is active) by selecting Tools Logical Servers Deactivate from the Matrix Operating Environment menu. The logical server typically has a name beginning with Clean-me- as shown in Figure 7 (page 47). Figure 7 Clean-me logical server 2. After the associated Matrix OE logical server deactivation is complete, delete the Matrix OE logical server from the Matrix Operating Environment menu by selecting Delete Delete Logical Server. In the Delete Logical Server screen, type Yes to proceed with this operation. IMPORTANT: Do not select the Unmanage Logical Server option. Doing so causes the used WWNs within the logical server storage pool entry to be marked as inoperable, regardless of presentation or zoning. If you mistakenly selected the Unmanage Logical Server option, follow the procedure to configure storage pool entries, FC zones, and disk array presentations in the HP Matrix Operating Environment Infrastructure Orchestration User Guide found at 3. Verify that in VCEM the VC profile associated with the failed server blade exists. If the profile exists, unassign and delete the VC profile associated with the failed server blade from VCEM. 4. Move the server blade out of the infrastructure orchestration maintenance pool and into its original pool. a. In the infrastructure orchestration console (reached from HP SIM), select the original server blade pool and click Modify. b. Move the failed server from the infrastructure orchestration maintenance pool to your server blade pool, and then save the pool. Manually cleaning up Clean-me logical servers (virtual) Clean-me logical server appears for a virtual server. An infrastructure orchestration virtual server deployment fails or infrastructure orchestration virtual server delete service request fails. With these failures, the server blade is moved to the infrastructure orchestration maintenance pool, and the Matrix OE logical server is renamed with the prefix Clean-me. When infrastructure orchestration deploys a VM, it creates a directory structure into which the VM is provisioned. The directory that is created follows the pattern: <VMFS Volume>\<Service Name>\<VM Name>. This folder contains all the folders and files corresponding to the virtual logical servers. Delete this folder and all its contents. For ESX: Matrix infrastructure orchestration troubleshooting 47

48 1. In Matrix OE visualization, select Tools Logical Servers Refresh. 2. In the Logical Server perspective, verify that the logical server no longer exists. If it exists, delete it using Delete Delete Logical Server. 3. Use the vcenter client to browse the ESX server datastore. 4. Delete the VM, if it exists. 5. Browse to the ESX server's datastore. 6. Navigate to the directory and delete the offending directory and directory contents. For Hyper-V: 1. In Matrix OE visualization, select Tools Logical Servers Refresh. 2. In the Logical Server perspective, verify that the logical server no longer exists. If it exists, delete it using Delete Delete Logical Server. 3. Connect to the Hyper-V Hypervisor. 4. Delete the VM from the Hyper-V Manager. 5. Navigate to the directory and delete the offending directory and directory contents. If HP SA deployment was used to provision the VMs: 1. Remove the ESX and/or Hyper-V VMs using the preceding steps. 2. Connect to the SA Core Server. 3. From the SA devices list (All Managed Servers), select the VMs. 4. Perform a deactivate and a delete. 48 Troubleshooting procedures

49 Matrix infrastructure orchestration users cannot view or receive HP Operations Orchestration 1 Matrix infrastructure orchestration users cannot view HP Operations Orchestration s due to an authorization error. Possible cause 1 SMTP server data was not provided during Matrix infrastructure orchestration installation. Suggested action 1 1. Open HP Operations Orchestration Studio. 2. In the Navigation tab, expand Configuration System Properties HpioSmtpHost. 3. Edit the property value to point to the configured SMTP server in the network. 2 Matrix infrastructure orchestration users cannot receive HP Operations Orchestration s. The following message is shown during deployment: could not be sent Possible cause 2 The antivirus software installed in the HP Operation Orchestration is blocking the SMTP port. Suggested action 2 Verify the antivirus software installed in the HP Operation Orchestration system is not blocking the SMTP port. If it is, create a rule in the antivirus to allow the CMS to use the SMTP port (normally port 25). Defined SPM storage template does not appear in infrastructure orchestration designer A storage template defined in SPM does not appear in infrastructure orchestration designer. The SPM storage templates were not created within the Matrix Security Group in SPM. The SPM storage templates were not defined correctly for Matrix to use it (Matrix will ignore storage templates that disallow key requirements or have requirements Matrix does not support). Suggested action Within SPM ( where localhost is the name of the CMS), look at the storage template that you want to make available to Matrix infrastructure orchestration. Make sure that the template: Is defined within the Matrix Security Group Has the correct requirements defined: Raid, Volume Capacity, Host, ConnectedToFCEndpoint, and Storage Capability must be set to Allowed or Must Include. PoolAvailableCapacityBytes, PoolAvailableCapacityPercent, PoolCapacitySubscriptionRatio, PoolVolumeCount, ResourceExists, ServiceDeactivationPolicy, UseResource, VolumeCommittedCapacityPercent, VolumeCreationSecurityGroup, ConnectedToNetwork must not be set to Must Include. VC domain error after CMS crash recovery Matrix infrastructure orchestration troubleshooting 49

50 One or more VC domains may have a configuration mismatch or an out-of-synch occurrence after recovering from a CMS crash. The CMS crash may have occurred while a VC server profile operation (assign/un-assign/delete/move) was in progress, causing the VC domain to go out of sync with the CMS. 1. Execute Mxsync in analysis mode (mxsync) to determine the nature of the failure on the VC domain during the CMS recovery phase. 2. Execute Mxsync in execute mode (mxsync -e) during the CMS recovery phase and follow the Mxsync instructions until the VC domains are synchronized. 3. If Mxsync detects and reports out-of-sync scenarios in additional subsystems, resolve them according to the instructions displayed. Exit Mxsync. NOTE: Executing mxsync e will help resolve VC domain synchronization issues across multiple VC domains. Unassign profile request on a VC logical server fails during deactivate after a VC failover An unassign profile request on a VC domain-based logical server fails during a deactivate sequence. The message is displayed after a VC failover of the primary VC module has occurred: Error during un-assign profile; Details: This action cannot be completed because a required object could not be found One or more VC domains may see a configuration mismatch or an out-of-synch occurrence after recovering from a CMS crash. The CMS crash may have occurred while a VC Domain unassign profile operation was in progress. 1. Execute Mxsync in analysis mode (mxsync) to determine the nature of the failure on the VC domain during the CMS recovery phase. 2. Execute Mxsync in execute mode (mxsync-e) during the CMS recovery phase and follow the Mxsync instructions until the VC domains are synchronized. 3. If Mxsync detects and reports out-of-sync scenarios in additional subsystems, resolve them according to the instructions displayed. Exit Mxsync. NOTE: Executing mxsync e will help resolve VC domain synchronization issues across multiple VC domains. Matrix infrastructure orchestration Create Service request does not complete successfully after CMS crash recovery Matrix infrastructure orchestration Create Service request does not complete successfully when resumed after CMS is recovered from a crash. Manual steps recommended by infrastructure orchestration are inadequate to recover the logical server and re-issue the request successfully. VCEM and VCM are out of sync due to a CMS crash that occurred while assigning a server profile. 50 Troubleshooting procedures

51 Perform VC Domain maintenance. For more information, see HP Virtual Connect Enterprise Manager User Guide at Delete the clean-me-<ls>logical server for any that are listed. For more information, see the HP Matrix Operating Environment Infrastructure Orchestration User Guide at Move the blade from Maintenance to another server pool which frees up the server IP address and marks the blade as available. Perform a logical server refresh. Tools Logical Servers Refresh. Re-issue the infrastructure orchestration Create Service request. Matrix infrastructure orchestration Create Virtual Servers Request does not complete successfully after a CMS crash recovery Matrix infrastructure orchestration Create Virtual Servers Request fails to complete successfully after a CMS crash recovery. The virtual resources are not released upon CMS recovery. A CMS crash occurred while vcenter was in the process of creating logical servers and VMs, causing the Insight Control virtual machine management and Matrix OE logical server management databases to get out-of-synch. 1. Perform a logical server refresh. Tools Logical Servers Refresh. 2. Manually delete the logical servers that were partially created before the CMS crash. Delete Delete Logical Server. 3. Manually delete the VMs from vcenter. See Manually cleaning up Clean-me logical servers (virtual) (page 47). 4. Re-issue the create virtual servers request. Matrix infrastructure orchestration Deactivate physical server does not complete successfully after CMS crash recovery Matrix infrastructure orchestration Deactivate physical server does not complete successfully when resumed after CMS is recovered from a crash. Manual steps recommended by infrastructure orchestration are inadequate to recover the logical server and re-issue the request successfully. VCEM and VCM are out of sync due to a CMS crash that occurred while un-assigning a server profile. Perform maintenance of the VC Domain Group in VCEM which un-assigns the server profile. For more information, see HP Virtual Connect Enterprise Manager User Guide at Move the logical server from Maintenance to another server pool to free up the server IP address and mark the blade as available. For more information, see the HP Matrix Operating Environment Infrastructure Orchestration User Guide at docs. Activate the server from Matrix OE logical server management. Re-issue the infrastructure orchestration Deactivate request. Matrix infrastructure orchestration troubleshooting 51

52 Matrix infrastructure orchestration Deactivate Virtual Servers Request does not complete successfully after a CMS crash recovery Matrix infrastructure orchestration Deactivate Virtual Servers Request fails to complete successfully after a CMS crash recovery. The targeted logical servers are placed in an Inoperable state and fail to deactivate and delete. A CMS crash may have occurred while vcenter is in the process of creating logical servers and VMs, causing the Insight Control virtual machine management and Matrix OE logical server management databases to get out-of-sync. 1. Perform a logical server refresh. In Matrix OE visualization, Tools Logical Servers Refresh 2. Re-issue the deactivate virtual servers request. Matrix infrastructure orchestration Add Disk request for an ESX virtual server does not complete successfully after CMS crash recovery Matrix infrastructure orchestration Add Disk request for an ESX virtual server does not complete successfully when resumed after CMS is recovered from a crash. A CMS crash occurred while Virtual Center is in the process of creating a Virtual disk, causing an incomplete disk creation. Remove the partially created virtual disk file from the Virtual Center. Perform a logical server refresh. Tools Logical Servers Refresh. Re-issue the Add Disk request. mxagentconfig command fails with SSH protocol error on Integrity managed node From the CMS command line, mxagentconfig a n <ip-or-managed-hostname> -u <user> -p <password> command fails and returns the following error: Failed to register as an agent to: managed-node.fqdn Error: Configuration failed to complete due to the following exception: Ssh Operation failed for node:managed-node.fqdn. SSH_PROTOCOL_VIOLATION:EOF received before message completed! [Unknown cause] Completed with errors Possible cause The SSH server of the managed node cannot authenticate the incoming request during the LoginGraceTime period (2 minutes by default). As a result, the SSH server disconnects and exits. 1. Review the LoginGraceTime period. a. Log into the managed node. b. Stop the sshd service: /sbin/init.d/secsh stop 52 Troubleshooting procedures

53 c. Start the sshd manually in foreground mode with infinite LoginGraceTime: /usr/sbin/sshd D o LoginGraceTime=0 d. From the CMS, issue the mxagentconfig command: mxagentconfig a n <ip> -u <user> p <password> e. Wait longer than 2 minutes (the original LoginGraceTime). f. Stop the foreground sshd (^C). If the mxagentconfig is successful, continue with next step. If not, go to step i. g. Edit /opt/ssh/etc/sshd_configto uncomment the LoginGraceTime and set it with a suitable value, for example, 3m. h. Save and exit /ssh/etc/sshd_config i. Restart sshd: /sbin/init.d/secsh start 2. If Kerberos authentication is not present in your infrastructure, disable it from the sshd configuration file: a. Log into the managed node. b. Stop the sshd service: /sbin/init.d/secsh stop c. Start sshd manually in foreground mode with KerberosAuthentication=no: /usr/sbin/sshd D o KerberosAuthentication=no d. From the CMS, issue the mxagentconfig command: mxagentconfig a n <ip> -u <user> -p <password> e. Stop the foreground sshd (^C). If the mxagentconfig is successful, continue with the next step. If not, go to step h. f. Edit /opt/ssh/etc/sshd_config, uncomment the KerberosAuthentaction line and set it to no: KerberosAuthentication no g. Save and exit /ssh/etc/sshd_config. h. Restart sshd: /sbin/init.d/secsh start Unable to activate an Integrity VM logical server due to a disk creation failure An Integrity VM host appears in an infrastructure orchestration server pool, but the host is not allocated for creation of an Integrity VM guest. Figure 8 (page 54), Figure 9 (page 54), and Figure 10 (page 55) illustrate the errors that occur when SG and SLVM are not properly installed and configured. Matrix infrastructure orchestration troubleshooting 53

54 Figure 8 Job status for logical servers Figure 9 Infrastructure orchestration requests 54 Troubleshooting procedures

55 Figure 10 Infrastructure orchestration servers The VM host SG and SLVM software is not properly installed and configured. No SLVM VGs of type CONTAINER_VOLUME_SLVM are configured in the Integrity VM device management database. The SLVM VG is not activated on the VM host. The VM host was restarted and the SLVM VG was not re-started. The version of the installed VirtProvider is not supported. Ensure that SG and SLVM VGs are properly created and added (with the CONTAINER_VOLUME_SLVM container type) to the device management database. Ensure the SLVM VGs are properly activated. Check if the VirtProvider version is supported based on the HP Insight Management Support Matrix found at To check the version, execute the command swlist -l fileset grep -i vmvirtprovider. Update to a supported version. For more information, see the HP Integrity Virtual Machines 4.3 or 6.1 Administrator Guide at Insight Control server deployment software is not listed in infrastructure orchestration Insight Control server deployment software is not listed in the Software tab in infrastructure orchestration. Insight Control server deployment is not configured in HP SIM Connection problems Proxy issues in the IE browser Matrix infrastructure orchestration troubleshooting 55

56 Ping the Insight Control server deployment server from the CMS to check if is reachable. Execute the command mxnodesecurity -l and check if the Insight Control server deployment IP is listed with the protocol dsc_rdp. If it is not listed, manually add using this command: "mxnodesecurity -a -p dsc_rdp -c <username>:<password> -t on -n <node_ip or node_name>. Add the IP/Range in the exclusion list in IE if a proxy is configured. VM snapshot creation fails The VM snapshot creation has failed for a given VM. Possible cause 1 The VM has shared disks. Suggested action 1 If the VM has shared disks, do not take snapshots. Possible cause 2 The Snapshot memory option is selected for the VM with independent disks. Suggested action 2 If the VM has independent disks, take snapshots without selecting the Snapshot memory option. Possible cause 3 The state of the VM has not changed since the last snapshot operation. Suggested action 3 Wait for the state of the VM to be changed before taking the next snapshot. Linked clone provisioning of a VM template containing shared disks fails During ESX server provisioning, an error appears: Error taking a snapshot on virtual Logical Server <nnnn>. Virtual machine is configured to use a device that prevents the snapshot operation: Device is a SCSI controller engaged in bus sharing. Possible cause The VM template created from an IO template specifies linked clone servers and a shared disk. This combination is not supported. Suggested action Edit the template in IO Designer and uncheck Deploy as Linked Servers or Disk is shared across servers and then retry the Create Service operation. VM instances not deleted after Matrix infrastructure orchestration fails If the Matrix infrastructure orchestration service is stopped during a provisioning operation, after Matrix infrastructure orchestration is restarted, it may not be able to delete all of the VM instances that were in progress. Possible cause 56 Troubleshooting procedures

57 VMs that were in progress at the time of Matrix infrastructure orchestration restart may not be uniquely identified as created by Matrix infrastructure orchestration, and cannot be deleted by Matrix infrastructure orchestration. Suggested action The Matrix infrastructure orchestration request history will indicate that it was unable to successfully cleanup the failed request. Check to see if the indicated VMs are still present on the hypervisor, and manually perform cleanup/deletion of the VMs as needed. Add disk operation is not working with SPM in VM cluster environment When adding a disk, the following error message appears: There is at least one vm running or defined in a vmhost, it should be removed or migrated before executing the requested operation. Possible cause Either there are VMs running, or Matrix infrastructure orchestration is unable to determine if VMs are running. Suggested action Remove or migrate any existing executing VMs on this VM host or fix any VM host communication issues. Ignite-UX troubleshooting Ignite-UX is missing as a software source In the infrastructure orchestration console, Ignite-UX is not listed as the software source in the Software tab. An Ignite-UX server has not been setup properly. Discovery for the Ignite-UX server has not completed properly. The Ignite server was registered with the CMS as a deployment server, but was not discovered in HP SIM. The /tmp directory on the Ignite server was 100% full. If output cannot be written to /tmp, the tool will fail on CMS/HP SIM. The privilege elevation level for the Ignite server has been enabled on the CMS. On the Ignite-UX server, use the following command to ensure depots have been set up: swlist l depot. If there is no depot listed, you need to ensure the depots are configured correctly. Ensure the Ignite-UX server is discovered correctly in the HP SIM with root credentials. After the successful discovery, login as Administrator on the CMS and set it as Ignite-UX server with this command: mxnodesecurity a p dsc_ignite c root:password n <ipaddress of Ignite-UX server>. Discover the Ignite server into HP SIM. From the HP SIM menu, select Options discovery. Clean up the /tmp directory. Once the /tmp has free space, the Ignite-UX image displays in the Matrix infrastructure orchestration software tab. Disable the privilege elevation level. From the HP SIM menu, select Options Security Privilege Elevation. Uncheck the Enable privilege elevation for Linux and HP-UX box. Matrix infrastructure orchestration troubleshooting 57

58 Ignite-UX deployments do not begin When trying to provision Ignite-UX systems, the target client waits for the Ignite-UX server to begin installation. Insight Control server deployment and Ignite-UX PXE servers are in conflict on the same network subnet. There are several suggested methods to resolve PXE collisions on the same subnet: Use separate subnets for Ignite-UX and Insight Control server deployment provisioning, one PXE responder per subnet. Add PXE timeout delays to the Insight Control server deployment responder, allowing the Ignite-UX server to reply first. Limit the responders with appropriate LAN MAC address ranges. For more information, see the Ignite-UX Administration Guide at the following website: Ignite-UX deployment fails with PXE error Ignite-UX deployment fails with the following message displayed on the console during infrastructure orchestration service creation. PXE-E18: Server response timeout Fully automatic Ignite-UX provisioning through infrastructure orchestration currently requires DHCP configured on the network. Either there is no DHCP configured for the network, or the configured DHCP is not functioning. Ensure that a DHCP server has been setup properly. If you do not have a DHCP server configured, perform the following steps to create one on the HP-UX server or the server available for DHCP clients. 1. Log on to the server for DHCP configuration as the root. 2. Start SAM. 3. Press n for Networking and Communications. 4. Press s for Network Services Configuration. 5. Press b for Bootable Devices. 6. Press b for DHCP Device Groups Booting From this Server. 7. Go to Actions Enable Boot Server and Add DHCP Group (specify DHCP Address Range). Integrity blade provisioning fails during the Ignite-UX installation phase The Remote Console of the Integrity blade being provisioned displays a user prompt during the HP-UX installation phase. The automatic provisioning fails eventually without manual intervention. The user prompt may request entry of the keyboard language. The user prompt may request the terminal type for the Ignite-UX instance to be installed. 58 Troubleshooting procedures

59 Set the_hp_keyboard in the /var/opt/ignite/data/config.local on the Ignite server to the appropriate language. For English: _hp_keyboard="ps2_din_us_english. Set an appropriate TERM to the terminal type on the Ignite server. For example, TERM=vt instl_adm d > /tmp/cfg 2. cp /tmp/cfg /tmp/cfg.orig 3. vi /tmp/cfg 4. Add a line: env_vars+="term=vt100" to /tmp/cfg 5. instl_adm f /tmp/cfg Request History shows an unexpected error has occurred on an Integrity blade Matrix infrastructure orchestration request history displays the following error message: An unexpected error has occurred. Check Alc1_0.0.log and HP SIM log files under HP SIM logs directory for additional details. The SAN disk cannot be seen from the blade. The Integrity blade connection in the enclosure might not be correct. The Integrity blade HBA (QLogic card) firmware might have a problem or might be corrupted. If it is corrupted, then the firmware version will not be shown correctly from the remote console of the blade. The zoning for the SAN disk is not correct. Verify all hardware configurations in the enclosure are correct and reseat the blade if necessary. Check under /var/opt/ignite/clients directory on the Ignite-UX server for the client that has failed provisioning. Look at the install.log, for the following information: NOTE: There were no disk devices found during the scan. Make sure that the destination disks are connected and powered on. You may choose to scan for more disk drives from next menu on the console. Go to the remote console of the Integrity blade and check the Console Log dump for evidence of firmware corruption. For replacement, contact your local HP Authorized Service Provider. If no evidence exists that the QLogic HBA is corrupted, update the EFI/FW to the current version supported by CloudSystem Matrix. For installation instructions, see the HP CloudSystem Matrix Release Notes at Make sure the storage WWN and server WWN are in synch between the allocated SAN disks, fabric switch and the storage pool entry from your CMS. The zoning is required for the physical Integrity blade provisioning. Error in Request History when erasing Integrity logical server disks The following message displays in the infrastructure orchestration request history: Failure: Error erasing disks for logical server. Manually erase the disk for this logical server and then delete this logical server from Matrix OE logical server management in Matrix OE visualization The delete Integrity server service from infrastructure orchestration failed. Matrix infrastructure orchestration troubleshooting 59

60 The IP address of the new Integrity server was not merged correctly with the blade node on HP SIM. The IP address of the Integrity server needs to be merged into the blade node from HP SIM. Perform the following steps after the Integrity server has booted up completely. 1. Fully discover the new Integrity server node from HP SIM, or: a. Add the root/password of the new Integrity server to the global credentials from HP SIM by selecting Options Security Credential Global credentials. b. Execute mxnode a ip w from the command line. 2. Deploy the HP SIM agent to the server either from the GUI, or via the command line mxagentconfig -a -n <target IP or DNS name> -u root -p <password>. Unable to power off device bay occupied by an Integrity blade because the operation timed out The following messages displays in the infrastructure orchestration Request History: Unable to power off device bay number {#} of enclosure {name} because the operation timed out. Check the general status of the blade on which the logical server was activated. An intermittent problem with press and hold and HP ilo issue on this Integrity blade. Select Tools Integrated Console VCEM. Then select Bays and power off by clicking Press and Hold. Once the blade is powered off, follow the regular manual cleanup procedure for physical provisioning documented in the HP Matrix Operating Environment Infrastructure Orchestration User Guide located at Matrix recovery management troubleshooting NOTE: CMS host names, storage management servers, hypervisors, etc., must be unique across sites in an Matrix recovery management configuration. NOTE: When planning a disaster recovery solution using Matrix recovery management with VMware hypervisor based IO services, proper planning is required to ensure the hosts at both the primary and remote sites match. For example, the hosts that have visibility to the replicated volume that the IO service uses and can be selected to activate the replica services at the DR site must be running a version equal to or higher than the primary site host on which the original IO service was created. For example, if the primary IO service was created on a vsphere (ESXi) host version 5.5, the DR site vsphere (ESXi) servers that have the replicated datastore mapped must be running vsphere (ESXi) 5.5 or later. Matrix recovery management Activate or Deactivate job fails Failure of a Matrix recovery management Activate, Deactivate, Recovery Group import or Site import job, may intermittently occur. Suggested action By default, Matrix recovery management automatically retries failed activation and deactivation operations on logical servers. However, if a Recovery Group import or site import job fails, there is no automatic retry, and the restart option is not available. To disable automatic retry: 60 Troubleshooting procedures

61 1. Edit the MAX_RETRY_ATTEMPTS property in the \conf\hp_ir.properties file where Matrix recovery management is installed. 2. Set the MAX_RETRY_ATTEMPTS property to 0. To re-enable automatic retry: 1. Edit the MAX_RETRY_ATTEMPTS property in the \conf\hp_ir.properties file where Matrix recovery management is installed. 2. Set the MAX_RETRY_ATTEMPTS property to 1. NOTE: Failed Activate and Deactivate jobs will be retried only after the operation has been attempted on all of the logical servers in the configuration. Job states for logical server and Recovery Group jobs will not be marked failed when a retry of the operation is pending. Matrix recovery management change local site job failed A "Change Local Site to Primary" or "Change Local Site to Recovery" job failed because one or more logical servers could not be activated or deactivated. Suggested action 1. Click the check box next to the Job Id to get detailed information about the associated Sub Jobs. A site Job contains a Sub Job for each Recovery Group. Similarly, each Recovery Group has Sub Jobs for its Storage Replication Group and logical server, respectively. 2. Drill down to each of the associated failed Sub Jobs to determine what operation failed and the reason for the failure, as shown in Figure 11 (page 61): Figure 11 Matrix recovery management Jobs Job Id 3. Once the problem has been corrected, return to recovery management. If the correction included reconfiguration of logical servers, go to the Recovery Groups tab and delete the Recovery Groups that contain the re-configured logical servers. 4. From the Jobs screen, restart the Job by clicking on restart in the Status column for the failed Job as shown in Figure 12 (page 62). Matrix infrastructure orchestration troubleshooting 61

62 Figure 12 Matrix recovery management Jobs restart NOTE: Restarting the Job retries only Sub Jobs that previously failed; servers associated with completed Jobs or Sub Jobs are already up and running and will not be affected. Failure in storage replication failover The Storage Management Server is not accessible during the time of the failover or the status of the Storage Replication Groups does not permit a storage failover. Ensure that the local Storage Management Server is accessible and actively managing one of the storage solutions: For HP P6000 storage solutions, view the STORAGE\EVA\log\clxevarun.log where recovery management is installed on the CMS. For HP P9000 storage solutions, view the logs\clxrun.log where HP P9000 Cluster Extension Software is installed on the local storage management server. For HP 3PAR StoreServ Storage solutions, view the logs\clx3parrun.log where HP 3PAR Cluster Extension software is installed on the CMS. Logical server not located during failover job A logical server may have been deleted using Matrix OE visualization without removing the Recovery Group containing the logical server from recovery management. Remove the logical server from the Recovery Group in recovery management and rerun the failover job. Also look at the lsdt.log file in the logs directory where recovery management is installed. Failover job succeeds, but logical servers are not activated Recovery Groups containing logical servers which are not activated may be in Maintenance Mode. Disable Maintenance Mode on the impacted Recovery Groups and rerun the failover operation. Also look at the lsdt.log file in the logs directory where recovery management is installed. Failover job fails because of insufficient servers There may not be enough licensed physical resources capable of hosting the logical servers. 62 Troubleshooting procedures

63 Ensure that there are enough physical servers licensed for Matrix OE available to host the logical servers managed by recovery management. Also look at the lsdt.log file in logs directory where recovery management is installed. Logical server troubleshooting Portable logical server must not include local storage Do not configure your portable logical server to include local disk storage. VC blades and hypervisors using local disk storage cannot be migrated without losing storage access. Suggested action If your logical server does not need to be portable, you can use local disk. Local disk support is not available through Matrix OE logical server management You cannot create a logical server with local disk using the Matrix OE logical server management GUI or import one. Suggested action You can specify local storage in an infrastructure orchestration service template and manually provide local disk information, and infrastructure orchestration will create the appropriate logical server using local storage. See provisioning physical servers using local disk in the HP Matrix Operating Environment Infrastructure Orchestration User Guide available at go/matrixoe/docs. Do not unmanage a logical server that has been previously activated Do not select the Unmanage Logical Server option to unmanage a logical server that has been previously activated. Unmanaging the logical server may result in an inconsistent state and may render the storage pool entries unusable. An exception to this recommendation is managing the profile using VCEM. Suggested action If you mistakenly select the Unmanage Logical Server option, follow the procedure to configure storage pool entries, FC zones, and disk array presentations in the HP Matrix Operating Environment Infrastructure Orchestration User Guide found at Physical disks with multiple partitions are unsupported unless all but the first partition are excluded If your environment contains a physical disk with multiple partitions (volumes with the same disk identifier) and you want to provision VMs on that disk, you must exclude all partitions except the first partition on the disk. Otherwise, the service could fail to deploy with the following error message: Import failed for logical-server. VM host datastore not found. Suggested action Exclude all partitions except the first partition on the disk in the VOLUMES TO EXCLUDE section of the hpio.properties file. For example, if there are three volumes (C:, D:, and E:) on a physical disk, exclude the D: and E: volumes to provision a logical server on the VM created on the C: volume of the physical disk. Matrix infrastructure orchestration troubleshooting 63

64 Unmanaging a logical server leaves WWNs used by VC profile unavailable After a logical server is unmanaged, any WWNs still being used by the underlying VC profile become unavailable. The storage pool entry is marked as Inoperable and the Ready check boxes for the unavailable WWNs are grayed out. These WWNs cannot be assigned to a new logical server while in this state. If you want to reuse the storage pool entry but do not need to keep the original WWNs: 1. Modify the storage pool entry to increase the Maximum Number of Sharers by 1. This assigns a second set of WWNs to the storage pool entry and allows the storage pool entry to be reused. 2. Modify the storage pool entry to replace the unusable WWNs with new WWN values. If you want to reuse the original WWNs, free them using one of the following methods: 1. If the VC profile is already assigned, use VCEM to unassign and delete the VC profile. 2. Using Tools Logical Servers Refresh..., refresh the VCEM GUID Server. The WWNs in the storage pool entry become available. OR 1. If the VC profile is not already assigned to a blade that is powered on, use VCEM to assign the VC profile to a server blade and power it on. 2. Using Tools Logical Servers Import..., import the server blade, which creates a new logical server that is automatically associated with the storage pool entry. 3. Deactivate the logical server. 4. Delete the logical server, which frees the storage pool entry and WWNs. Storage pool entries marked Inoperable or available storage is not available Storage pool entries are marked Inoperable and no logical server has been unmanaged. Matrix infrastructure orchestration cannot find available storage pool entries when trying to deploy a new service and displays the error: Provisioning request has paused. The necessary disks are not available. Manual storage allocation using VSE logical server storage pools is required. Possible cause The VCDG was deleted and recreated, causing a new portability group to be created for the new VCDG, even if the VCDG was recreated with the same name. The previously created storage pool entries remain in the old, now unavailable, portability group. Suggested action 1. Select Tools Logical Servers Refresh. 2. Check the Virtual Connect Enterprise Manager and Change Logical Server Associations checkboxes. 3. Enter the old VC domain group name and new VCDG name (even if they are the same), then click Refresh. The storage pool entries are moved from the old portability group to the new portability group so that they can be used again. 4. If an infrastructure orchestration provisioning request was paused, go to the Systems Insight Manager menu and select Tools Insight orchestration. Then select the Requests tab. Select the paused service request, and click Continue. 64 Troubleshooting procedures

65 Storage pool entry status is Presentation Completed with Errors Inoperable or Presentation Completed Inoperable After creating a storage pool entry from an array imported into SPM, the status of the storage pool entry on the Matrix OE visualization Manage Storage Pool screen is one of the following: Presentation Completed with Errors Inoperable Presentation Completed Inoperable Possible cause The error occurs when trying to activate/provision one or more storage volumes within SPM after the storage pool entry was saved. The error can occur due to: a communication problem with the SAN array a communication problem with the FC switch a volume service that becomes non-conformant within SPM (for example, if the volume was deleted directly from the array) Suggested action From the Manage Storage Pool screen, view the storage pool entry details to determine the identity of the volumes and reasons why the storage pool entry is inoperable. If more details are needed from SPM to determine the underlying cause, an SPM administrator can perform the following steps. 1. Start SPM by pointing a browser to localhost is the name of the CMS. 2. Look at the services associated with the Inoperable storage pool entry to see if any are non-conformant. The service names within SPM embeds the storage pool entry name. 3. Look at the SPM job history to see if any Activations associated with the storage pool entry failed. (Expand the job details for the detailed reason of the failure.) Based on the reason for the error, take the appropriate corrective action to resolve the issue. Matrix infrastructure orchestration pauses for manual zoning information on auto-generation storage When using the storage auto-generation feature from within infrastructure orchestration, if manual zoning is required IO pauses to wait for the Storage Administrator to perform the manual zoning. The IO Request turns yellow, and a notice saying that the request is paused for manual zoning is shown at the top of the Request Job table. Suggested action 1. From the Request job table, copy the manual request message. An example of this request message is shown below. 2. Send the manual request message to the Storage Administrator so that they can create the appropriate zoning within the FC switch. SPM requires that the new server's WWN be manually Matrix infrastructure orchestration troubleshooting 65

66 added to the XML file defining the Unmanaged Network to indicate to SPM that the zoning has occurred. Unable to update storage pool entry after deleting service request in SPM The following error message displays when trying to modify a storage pool entry: Unable to update Storage Profile. The given profile was not found. Possible cause The storage pool entry profiles were deleted in Matrix OE logical server management so the relationship between the storage pool entry s and the VC profiles has been broken and the storage pool entry cannot be recovered. Suggested action To salvage the zoned WWNs and recreate the storage pool entries one by one: 1. Capture the WWNs that you are using for each storage pool entry. NOTE: If you had exported the WWNs, use the lsmutil import spe command to capture the WWNs. 2. Delete the storage pool entries. 3. Recreate the storage pool entries one at a time using the capture WWNs from the deleted storage pool entries. 4. Re-add all the SPM volumes as new candidates in the same order you had previously done. Recovery after logical server operation failures In the Report Logical Server Job Status screen, an error message similar to one of the following displays: Unable to unregister virtual machine from VM Host.{(Virtual Machine not found), (Host is not connected)} Operation failed. Cannot open connection. The resource {.vmx file} required to activate logical server {logical server name} is not found. Environment configuration problems. The resource became unavailable during the runtime of the logical server operation. The resource has been changed outside of logical server management. For example, the VM was deleted or the server blade profile was changed or deleted. Communication between VCEM and VCM is temporarily unavailable. 1. Ensure that the following configurations are set correctly in your environment. Failure to do so can cause errors. DNS is configured for all managed servers. Time synchronization has been performed among the CMS, managed systems (including ESX hosts), and VMware vcenter. VMware vcenter credentials and URL settings in HP SIM are correct. Onboard Administrator credentials and URL settings in HP SIM are correct. 66 Troubleshooting procedures

67 2. Refresh the server resources to reflect any environment configuration changes in the last 30 minutes (logical server management refreshes automatically every 30 minutes, while the following steps perform the refresh immediately): For a VC logical server, access the Matrix OE visualization screen, select Tools Logical Servers Refresh, then select Virtual Connect Resources. For a VM logical server, select Tools Logical Servers Refresh, then select Virtual Machine Resources. 3. Try the logical server operation again. If it is a temporary resource problem, another try may succeed. 4. If the above steps do not correct the problem, unmanage the logical server. From the Matrix OE visualization screen, select Delete Unmanage Logical Server. 5. Import the server with VC or VM as a logical server when the resource is available. Synchronizing the clocks on the hosts and VMs If you are performing operations on ESX VM logical servers, be sure that you have synchronized the clocks on the hosts and VMs. If the VM hosts are managed by VMware vcenter and the host clocks are not synchronized, some issues might occur while managing the VMs that might affect logical servers. Use clock synchronization software such as the Windows Time Service or the Network Time Protocol to synchronize the host clocks. Recovering an inoperable logical server s On the Matrix OE visualization user interface a logical server is marked inoperable, or on the logical server job status screen an error message reports that a logical server is inoperable. Review Table 1 (page 67) for several possible causes and suggested actions. Table 1 Recovering an inoperable logical server The server with VC or VM host Perform one of the following actions: resources associated with a logical Refresh the appropriate logical server resources (VC or VM) using server has been made unavailable Tools Logical Servers Refresh... for further management operations. Examples are: Reverse the operation that made the underlying server blade or VM host unavailable for management operations. If the underlying resource is a blade, Using VCEM to remove a consider restoring the domain group or bringing the domain group back from domain group from management maintenance mode. For a VM host, re-register the VM host with Insight Control virtual machine management, or restart the VM host. or to move a domain into maintenance mode Using Insight Control virtual machine management to unregister or shutdown a VM host Refresh the appropriate logical server resources (VC or VM) using Tools Logical Servers Refresh... Unmanage the inoperable logical server. Both the blade and VM guest resources of a cross-technology logical server have been activated and are operating simultaneously. When blade or VM guest resources associated with logical servers are activated using VCEM or Insight Control virtual machine Perform the following sequence of steps: 1. Unmanage the inoperable logical server 2. Determine the underlying resources that is the intended instance, and delete the other instance. Delete the underlying VM with Insight Control virtual machine management, or delete the blade instance using VCEM Verify the integrity of the remaining instance; if necessary, restore the remaining instance from backup and unmanage and re-import the recovered instance. If Matrix infrastructure orchestration troubleshooting 67

68 Table 1 Recovering an inoperable logical server (continued) management, dialog boxes display warning that such activation is not recommended. Inoperable logical server appears immediately after a failure of the logical server management service or of the entire CMS. Partially completed operations, such as move operations, will not resume after the logical server management service or CMS restarts, and the CMS may not be able to synchronize the state of the recovered underlying resource with the logical server. Logical server is marked inoperable during a roll back of VM registration after credentials were not retrieved from HP SIM. the integrity of the remaining instance cannot be verified then remove it and recreate. Unmanage and re-import the impacted logical server. For an ESX VM logical server, unregister the VM using Insight Control virtual machine management. For a Hyper-V VM logical server, use the Insight Control virtual machine management CLI to unregister the VM. Perform the steps under Unregister the VM below. 1 VCEM displays a warning when it detects that Matrix OE is installed and you are attempting to perform management operations on a server blade associated with a logical server. Performing these operations using VCEM may lead to inoperable logical servers. Take care to ensure that the blade instance (profile) you are deleting is correct, then type YES to proceed. The warning message reads as follows: IMPORTANT: VCEM has detected you may be using other products such as Matrix OE. Assigning this Server Profile can make it inconsistent with the upper level manager. It is recommended to use the upper level manager to assign this Server Profile. Type 'YES' to proceed with this operation. Unregister the VM 1. Change directory to the installed Insight Control virtual machine management directory and to clientapi\bin C:\Program Files\HP\Insight Control virtual machine management\clientapi\bin 2. Authenticate the session using the adminlogin.cmd command: C:\Program Files\HP\Insight Control virtual machine management\clientapi\bin>adminlogin.cmd Login successful. 3. Find the VM URI or configuration path cli getvmids C:\Program Files\HP\Insight Control virtual machine management\clientapi\bin> cli getvmids vm:// /c:\vse_virtualmachines\vsels_ls-hyperv-01\ Virtual Machines\880401BA E76-9CD3-AA5F706F3332.xml 4. Unregister the VM using cli unregistervm vm vmuri C:\Program Files\HP\Insight Control virtual machine management\clientapi\bin>cli unregistervm -vm "v m:// /c:\vse_virtualmachines\vsels_ls-hyperv-01\ Virtual Machines\880401BA E76-9CD3-AA5F706F3332.xml ok C:\Program Files\HP\Insight Control virtual machine management\clientapi\bin> 5. Refresh logical server Insight Control virtual machine management resources using Tools Logical Servers... Refresh Troubleshooting procedures

69 Logical server fails to power on when logical server is activated with power on When a logical server is activated with the power on option, in rare cases the activation step might succeed while failing to power on the logical server. This is indicated by a job status of SomeFailure Power on the logical server by selecting Tools Logical Servers Power On. Logical Server Automation service fails to start if TCP layer ports are in use Logical Server Automation service fails to start and displays the following error message: The nnn port used by the Logical Server Automation service is currently in use. Ensure that there is a free port, edit the lsaclient.properties file to replace the port in use with a free port, and restart the service The Logical Server Automation service reads the TCP layer port from the lsaclient.properties file. By default, the Logical Server Automation service uses the following port numbers: LSA_RMI_PORT=51001 LSA_JMS_PORT=51002 If either of these ports is currently in use, the Logical Server Automation service will look for open ports using a random algorithm. If the service finds available open ports, the service will write the port numbers into lsaclient.properties. If the Logical Server Automation service cannot find two available ports, the service will fail to start. Edit the lsaclient.properties file and replace the port in use with a free port, and restart the service. To determine a free port, use the command netstat -A. VM logical server is incorrectly reflected in the Logical Server Status The power state of a VM logical server is incorrectly reflected in the Logical Server Status post crash and recovery of the VM logical server. The HP SIM and Insight Control virtual machine management databases may contain information about the VM logical server before the crash and may not have been refreshed post crash and recovery. Initiate a Matrix OE logical server management Refresh operation post recovery to obtain the updated power status. Select Tools HP Matrix OE visualization. Then select Logical server from the Perspective drop-down menu. Then select Tools Logical Servers Refresh. Choose HP SIM and Insight Control virtual machine management subsystems. VM logical server is reflected as Active on unknown host while performing an activate operation The VM logical server is reflected as Active on unknown host after a recovery is completed on the VM host that crashed while performing a VM logical server activate operation on the host. The activate operation was completed successfully post crash. Matrix infrastructure orchestration troubleshooting 69

70 The HP SIM and Insight Control virtual machine management databases may not contain the latest information on the VM guest associated with logical server and the associated VM host post recovery. Initiate a re-discovery of the recovered VM host followed by a Matrix OE logical server management Refresh operation of HP SIM and Insight Control virtual machine management. Re-run an HP SIM discovery task for the VM host followed by a refresh. VM logical server is reflected as Active on unknown host while performing a move operation The VM logical server is reflected as Active on unknown host after a recovery is completed on the target VM host that crashed during a VM logical server move operation to the host. The HP SIM and Insight Control virtual machine management databases may not contain the latest information on the VM guest associated with the logical server on a VM host post recovery. Initiate a re-discovery of the recovered VM host followed by a Matrix OE logical server management refresh operation of HP SIM and Insight Control virtual machine management. VM logical server is incorrectly activated after a CMS crash recovery The VM logical server is incorrectly activated after the CMS is recovered from a crash during a VM logical server delete operation. The VM logical server was in a deactivated state before the crash. If the CMS crash occurred while the Insight Control virtual machine management was in the process of deleting and un-registering the VM, the previously activated and registered state of the VM is maintained, and the VM logical server is activated after recovery. In the process of deleting the VM logical server, the Insight Control virtual machine management registers the VM (associated with the logical server) with a VM host where the VM was registered before deactivated. When a CMS crash occurs during registration verification, the Matrix OE logical server management activates the VM logical server after recovery since the VM is registered with a VM host. Re-Initiate the VM logical server delete operation after recovery is completed. Choose a logical server that is already de-activated. Select Delete Delete Logical Server. VM host does not appear on move logical server target screen A VM host does not appear on the Move: Assign Logical Servers to Target Hosts screen in the Available Target Hosts and Rejected Target Hosts tables, or a VM fails to be imported with a "cannot find VM" or "cannot find VM Host" message. The UUID is null or missing. The VM hosts do not have the proper licenses installed. The target VM host is not discovered during the host Discovery phase. 70 Troubleshooting procedures

71 Integrity VM targets: The private network between the source and target servers does not have SSH set up properly. Verify that the VM host is properly configured and has a UUID. To perform this check, view the HP SIM system page for the VM host and look for the UUID field. Perform the following steps if vselog.lsm.log contains the error: :58:41,675 [pool-8-thread-9] WARN app - (LSM) Populate VM Host ( ) failed with exception: LSM_ADPT_VMM_ERR_NO_VM_HOST_UUID For more information, see the HP Insight Control virtual machine management documentation at 1. Identify the system using HP SIM. 2. Check the VM host status in HP SIM. 3. Unregister the host, then register it again. 4. When the VM status is green, refresh Insight Control virtual machine management resources using the Tools Logical Servers Refresh... menu selection. Run the Insight managed system setup wizard to verify that the licenses are properly installed on all of the managed servers. Verify the credential settings are properly configured on the VM host and rerun the host Discovery process. Integrity VM targets: To enable secure communication between the source and target hosts, generate SSH keys on both systems. You need root privileges to generate and set up the SSH keys required for guest migration. Use the secsetup script provided by Integrity VM. Execute the following command on both the source and target hosts: # /opt/hpvm/bin/secsetup -r otherhost For more information, see the HP Integrity Virtual Machines 4.3 or 6.1 Administrator Guide at Unable to import an Integrity VM logical server The following error displays on the Import Logical Server screen when attempting to import an Integrity VM guest as a logical server: Error: Import failed for system-name. InvalidStorage. The Integrity VM being imported is not properly configured with the appropriate storage devices. You must use whole LUNs or virtual disks created with SLVM LV in VGs with the CONTAINER_VOLUME_SLVM type. Ensure that the Integrity VM storage is properly configured with a whole LUN, or virtual disk created using a SLVM LV. The volume group that contains the SLVM LV must have a container type of CONTAINER_VOLUME_SLVM. For more information, see the HP Integrity Virtual Machines 4.3 or 6.1 Administrator Guide at Unable to create an Integrity VM logical server An Integrity VM logical server creation failed because the VM host did not have the SLVM volume groups registered in the Integrity VM device management database. Figure 13 (page 72) and Matrix infrastructure orchestration troubleshooting 71

72 Figure 14 (page 73) illustrate the logical server management and infrastructure orchestration errors that are displayed as a result of this issue. Figure 13 Matrix OE logical server management 72 Troubleshooting procedures

73 Figure 14 Matrix infrastructure orchestration The required LVM software is not properly installed. Ensure that SG and SLVM volume groups are properly created and added (with the CONTAINER_VOLUME_SLVM container type) to the device management database. Ensure the SLVM volume groups are properly activated in shared mode (vgchange -a s). Logical server boot errors when booting as an Integrity VM guest Logical servers experience boot problems when a long form EFI boot path does not exist. Symptoms include failure to boot and P2V failures with the following error message: Failed to Update the VM's Resources. Exception: Unable to modify the guest - The HPVM API used to set the EFI failed, most likely an invalid EFI passed in. Possible cause HPVM and Integrity blades use different versions of EFI. Integrity blades with VC can boot HP-UX using the FC boot WWN and LUN specified in the server profile created by Matrix OE logical server management. However, Integrity VM does not have this capability. Thus, a logical server with a pre-provisioned disk with an OS can be booted on a VC blade, but after P2V the logical server cannot boot as an Integrity VM guest. The boot path inserted by VC is not a long form EFI boot path as it does not include a disk signature and the full path to the EFI boot file. Furthermore, each time the logical server is moved back to the VC blade from Integrity VM (V2P), an additional boot path might be inserted by VC. This happens because the boot path inserted by VC is modified upon import into the Integrity VM guest so when the logical server returns to the VC blade, it s no longer functional. Matrix infrastructure orchestration troubleshooting 73

74 Suggested action Install a valid long form boot path before performing P2V and V2P. There are several options for creating the boot path including: Use the EFI boot manager to add a boot path. If HP-UX is installed, boot it and use the setboot -p <path> command. If HP-UX is not installed then when it is installed a boot path will automatically be added. Editing an Integrity VM outside of logical server management results in an error with the storage port A virtual FC HBA added to an Integrity VM guest outside of Matrix OE logical server management will not add a storage port to the guest s associated logical server. Possible cause Changes or additions of a virtual FC HBA (NPIV) are not reflected in the associated logical server on the next refresh of Insight Control virtual machine management resources. As a result, the logical server and the Integrity VM guest will not be in synch which causes problems if the logical server is subsequently moved to a VC blade. Suggested action Avoid managing a Integrity VM guest s virtual FC HBA using tools external to Matrix OE logical server management when the guest is associated with a logical server. For example, do not use Integrity Virtual Server Manager to add a new virtual FC HBA device to the VM created using a logical server nor run hpvmmodify command to add a new virtual FC HBA device to the VM created using a logical server. Networking error occurs when attempting to activate a logical server on Integrity VM Activation of a logical server on Integrity VM fails with the following error: The operation has failed. (The save Configuration operation on virtual machine configuration [vm:// /vsels_g1] failed: [managevnics] Error: [addvirtualsystemresources] : Failed to Add the VM's Resources. Exception: Unable to add guest resources - Network device add failed.) And an error similar to the following appears in the Integrity VM command.log (/var/opt/hpvm/common/command.log): 09/01/11 10:40:38 ERROR host root Invalid Ethernet MAC Address '0x00215A9B0004' - Linkloop failure. Possible cause A duplicated GUID is being used to attempt to create a new VM on the same VM host. This error can happen in the following scenario: 1. Create and activate a logical server on Integrity VM. The guest gets a MAC address allocated from VCEM GUID server. 2. Reinstall the CMS for upgrade. The Integrity VM guest remains installed. 3. Create and activate a new logical server on Integrity VM. The guest gets a MAC address identical to the first guest s because VCEM GUID server s allocation pool was reset during reinstallation. Note that the hosts for the logical servers in steps 1 and 3 do not necessarily need to be the same one. Suggested action Import the existing guest; its MAC address will be reserved by the VCEM GUID server. 74 Troubleshooting procedures

75 An Integrity VM target host is rejected after a P2P move of a VM host After manually performing a P2P move without activation, the activation fails. When attempting to activate an Integrity VM logical server, the target VM host is incorrectly put in the Rejected Target List with a hard exception of: Device is unavailable. If the Integrity VM host is moved back to its original location, it will appear in the Available Target list for VM logical server activation. Suggested action 1. Edit the portability group of the logical server and remove the portability group entries for both the source and target. a. Access Tools Matrix OE logical server management menu. b. Select Modify Logical Server Portability Groups... and select the portability group to edit and remove the entries. 2. In HP SIM, re-register the Integrity VM host for virtual machine management and re-identify the host. 3. Perform a logical server refresh. Tools Logical Servers Refresh Edit the portability group and re-add the entries for the newly-registered VM host. 5. Activate the VM on the Integrity VM host which is now listed in the target list. Importing an Integrity VM guest fails with no boot path An Integrity VM guest configured with SAN-based storage fails with the following error during an import operation: Import failed. The HP Integrity VM guest uses NPIV storage but has no boot path configured. A boot path is typically inserted when HP-UX is installed; or by the HP-UX setboot command, or through the EFI Boot Maintenance Manager. Possible cause A boot path has not been configured. This issue may occur in the following scenarios: 1. Creating a new Integrity VM guest using SAN-based storage and a LUN that has not been provisioned with an OS: a. Create an Integrity VM guest on the VM host using the hpvmcreate command. b. Start the guest. c. Try to import the guest into Matrix OE logical server management using the import operation. The import fails. 2. Creating a new Integrity VM guest using SAN-based storage and a LUN that has been previously provisioned with an OS: a. Create an Integrity VM guest on the VM host using the hpvmcreate command. b. Start the guest. c. Boot the OS directly from EFI. d. Try to import the guest into Matrix OE logical server management using the import operation. The import fails. Matrix infrastructure orchestration troubleshooting 75

76 1. Add a valid boot entry. If the guest is not booted, add a valid boot entry for the Integrity VM guest using EFI. If the OS is booted, add a valid boot entry for the Integrity VM guest using the setboot command. 2. Run a Matrix OE logical server management refresh and select the virtual machine management entry. EFI boot paths are empty after adding an LFBP and then performing a V2P move EFI boot paths may disappear if a new LFBP is added after an automatically generated VC Primary boot path is created and a V2P move is performed. Possible cause If an LFPB is added after an automatically-generated VC Primary boot path is created, there is both an LFBP and a VC Primary pointing to the same path. However, the VC Primary is Short Form and the LFBP is Long Form. After performing a P2V cross-technology move, the move completes successfully. But when performing a V2P cross-technology move, the server blade can not boot and EFI shows the boot paths are empty: Shell> bcfg boot dump The boot option list is empty. Suggested action 1. After creating the logical server and activating it on a server blade, power up the server blade to EFI. 2. Add an LFBP. 3. Move it to the top with the bcfg boot mv command. For example, the boot option list is: 01. PcieRoot(0x )/Pci(0x5,0x0)/Pci(0x0,0x0)/Fibre(0x207000C0FF1145FB,0x )/HD(1,GPT,F463191C-7CF5-11E D6217B60E588)/\EFI\HPUX\hpux.efi "My new SAN boot path" 02. PcieRoot(0x )/Pci(0x5,0x0)/Pci(0x0,0x0)/Fibre(0x207000C0FF1145FB,0x ) "VC Pri. Fibre(0x207000C0FF1145FB,0x )" 4. Remove the VC Primary path. If you have already moved the logical server to a server Blade, them temporarily move it back to a VM logical server and then remove the VC Primary path. Failure during VM deletion leaves active but unpowered logical server A failure occurs during the delete operation causing the VM to be left active. The Report Logical Server Job Status... screen shows the progression of job detail messages for a delete operation. After the failure, the delete job appears to be incomplete, and the logical server is active, but not powered on. 1. Refresh Insight Control virtual machine management server resources using the Tools Logical Servers Refresh... menu selection. 2. Deactivate the logical server using the Tools Logical Servers Deactivate... menu selection. 3. Repeat the delete operation using the Delete Delete Logical Server... menu selection. 76 Troubleshooting procedures

77 Deactivating or powering off logical server causes license error check Deactivating or powering off a logical server on an ESXi 4.0 or ESXi 4.1 host causes a license check error. After deactivating or powering off a logical server on an ESXi 4.0 or ESXi 4.1 host, you may see the following error message on the Report Logical Server Job Status screen: Unable to power off virtual machine [virtual machine name] This error should have an associated log entry in the HP SIM log of this form: 12 Mar 22:01:52,502 INFO [HPSIM_DEBUG] [Panic Logger-0] (Identification Via Discovery:239) Unique identifier for dev=com.hp.mx.core.nodemanager.mxnode@ffffffff : Key :-1 : Name :lsdtesx4 : state :1 changedflag :1727, is not the same. current= d a59, found by identification=d99f1009-fd55-44b4-842a-fbceafdc3a2b ESX hosts running ESXi 4.0 or ESXi 4.1 software obtained directly from VMware require the installation of the HP ESXi Offline Bundle for ESXi Server. This bundle includes the latest HP CIM providers. These providers collect many important system parameters, including: Smart Array Provider Smart Array controller, external storage enclosure, internal drive cage, data and spare drive and logical volume information Networking Providers Ethernet ports, statistics, port link status, IP and MAC addresses Download the latest providers for ESXi systems from the HP web portal and note the installation instructions located at the following website: prodtypeid=15351&prodseriesid= &switem=mtx-25f06077ad5541f5a962dd2a69& prodnameid= &swenvoid=4040&swlang=8&taskid=135&mode=4&idx=1 Logical Server Automation service may run out of memory on large scale systems On large scale systems, with large numbers of managed nodes and/or large numbers (1000+) of logical servers on a 64-bit CMS, the HP Logical Server Automation service may run out of memory and display out of memory errors in the /logs/hp_lsa_service.log or /logs/ vselog.lsm.log file. You may see the following errors: java.lang.outofmemoryerror: GC overhead limit exceeded java.lang.outofmemoryerror: Java heap space java.lang.outofmemoryerror: PermGen space Suggested action 1 To correct the garbage collection overhead limit and/or Java heap space errors, modify the configuration file to increase memory allocation: 1. Modify the configuration file (shown in the default location) at C:\Program Files\HP\ Virtual Server Environment\conf\hp_lsa_service.conf. 2. Increase the value of wrapper.java.maxmemory=1024. HP suggests a value of Restart the HP Logical Server Automation service. Matrix infrastructure orchestration troubleshooting 77

78 Suggested action 2 To correct the PermGen space error, modify the configuration file to increase permanent generation size: 1. Modify the configuration file (shown in the default location) at C:\Program Files\HP\ Virtual Server Environment\conf\hp_lsa_service.conf. 2. Add the following line. (Change additional.2 to additional.3 if a line with additional.2 already exists; change additional.2 to additional.4 if lines with additional.2 and additional.3 already exist, and so on.) wrapper.java.additional.2=-xx:permsize=32m -XX:MaxPermSize=128m 3. Restart the HP Logical Server Automation service. Profile is created but not associated with logical server when service is stopped during logical server activation If the first activation of a new logical server is in progress when the HP Logical Server Automation service is stopped, the logical server definition may be left in a state where the VC profile has been created but is not associated with the logical server. When the logical server is in this state, an attempt to activate it will fail with the following error: Unable to create profile {profile name}. (Cannot perform server profile operation because: Server profile name {profile name} is duplicate. Another server profile with the same name already exists in VCEM. Verify all the server profile information is correct and perform the operation again.) IMPORTANT: This error also occurs during the first activation of a logical server if the designated VC profile name is assigned to an existing profile. In this case, following the suggested action to delete the underlying profile will result in a loss of data if the profile is in use. To avoid data loss by accidentally deleting an existing profile, follow steps 1 and 2 below to verify that the activate operation did not successfully complete and that the VC Profile was created as part of the failed operation. 1. View the status of the activate logical server job using the Report Logical Server Job Status menu selection from the lower, gray menu-bar. Verify that the activate logical server job did not complete successfully. 2. Verify that the underlying VC profile definition is consistent with the logical server definition. a. View the logical server definition using the Report View Logical Server Details menu selection. Note the Managed Resource Name. b. View the VC profile by selecting Tools Consoles Virtual Connect Enterprise Manager (VCEM) from the upper, blue menu-bar. From the Server Profiles tab, select the VC profile and click Edit 3. Delete the VC profile using Tools Consoles Virtual Connect Enterprise Manager (VCEM) and selecting the Server Profiles tab. Select the profile, and click Delete Activate the logical server using Tools Logical Servers Activate... from the lower, gray menu-bar. Logical server is missing from Matrix OE visualization or has the wrong type During HP SIM Discovery a system was discovered with the same name as an existing HP logical server causing the logical server to be deleted. Or, HP SIM may show the node with both HP Logical Server subtype along with another subtype making it unusable. Suggested action 78 Troubleshooting procedures

79 Manually recreate the logical server with a unique name: 1. If the logical server is unusable, delete it. If the logical server has associations to other systems, delete all the associated systems first before deleting the logical server or delete the logical server by using the following command: mxnode -r <NODE_NAME> -x force 2. If the logical server had been active, re-import the logical server by selecting Tools Logical Servers Import. If the logical server had not been active, recreate the logical server with a different name from the VM host. 3. Refresh the server resources in Matrix OE visualization by selecting Tools Logical Servers Refresh. 4. Re-discover the server. If the server is a VM host system, rediscover the VM host whose name conflicts with the logical server. Integrity Virtual Server Manager troubleshooting Access failure occurs when attempting to invoke Integrity Virtual Server Manager Access failure occurs when attempting to invoke Integrity Virtual Server Manager. User is not authorized to monitor the VM host. VM host is not known to HP SIM or Matrix OE. User is not authorized to use Integrity Virtual Server Manager. The HP SIM/Matrix OE user must have at least Monitor Tools authorization for the VM host system. Both the virtual machine and the containing VM host must be discovered by HP SIM. Users need VSE All Tools authorization to view all Integrity Virtual Server Manager screens and perform all Integrity Virtual Server Manager menu actions. Failure starting an Integrity VM An Integrity VM fails to start due to missing, insufficient, or unavailable resources. There is not enough memory on the VM host available for the virtual machine. You specified a device that is already in use by another VM. A file specified as a backing device does not exist. The VM is currently migrating online or has migrated to another VM host (and is in a Not Runnable state). Make sure the required resources are made available. For example: Make sure that enough memory is available on the VM host to provide the memory you allocated for the new VM (in addition to providing for the existing VMs). You might have specified a network or storage device that is already in use by another VM. You need to return to the appropriate Create Wizard page to select a device that is not in Matrix infrastructure orchestration troubleshooting 79

80 use. To determine what devices are already in use, examine the VM Host Network or VM Host Storage tab. You might have incorrectly specified the name or path of a file to be used as a backing storage device, or you specified a file that does not exist. Failure to modify the Integrity VM vcpu entitlement If you attempt to modify (by selecting Modify Virtual Machine vcpu Entitlement) the vcpu entitlement of a VM that is being managed by gwlm, the following error message is displayed, where virtual-machine-name is the name of the VM: The processor entitlement for VM virtual-machine-name cannot be modified because it is being managed by gwlm. To adjust the processor entitlement for this VM, use gwlm to change the policy associated with this VM. Create or modify the gwlm policy for this VM, using the Policy menu (available when using VM Manager). Integrity VM information is out of date, incomplete, or missing Information about certain Integrity VMs is missing from an Integrity Virtual Server Manager screen. NOTE: In general, Integrity Virtual Server Manager tabbed view screens are refreshed automatically every five minutes. An indicator on these visualization pages notifies you when the data was last refreshed. Integrity Virtual Server Manager screens that display configuration data are updated instantaneously when you use Integrity Virtual Server Manager to change the related configuration parameters. The screen might need refreshing. The VMs might not yet be started. The VM Provider or certain WBEM components might not be running on the VMs or on the VM host. Manually refresh a tabbed view screen and certain dialog screens (such as those for adding storage or network devices) by clicking on the Refresh Data link. Verify that the VM Provider is running by entering the following command on the VM or VM host: cimprovider -ls. The response HPVMProviderModule OK indicates the VM Provider is running. Deleted Integrity VM still appears in Matrix OE visualization Using Integrity Virtual Server Manager, a VM could be created and then deleted before the created VM has been identified by HP SIM. Therefore, the deleted VM might appear in Matrix OE visualization as though it had not been deleted. The appearance of the VM in Matrix OE visualization depends on when the deletion occurred during the HP SIM identification process. Use VM Manager from HP SMH. Wait a few minutes before deleting a VM that was just created. 80 Troubleshooting procedures

81 If the deleted VM does appear in Matrix OE visualization, delete it from the HP SIM All System view. From the HP SIM Systems and Event Collections list in the navigation area, select All Systems. Select Delete. gwlm troubleshooting Unable to build a single SRD Unable to build a single shared resource domain from the set of specified hosts as seen in one of the following error message: Unable to build a single shared resource domain from the set of specified hosts: myhosta.mydomain.com myhostb.mydomain.com The versions of the gwlm agents: <host1> : 6.3.0, <host2> : are different. Please correct and retry. The gwlm agent is not running. The gwlm agents on the managed nodes within an SRD are different versions or they are higher than the CMS version. Go to the managed node and start the gwlm agent ($/opt/gwlm/bin/gwlmagent) or start the gwlm agent from HP SIM. gwlm agents on all managed nodes within an SRD must be the same version and must be the same or lower than the CMS version. For example, a CMS that has A can manage an SRD that uses A agents or below. 1. View the versions: $/opt/gwlm/bin/gwlmstatus. 2. Install the same version of gwlmagent on all the managed nodes, making sure the version is the same or lower than the CMS version. You can use Insight managed system setup wizard to install the latest version on all the managed nodes in the SRD. Old host name not changed After changing a hostname or partition name, the old name still appears. HP SIM does not automatically delete the old name after a host or partition name change. Use ParMgr to change partition names. For the duration of the CMS configuration, change the following scheduled tasks: Disable daily system identification. Disable initial data collection. Change hardware status polling to every hour or less. Unable to determine the network address and/or hostname of the current host Unable to determine the network address and/or hostname of the current host, which indicates a misconfigured network and/or a host name resolution issue for this host. Matrix infrastructure orchestration troubleshooting 81

82 with DNS, /etc/resolv.conf, /etc/hosts, or /etc/nsswitch.conf. Verify that your name resolution service configuration is configured properly so that your system's hostname resolves with the nslookup command. 1. Add the IP address and domain name of the CMS in the /etc/hosts in the managed node. 2. Add the IP address of the managed node in the hosts file in the CMS. On HP-UX, the hosts file is located in /etc/hosts. On Windows, the hosts file is located in C:\Windows\ System32\drivers\etc\hosts. 3. Restart the managed node agent: # /opt/gwlm/bin/gwlmagent --restart. 4. Restart gwlmcmsd: For HP-UX CMS, #/opt/gwlm/bin/gwlmcmsd-stop # /opt/gwlm/bin/gwlmcmsd For Windows CMS, Restart Global Work Load Manager CMS service. Cannot create an SRD due to a firewall between managed nodes When attempting to deploy the SRD, gwlm times out and displays a blank screen with no error message. Each managed node will generate events similar to the following: gwlm Agent MySystem.MyDomain.com Information Unable to manage the following hosts: Associated Exception Unable to manage the following hosts: MySystem.MyDomain.com: The gwlm agent process on the host is not running -- start the agent and retry. A CMS can only manage hosts on the same LAN as the CMS itself when a CMS or managed node is configured with multiple network interfaces where application traffic is handled through one interface and management traffic through an alternate. Hosts on multiple LANs or separated by firewalls may not be able to communicate with each other due to LAN configuration or firewalls. Identify the management LAN host name on each of the affected systems in the gwlm property file. gwlm will then use that name instead of the OS host name. Set up a separate LAN (sometimes referred to as a management LAN or a backup LAN) that includes the CMS and all the hosts to be managed as shown in Figure 15 (page 83). When all managed hosts are on the same management LAN they can participate in a single SRD. 82 Troubleshooting procedures

83 Figure 15 Management LAN 1. For each host in the management LAN which is to be in the same SRD, perform the following steps: Edit the /etc/opt/gwlm/conf/gwlmagent.properties file to include the following property: com.hp.gwlm.security.virtuallocalhostname=hostnameonlan For example, with the host mgmta, its property would be: com.hp.gwlm.security.virtuallocalhostname=mgmta Restart gwlmagent on the host: # opt/gwlm/bin/gwlmagent restart 2. Add the CMS to the management LAN. If the primary hostname for the CMS is not the name it uses in the management LAN, perform the following steps: Edit the gwlmagent.properties file on the CMS to include the property: com.hp.gwlm.security.virtuallocalhostname=hostnameonlan. On HP-UX, the gwlmagent.properties file is in /etc/opt/gwlm/conf/. On Windows, it is in C:\Program Files\HP\Virtual Server Environment\ conf\. On Windows, the gwlmagent.properties file is in C:\Program Files\HP\ Virtual Server Environment\conf\ Restart HP SIM and gwlmcmsd. For HP-UX CMS, # /opt/gwlm/bin/gwlmcmsd - stop # /opt/gwlm/bin/gwlmcmsd For Windows CMS, Restart Global Work Load Manager CMS service. 3. On each host in the SRD (CMS and managed nodes), ping every other host in the SRD using the hostname you intend to have gwlm discover (its hostnameonlan) to verify communication is possible. 4. Discover (or re-discover) the hosts using thegwlm Manage Systems and Workloads wizard. Managed system unable to rejoin an existing SRD Matrix infrastructure orchestration troubleshooting 83

84 A managed node is unable to rejoin an SRD after recovering from a crash, after the CMS recovers from a crash, or after resolution of a network communication issue. gwlm maintains the concept of a cluster for the nodes in a SRD. When a SRD is disrupted, gwlm attempts to reform the SRD. In a cluster, one node is a master. If the master loses contact with the rest of the SRD, the rest of the SRD can continue as a partial cluster by unanimously agreeing on a new master. If a node that is not the master loses communication with the master, the resulting partial cluster continues operation without the lost node. The master simply omits the missing node until it becomes available again. Use the gwlmstatus command to monitor SRD availability. This command will show whether any hosts are unable to rejoin a node's SRD as well as whether hosts in the SRD are nonresponsive. Agent will not restart on reboot gwlm managed node agent will not restart automatically on reboot. GWLM_CMS_START is 0 in the gwlm configuration script /etc/rc.config.d/gwlmctl Change GWLM_CMS_START to 1 in /etc/rc.config.d/gwlmctl Run "gwlmagent --enable_start_on_boot" Insight Control troubleshooting Insight Control virtual machine management troubleshooting VM logical server displayed with incorrect association After a VM logical server has been created, it appears in the Logical Server perspective of Matrix OE visualization. This screen might display Active on unknown host for the location of the logical server, or the logical server might be displayed without any host. The Active on unknown host condition is normal for up to 5 minutes after a new VM logical server has been created. Wait at least 5 minutes before attempting corrective action. When a VM logical server is deactivated, the VM is unregistered from the VM host. The VM is then displayed without an association in Matrix OE visualization perspectives. This behavior is expected. The VMware WBEM provider might have returned invalid or incomplete data due to an error in the WBEM provider. Other environmental changes: Name resolution of the VM host might no longer be configured correctly. Credentials for the VM host might have been modified and are no longer correct in HP SIM. Network connectivity between the CMS and the VM host may not be available. 84 Troubleshooting procedures

85 For a newly-created VM, navigate to the VM host System page, and verify that the newly-created VM appears. If the VM appears, select Options Identify Systems. This operation adds the VM to HP SIM and update the Logical Server view to display the VM name as the location. Check the VM host System Page in HP SIM to determine if HP SIM has the correct host-guest associations: 1. Locate the ESX host System Page by searching for it or by selecting it in All Systems Virtual Machine Hosts ESX Hosts nodename. 2. Expand both the Virtual Machines section and the Associations section. If the VM guest appears in the Virtual Machines section but not in the Associations section, the VMware WBEM provider might have returned incorrect or incomplete data. 3. Restart the VMware CIM service, and identify the ESX host again: a. Log on to ESX as root. b. Enter the following command: /etc/init.d/sfcbd-watchdog restart c. Manually re-identify the ESX host using Options Identify Systems. HP and VMware are aware of this issue and working on a resolution. Check the environment changes: Verify name resolution of the VM host. Verify the credentials in HP SIM are correct. Verify network connectivity between the ESX host and the CMS. VM guests appear without system addresses in HP SIM HP SIM shows VM guests with no system addresses (IP addresses) and/or Insight Control virtual machine management displays two or more VM guests with the same IP address or some guests with incorrect IP addresses. This issue may occur if the VM guest was first detected by HP SIM prior to customization or while powered off. In this situation, HP SIM is unable to perform a complete discovery of the VM guest. Verify that the guests are powered on and fully operational. Verify that the guests have valid IP addresses. Perform a discovery of the guests in HP SIM: 1. Select Options Discovery 2. Do one of the following: a. If you already have a discovery task for the VM guests, verify their credentials and execute that discovery task. b. If you do not have a discovery task for the VM guests, create a new discovery task for the VM guests with valid credentials for the guest, and execute that task. Using HP Virtual Connect Flex-10 with VMware ESX Insight Control troubleshooting 85

86 One physical link from the Flex-10 fabric appears as four Flex-10 NICs (physical NICs) to the VMkernel. Because there is only one physical link, states of all the Flex-10 NICs change together so that the link state of individual Flex-10 NICs is reported correctly. As a result, ESX might not be able to detect failures when a corresponding external uplink is down, even if the SmartLink is turned on and the Ethernet Connection Mode is connected to disable failover. Before attempting the following workarounds, use the VCEM user interface to place the VC Domain that contains the blade which will be running ESX into VC Domain Maintenance. To invoke the VCM user interface, click Make changes via VC Manager. From the VCEM user interface, perform the following steps: 1. Click the VC Domains tab. 2. Select the VC Domain. 3. Click VC Domain Maintenance. This performs a partial unlock of the VCM user interface and allows you to perform the following workaround steps. Workaround 1: Turn off SmartLink and then configure Ethernet Connection Mode to enable failover. This allows VC to automatically fail over to another route in the event of external uplink failure without affecting the individual Flex-NIC connected to ESX. Workaround 2: Enable beacon probing in the vswitch or port group with multiple Flex-NICs teamed. Verify the following: SmartLink is turned on. Ethernet Connection Mode is configured to disable failover. After the workaround steps are completed, go back to the VCEM user interface and click Complete VC Domain Maintenance. This will bring the changes performed above back into VCEM and allow VCEM to propagate these to the other VCDs in the VC Domain Group. Detecting and repairing database connectivity problems s Any of the following: An error communicating with Insight Control virtual machine manager from infrastructure orchestration, recovery management and Matrix OE occurs. Deployments fail in infrastructure orchestration due to connectivity problems with Insight Control virtual machine manager. Logical servers recovery failures occur in recovery management due to connectivity problems with Insight Control virtual machine management. Performance data meter in Matrix OE visualization is unavailable. Unable to communicate with Insight Control virtual machine manager message in the System page of the VM host/guest. Insight Control virtual machine manager service no longer running. This can be verified in Windows Services on the CMS. The following error message displays in the Insight Control virtual machine manager service log file: Error connecting to Insight Control virtual machine management database instance. Ensure the database instance is up and running. Initiating Insight Control virtual machine management service shutdown. 86 Troubleshooting procedures

87 Database connectivity is not established between the CMS and the database server, due to one of the following conditions: Network failures Invalid database credentials Database password lockout Change of database password and not syncing up the password in HP SIM Duplicate IP address in the network 1. Check whether the database server or service is up and running. If the database is up and running, try restarting Insight Control virtual machine manager service from the Windows Services. If the Insight Control virtual machine manager service shuts down again continue to the next step. 2. Check the database connectivity using the database user ID and password given in the HP SIM through a database client software. If the connection is successful, sync up the database user ID and password in HP SIM. Restart HP SIM and Insight Control virtual machine manager services. 3. If database connectivity cannot be established with the credentials as given in the HP SIM, check whether the database user id has been locked out. Take the appropriate action to unlock the user and try restarting the Insight Manager and Insight Control virtual machine manager services. 4. Ensure there are no duplicate IP Address of the database server in the network. Operations related to Hyper-V guests The operations from Insight Management components on the Microsoft Hyper-V servers could result in temporary files being created at C:\ on the Hyper-V host. These files are not required after the operation completes and can be removed manually. These operations include VM copy / move / template / back-up / linked clone. Manually remove these files. Linked clone provisioning fails with concurrency set to 10 on Hyper-V Due to a limitation with Microsoft, linked clone provisioning on Hyper-V does not support concurrency set to 10. Suggested action For Hyper-V, only use linked clone provisioning for concurrency set to 5. To change the concurrency: 1. Access the hpio.properties file located at C:\Program Files\HP\Matrix infrastructure orchestration\conf\hpio.properties. 2. Set the value for hyperv.max.concurrent.requests = Save the file. 4. Restart the HPMatrix infrastructure orchestration service. 5. Revert back to the default of 10 for normal provisioning. Hyper-V R2 hosts may hang while undergoing rapid VM creation/deletion operation Insight Control troubleshooting 87

88 Microsoft Hyper-V host servers might hang in situations where many simultaneous VM operations (such as creating, deleting, or activating VMs) are taking place. The source server is running without Windows Server 2003 R2 Service Pack 2 or later. The source server does not have the Microsoft Storport Storage Driver from KB article installed. The migration is performed to a destination server connected to a SAN LUN The SAN fabric zone has multiple paths between the destination server and the SAN storage. Microsoft Corporation has supplied a patch that corrects the problem. To download the patch and install it on your Hyper-V hosts, see the following website: If the IP address of the host changes, Insight Control virtual machine management cannot manage the host If the IP address of a hypervisor host is changed, Insight Control virtual machine management will not be able to manage the host. HP recommends that you unregister the hypervisor host before changing the IP address. After the IP address is changed, identify the host in HP SIM and register with virtual machine management. If the IP address is changed without unregistering the host, unregister the host with the old IP address using the vmcli tool and register with the new IP address. Cannot register vcenter in Insight Control virtual machine management Insight Control virtual machine management can discover a vcenter server, but it cannot find the serial number or UUID. An attempt was made to discover the vcenter server using a non-administrator account. WMI service is not running on the vcenter server. WMI mapper proxy is not configured properly on the CMS. If vcenter is installed on a VM, then install WMI Mapper on the vcenter server. Use WMI to re-identify the vcenter server with administrator credentials. If vcenter is installed on the physical server, use WMI to re-identify the vcenter server with administrator credentials in HP SIM. Configure WMI mapper proxy properly on the CMS. In HP SIM, navigate to Options Protocol Settings WMI Mapper Proxy. VM names do not display correctly in the Insight Control virtual machine management user interface VM names may not be displayed correctly in the Insight Control virtual machine management user interface if the VM name contains special characters. HP recommends avoiding special characters in VM names. Rename such VM names appropriately. 88 Troubleshooting procedures

89 Host/Guest control buttons disabled If the VM names have non-ansi characters, the control buttons in the Host/Guest system page are disabled. From the HP SIM user interface, run Tools Virtual Machine, then select the appropriate control operations. Hyper-V provisioning fails on Hyper-V 2012 and Hyper-V 2012 R2 with timed out errors Hyper-V provisioning fails on Hyper-V 2012 and Hyper-V 2012 R2 with timed out errors and, sometimes, fails to register the deployed VM. The slow performance of the Hyper-V host delays the retrieval of the required data through WMI query. To solve this issue: 1. Increase the CIM Client timed out period to more than (default value) by adding the parameter HyperVQueryTimeout in the <vmm-install>\bin\hpvmm.conf and restart the VMM service. 2. Set the value of servertimeout to more than (default value) in cimserver_planned.conf file of <MAPPER_INSTALLED_DIR>, and then restart the Mapper. Insight Control server deployment troubleshooting For more troubleshooting information, see the Insight Control server deployment Knowledge Base at the following website: PXE boot to Insight Control server deployment fails The ilo console displays the following message: Attempting Boot From Hard Drive, NTLDR is missing, Press Ctrl+Alt+Del to restart. The boot disk was not erased. SAN zoning or presentation errors. Erase the boot disk and attempt the PXE boot again. To erase the boot disk: 1. Power on the server blade. (Cycle power if the blade does not respond.) 2. Manually delete the server jobs from the deployment console. 3. Schedule the Erase ProLiant Hardware job from the deployment console. This job can be found in Server Deployment Toolbox Array Hardware 4. Power on the server blade and allow it to PXE boot and execute the Erase job. If the server does not automatically PXE boot, cycle power to the server blade and, using the ilo console, press F12 to PXE boot the server. Press F8 to force a PXE boot and to execute the Erase job. 5. Ensure the server blade is powered off after the Erase job completes; manually power it off if necessary. 6. Manually delete the server blade and all its jobs from the deployment console. 7. Deactivate any associated Matrix OE logical server (if it is active) by selecting Tools Logical Servers Deactivate from the Matrix Operating Environment menu. The logical server typically Insight Control troubleshooting 89

90 has a name beginning with the prefix Clean-me. In the Deactivate window, you might be prompted to enter YES to proceed with this operation. 8. After the associated Matrix OE logical server deactivation is complete, delete the Matrix OE logical server from the Matrix OE visualization menu by selecting Delete Delete Logical Server. In the Delete Logical Server screen, you might be prompted to enter YES to proceed with this operation. See Do not unmanage a logical server that has been previously activated (page 63). 9. Use VCEM to verify that the VC profile associated with the server blade has been deleted. If the profile exists, unassign and delete the VC profile. 10. Move the server blade out of the infrastructure orchestration maintenance pool, and into its original pool. a. In the infrastructure orchestration console (accessible from HP SIM), select the original server blade pool, and click Modify. b. Move the failed server from the infrastructure orchestration maintenance pool back to your server blade pool, and then save the pool. PXE error: No boot filename received PXE-E53: No boot filename received. The client received at least one valid DHCP/BOOTP offer, but does not have a boot filename to download. and suggested actions The DHCP server and PXE server were previously located on the same server, but one of them has been moved to a different server. This might result in an incorrect PXE server configuration. To resolve this issue, reinstall the PXE server component of the Altiris Deployment Solution. The DHCP relay agent, either a Proxy DHCP server or a switch configured with helper addresses, is not configured correctly. For example, if DHCP and PXE are on separate servers, the DHCP relay agent needs to have both addresses in its configuration. To resolve this issue, correct the DHCP relay agent configuration. If the Microsoft DHCP service is installed on the PXE server, but is disabled or unconfigured, Altiris PXE Setup configures PXE to work with the local DHCP service (even if the DHCP service is disabled). This causes the PXE server to not respond to PXE clients that get a DHCP address from DHCP services running elsewhere on the network. To resolve this issue, remove Microsoft DHCP services from the PXE server and reinstall the PXE server component of the Altiris Deployment Solution. An Insight Control server deployment installation on Windows 2008 with the DHCP service installed locally does not set the 060 DHCP option. To resolve this, modify the following PXE configuration option if DHCP service is required to be installed locally on a Windows 2008 Deployment Server: 1. In the PXE Configuration Utility, click the PXE Server tab. 2. Select the DHCP Server Discovery setting, and change the setting to: No DHCP server installed on PXE server (Use DHCP port). A future release of Altiris Deployment Solution will correct this issue. 90 Troubleshooting procedures

91 Firewall and port configuration problems. For example, if the Symantec client endpoint protection on the Insight Control server deployment is enabled, many required ports are blocked. To resolve this issue, be sure the following ports are allowed through the firewall when utilizing a firewall locally or on the network. Service Altiris PXE Server Service MTFTP TFTP Port number UDP 67, 68: Reply to DHCP and PXE requests UDP 69: MTFTP file transfer UDP 1758: Multicast Client UDP 1759: Multicast Server UDP 4011: Required if the PXE server is installed on the DHCP server Altiris PXE Manager Service Altiris express Server Service Altiris Deployment Server Data Manager Service RapiDeploy Imaging Engine Deployment Server Web Console Client/Server File Transfer Remote Control via Deployment Server Console Deployment Agents TCP 405, 406 TCP 402: Client\Server communication. This port can be edited in the PXE Configuration Utility and at the DS tab. TCP 505. This port can be edited in the PXE Configuration Utility and at the DS tab. UDP 401: Used by the RDeploy imaging engine when multicasting an image. This port can be edited in the Deployment Server Console and at Tools Options Global. TCP 80. The port is configured by IIS. TCP?: This is a configurable option that can be considered when configuring your firewall settings. This port can be edited in the Deployment Server Console and at Tools>Options>Global. TCP 5001, 5002: Required for Windows agent remote control functionality. This port can be edited in the Deployment Server Console and at Tools Options Global. TCP 402: Client\Server communication UDP 402: Required if multicast communication is enabled NOTE: By default, RDeploy multicasts an image deployment when four or more targets are deployed simultaneously. To edit RDeploy imaging multicasting settings, select Start All Programs Altiris Deployment Solution+Configuration+Options+Disk Imaging. Special considerations for Insight Control server deployment software installed on Windows Server 2008 Insight Control troubleshooting 91

92 Windows Server 2008 has multiple firewall profiles that need to be configured depending on the Insight Control server deployment network environment. When updating the Windows Server 2008 firewall configuration, make sure all these profiles are taken into consideration. For more Insight Control server deployment related troubleshooting information, see the Insight Control server deployment Knowledge Base: Install MPIO EVA DSM or Install MPIO XP DSM job hangs When executing the Install MPIO EVA DSM or Install MPIO XP DSM on other storage devices such as 3PAR StoreServ, the job hangs. These jobs are not intended to use on other storage devices. Use the provided generic (non-device specific) Microsoft MPIO DSM job. The job provided will install the Microsoft MPIO feature. Then manually configure the MPIO. See Microsoft Multipath I/ O Step-by-Step Guide at Server deployment failure in a multipath I/O environment Server fails during boot of WinPE or immediately after OS installation in a multipath I/O environment. WinPE has not been configured to support multipath I/O. The target OS image does not have multipath, I/O-capable drivers loaded. Set the WinPE SAN policy. Verify your SAN is properly configured and zoned for multipath I/O. Install the proper device-specific module driver package for your storage solution into your deployment images. For details, see the article Multipath SAN Deployment Support on the Insight Control server deployment Knowledge Base: html Using multipath I/O with QLogic QMH2462 4Gbps Fibre Channel HBA and Intel-based HP G6 BLc servers The blue screen parity error appears as shown in Figure 16 (page 92): Figure 16 Parity error 92 Troubleshooting procedures

93 There is a known issue in QLogic firmware or drive that QLogic is working to resolve. Using mutipath I/O with QLogic QMH2462 4Gbps Fibre Channel HBA and Intel-based HP G6 BLc servers does not work until you have the QLogic fix. Either replace the 4Gbps HBAs or disable MPIO. Using multipath I/O with Emulex Fibre Channel HBA and HP servers Using multipath I/O with Emulex Fibre Channel HBA does not work due to a known issue in Emulex firmware or driver. At the time of this document's publication, HP and Emulex are working to resolve this issue. To work around the problem, perform the OS installation with only single path I/O enabled by omitting the selection of I/O redundancy in the Matrix OE storage pool entry. After the OS installation completes, you can then enable the redundancy in the Matrix OE storage pool entry, and the OS will have multipath I/O. Insight Control troubleshooting 93

94 Servers configured with system ROM version do not install the operating system to the SAN Insight Control server deployment does not install the operating system to the SAN on a ProLiant BL465c G7 or ProLiant BL685c G7 server blade configured with System ROM version and a SAN. When attempting to install the operating system to the SAN, it installs to the local hard drive. The system ROM does not allow the Insight Control server deployment software to disable the embedded Smart Array Controller during the operating system install to the SAN controller. This issue is referenced in Customer Advisory c at the following website: h20000.www2.hp.com/bizsupport/techsupport/document.jsp?lang=en&cc=us& objectid=c A future system ROM release will allow the operating system to install on the SAN. Perform the following steps to temporarily disable the embedded Smart Array Controller and allow the operating system to install on the SAN: 1. Select the F9 key during POST. 2. Select the PCI Enable/Disable Menu. 3. Select the HP Smart Array P410i Controller. 4. Select Disable. 5. Select the F10 key to Save and Exit settings. 6. Reboot the server. After the operating system installs on the SAN, the Smart Array controller can be re-enabled by performing the steps above and select Enable instead of Disable. Server Automation or Insight Control server provisioning troubleshooting Server Automation or Insight Control server provisioning fails due to boot order changes The boot order gets reset any time the boot device count changes. For example, adding or removing a mezzanine card or changing the personality from iscsi (default) to FCoE for Converged Network Adapters. Suggested action Perform the following steps any time you are adding a blade, adding or removing a mezzanine card, or changing the personality in a Converged Network Adapter. 1. Boot the blade and press F9 to enter the ROM-Based Setup Utility. 2. Navigate to the NIC personality. a. Press Enter. b. Select NIC Personality Options and press Enter. c. Select the NIC and press Enter. 3. Change from an iscsi personality to an FCoE personality. 4. Optionally set the IPL boot order. a. From the BIOS menu, select Standard Boot Order. b. Move the NIC card before the Hard Drive C: step. 5. Optionally, save the setting as user defined default setting (so they get reloaded if a reset to default settings is requested). a. From the BIOS menu, select System Default Options. 94 Troubleshooting procedures

95 b. Select Save User Defaults. 6. Press F10 to save the changes and exit. 7. Boot the server. Provisioning of Matrix service fails in customization Provisioning of Matrix service fails in customization Possible cause Provisioning of Matrix services can fail in customization if Insight Control server provisioning or SA deployment is attempting to join the server to an Active Directory or performing NIC teaming during the OS installation. Doing so will cause the Matrix post-install customization step to fail with an error stating Error customizing Logical Server <Logical server name> realized by <target UUID>. Suggested action Matrix OE environments can have the newly provisioned server join the domain or perform NIC teaming by using enahced server customization. For additional information on this, see the HP Matrix Operating Environment Infrastructure Orchestration User Guide. NOTE: HP does not recommend using an OO workflow to join a newly provisioned server to the domain as it requires adding all computers as trusted hosts. Failure when creating or deactivating a High Availability Hyper-V virtual machine with RDM/pass-through cluster disks Service provisioning or logical server VM creation or reactivation fails with a job message similar to The operation has failed. (The start function on virtual machine vsels_name [VM info and XML path] failed.). Possible cause Matrix supports High Availability (HA) Hyper-V VMs with RDM/pass-through disks, with various constraints from Hyper-V. Violating these constraints can result in failures. Only HA VMs can have pass-through cluster disks attached. In addition, HA VMs can only have such disks attached. The cluster disk must be in maintenance mode prior to provisioning and activation. Suggested action 1. If the error occurs when provisioning an HA VM using one or more pass-through cluster disks, turn on maintenance mode and repeat the provisioning operation (submitting a service create Server Automation or Insight Control server provisioning troubleshooting 95

96 or creating a logical server). Turn on maintenance mode for the cluster disks via the Microsoft Failover Cluster Manager Turn off maintenance mode once the VM is provisioned and the OS is deployed. 2. If the error occurs when reactivating an HA VM using one or more pass-through cluster disks, turn on maintenance mode and repeat the reactivation. Turn on maintenance mode via the Microsoft Failover Cluster Manager. Turn Off Maintenance Mode once the VM is reactivated. (See screenshots above) Storage troubleshooting Storage Provisioning Manager issues For additional SPM troubleshooting tips, see the HP Storage Provisioning Manager (SPM) User Guide at Communication errors occurred between SPM and LSM Communication errors between SPM and LSM have occurred. Authentication errors are shown in the SPM logs. Possible cause 1 The MatrixService1 user may not be listed in the Matrix Security Group Role in the Users and Groups tab. The user was removed accidently or was not set up initially. 96 Troubleshooting procedures

97 Suggested action 1 1. In SPM, select Roles in the Administration section. 2. Select the Matrix Security Group. 3. Select the Users and Groups tab. 4. Verify that the MatrixService1 user is listed. If not, add the MatrixService1 user by clicking Add and entering the user information. Possible cause 2 The certificates may have gotten out of sync between SPM and LSM. This can occur if the SPM certificate is regenerated on the CMS. Suggested action 2 1. Change directory to <Install Path>/Storage Provisioning Manager/bin 2. Run: keytool -printcert -file "d:\program Files\HP\Systems Insight Manager\bin\server_cert.pem" 3. Record the SHA1 key, removing the ":" characters. 4. Run: spmadmin /lcu 5. Record the certificate. 6. Compare the values in step 3 (the SHA1 key) to step 5 (the certificate). 7. If the values are different, this indicates a condition where a new certificate is generated, hostname is changed or IP address is changed. Please contact HP Technology Services. Cannot import 3PAR StoreServ Storage Systems Importing an 3PAR StoreServ System fails. Possible cause 1 The 3PAR InServ floating IP is not accessible from the CMS. Suggested action 1 Ping the 3PAR InServ Node address. Verify that the InServ Node IP Address is accessible from the CMS. Possible cause 2 SMI-S is not active on the 3PAR StoreServ. Suggested action 2 Verify SMI-S is running on the 3PAR StoreServ by logging onto the node floating IP: 1. At the command prompt, type showcim. 2. Verify that SMI-S is enabled and running. If it is not, run startcim. Possible cause 3 Inaccurate credentials (user does not exist or password does not match) has caused CIMOM not to respond. Suggested action 3 Wait 30 minutes until the request times out or restart SPM. Storage pool issues Matrix infrastructure orchestration service fails to auto-generate multi-path storage Storage troubleshooting 97

98 In some configurations, Matrix infrastructure orchestration fails to auto-generate storage that is defined as requiring redundancy. A fabric pairing cannot be found to return volumes from SPM. Some configurations that cause auto-generating storage issues include: The SAN array s ports are all connected to a single fabric. Matrix requires that the array be connected to two or more fabrics for redundancy. The redundant SAN fabrics are not connected to matching bays within the enclosure. If there are more than 2 fabrics, Matrix automatically pairs fabrics connected to Bays 1&2, or 3&4, for example, not 1&4. The redundant SAN fabrics are connected to the same connection bay, but different uplink ports. Matrix only tries to match fabrics from different connection bays for multi-pathing. Suggested action Create manual storage pool entries (click Add Entry on the Manage Storage Pool screen). Specify the fabrics needed for your configuration and then use this Storage Pool Entry storage for the Matrix infrastructure orchestration service creation. No matching candidates found for this volume After clicking Show Candidates in the Storage Pool Entry screen (by clicking the Add Entry button on the Manage Storage Pool screen from the Matrix OE visualization Modify Menu+Logical Server+Storage Pools), one of the following error messages displays: No matching candidates were found for this volume. Verify that the properties (RAID Level, Size, Operating System, and Fabric) are correctly specified, and ensure that the Storage Provisioning Manager is correctly configured. Matching Storage Candidate Services not found for the volume. Connection refused. The connection refused message indicates Matrix OE is not able to communicate with SPM properly. SPM is an optional tool for managing storage provisioning with the Matrix OE software environment. Other possible causes include: Specified properties for the volume do not match any available candidates in SPM s storage catalog. Improper configuration of SPM. The storage administrator has not yet created the necessary storage and made it available through SPM s storage catalog and on-demand provisioning is not enabled or has failed. The SPM resources and template are not owned by the Matrix Security Group. If suitable storage is not within the SPM catalog, the storage administrator will need to create or zone appropriate storage and make it available in the catalog or enable on-demand provisioning to allow Matrix OE/SPM to automatically create the storage and handle zoning (in a Brocade environment). Unfulfilled requests are easily visible and can be manually fulfilled (or subsequent attempts to find matches can succeed). If there is an error communicating with SPM, verify the SPM service is running. If there is no connection error and you believe the storage catalog contains suitable storage which is not being returned as matching candidates and on-demand provisioning is enabled, consider the following: Check the requirements specified in the logical server storage pool entry (size, redundancy, fabric, tags, etc.). 98 Troubleshooting procedures

99 Verify that the Proxy WWN (connected to WWN) associated with the selected Fabric Name is correct and known to the Brocade switch, or correctly defined within an unmanaged network. Verify the tag values are an exact match between the logical server storage pool entry and SPM. Select the Masked check box on the storage pool entry screen (you may deselect it in the future for data volumes prior to OS deployment, or infrastructure orchestration may do so for you, but it should be selected initially to find candidate volumes to bind). Determine if the administrator has left the host mode as unspecified in the storage pool entry. If so, Matrix OE will match for storage which can support all possible host modes (including Windows 2008). HP recommends specifying a host mode (and if the storage pool entry is using the SPM storage catalog, the storage administrator may have enabled the ability to adjust host mode as necessary). Alternatively, ensure the storage volumes in SPM support all possible host modes (including Windows 2008). Verify the storage volumes support Windows 2008 host mode if the host mode has been set to Windows Check the checkboxes on the volume Presentations tab in SPM and ensure the appropriate OS host mode, WWN values, or the catalog entry has permissions for the presentation to be modified. Check to see if another storage pool entry may have bound to the storage volumes you intended to use. Either obtain additional storage or unbind that storage if it was mistakenly bound. Ensure the catalog entries have suitable permissions (allowing change of host mode and LUN masking) if using infrastructure orchestration. Verify the SPM service is running. Access the SPM Fulfillment screen for additional details (including the ability to browse all unbound volumes and see an evaluation of how they match to the specific requirements of the storage request). The administrator can force a binding within SPM even if all requirements are not met (creating a non-conformant fulfillment). Other storage issues Port conflict when HP P6000 Command View Software runs on Matrix OE CMS HP SIM cannot successfully discover storage devices. The HP P6000 Command View Software SMI-S CIMOM is running on a port other than Determine which ports were used for storage management. Look at the cxws.https.port value in the cxws.properites file: For Microsoft Windows (64 bit version) in C:\Program Files(x86)\ Hewlett-Packard\SMI-S\CXWSCIMOM\config\ For Microsoft Windows (32 bit version) in C:\Program Files\Hewlett-Packard\ SMI-S\CXWSCIMOM\config\ Make a note of this port value. 2. Edit the following file: C:\Program Files\HP\Systems Insight Manager\config\ identification\wbemportlist.xml. 3. Type the following XML phrase inside the <wbemportlist> near the bottom (just above </wbemportlist>). Do not modify any existing XML phrases. Storage troubleshooting 99

100 <port id="60989" protocol="https"> <interopnamespacelist> <interopnamespace name="interop"/> <interopnamespace name="root/interop"/> <interopnamespace name="root/cimv2"/> </interopnamespacelist> </port> 4. Replace the port ID value with the https port you found in cxws.properties. The result should look similar to the following: 5. Save the file and restart HP SIM. 6. Discover the system that is running SMI-S Provider. NPIV/vport storage not available with FlexFabric Converged Network Adapters or FC mezzanine Host Bus Adapters vport/npiv storage is not available in the operating system with NPIV configuration using FlexFabric Converged Network Adapters or FC mezzanine HBAs. The current Emulex driver for Windows does not permit the physical initiator port and a vport to access LUNs over the same target WWPN. The driver for Linux does not have the same limitation as Windows, but it is possible the NPIV mode for the Emulex driver is not enabled. 100 Troubleshooting procedures