SolarWinds. Server & Application Monitor. Troubleshooting Guide. Version 6.3

Transcription

1 SolarWinds Server & Application Monitor Version 6.3 Troubleshooting Guide Last Updated: November 7, 2016 Retrieve the latest version from:

2 2016 SolarWinds Worldwide, LLC. All rights reserved. This document may not be reproduced by any means nor modified, decompiled, disassembled, published or distributed, in whole or in part, or translated to any electronic medium or other means without the prior written consent of SolarWinds. All right, title, and interest in and to the software and documentation are and shall remain the exclusive property of SolarWinds and its respective licensors. SOLARWINDS DISCLAIMS ALL WARRANTIES, CONDITIONS OR OTHER TERMS, EXPRESS OR IMPLIED, STATUTORY OR OTHERWISE, ON SOFTWARE AND DOCUMENTATION FURNISHED HEREUNDER INCLUDING WITHOUT LIMITATION THE WARRANTIES OF DESIGN, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT. IN NO EVENT SHALL SOLARWINDS, ITS SUPPLIERS, NOR ITS LICENSORS BE LIABLE FOR ANY DAMAGES, WHETHER ARISING IN TORT, CONTRACT OR ANY OTHER LEGAL THEORY EVEN IF SOLARWINDS HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. The SolarWinds and other SolarWinds marks, identified on the SolarWinds website, as updated from SolarWinds from time to time and incorporated herein, are registered with the U.S. Patent and Trademark Office and may be registered or pending registration in other countries. All other SolarWinds trademarks may be common law marks or registered or pending registration in the United States or in other countries. All other trademarks or registered trademarks contained and/or mentioned herein are used for identification purposes only and may be trademarks or registered trademarks of their respective companies. Server & Application Monitor Administrator Guide, Version 6.3, 10/19/2016 2

3 Table of Contents Troubleshooting SAM issues and errors 7 Troubleshooting performance 8 Troubleshoot slow performance on Windows Server Optimize SolarWinds server performance with Temporary Directories 9 Move the SQL server temporary directory 9 Redefine Windows system temporary directories 9 Troubleshooting WMI 11 SAM WMI monitors show a status of Unknown 13 Test WMI services using WBEMTest 13 Manually reset the WMI counters 15 Test remote WMI connectivity 16 Verify administrator credentials 17 Enable Remote Procedure Call (RPC) 17 Configure Distributed Component Object Model (DCOM) and User Account Control (UAC) 18 Enable DCOM permissions for your SAM credentials 18 Enable Account Privileges in WMI 19 Disable Remote User Account Control for workgroups 20 Add a Windows firewall exception for remote WMI connections 20 Determine if an additional polling engine is needed 21 Verify SAM component configuration 21 Service reporting Invalid Class 22 Verify the polling type 22 Repair the WMI Repository 23 Troubleshoot hardware health 24 Hardware prerequisite checklist 25 Hardware troubleshooting flowchart 26 3

4 SNMP monitored node not returning hardware information 28 Verify the node was successfully added using SNMP 28 Check OIDs using SNMPWalk.exe 29 Troubleshoot a WMI node 30 Troubleshoot a VMWare node 32 Troubleshoot ESX hardware monitoring 34 Enable to connect to a remote server 34 Enable to establish sessions 34 Troubleshoot AppInsight for Exchange 35 Troubleshooting Exchange Performance Counters 35 Unload Performance Counters 38 Reload Performance Counters 38 Troubleshoot error codes in AppInsight for Exchange 39 Deployment service executable not found 39 Failed to start Windows Remote Management 39 Unable to access remote admin 40 Configurator exe not found on the Orion server 40 Access denied 40 Configurator exe has an invalid signature 41 Network path not found 41 PowerShell 2.0 not detected on the Exchange server 42 Account must be an Active Directory account with local admin privileges 43 Account must be an Active Directory account with organization wide Exchange access 43 Error occurred during Exchange Server Configuration 44 PowerShell web site testing failed: Connecting to remote server 45 Troubleshooting AppInsight for IIS 46 Access is denied when configuring AppInsight for IIS 47 Change between 32-bit and 64-bit polling 48 Credentials test failed 49 4

5 Polling fails due to a missing certificate 49 IIS Polling Failed 50 Node unreachable 50 IIS version failed 50 WinRM testing failed 50 Error code 1367 incorrect credentials 51 Error code 1726 server configuration 51 Error code deployment service exe not found 52 Error code configuration exe not found 52 Error code access denied 53 Error code configuration exe has an invalid signature 53 Error code network path not found 53 Error code configuration file not found 54 Error code unable to access remote admin share 55 Error code HTTPS listener exists on port Error code unable to create self-signed certificate 58 Error code unable to create wsman listener 59 Error code failed to start Windows Remote Management 59 Error Code unknown error IIS server configuration 59 Error code PowerShell 2.0 or higher required 59 Other errors 60 PowerShell 2.0 not installed 60 PowerShell WebAdministration snapin not installed 61 Polling IIS unsuccessful 61 Event IDs 62 Troubleshoot permissions 71 Non-Domain Account 71 Add Local Administrative privileges to Active Directory Account 71 Exchange Access 73 5

6 Mailbox Search Access 73 6

7 Troubleshooting SAM issues and errors This guide walks through errors, issues, and troubleshooting for SAM. You can search for errors or general issues, using the resolution to update configurations and clear issues. Alerts trigger and display in the Orion Web Console as issues trigger according to thresholds, conditions, and enabled options. After resolving issues or modifying customizations, monitor data after the next poll to verify updates. 7

8 Troubleshooting performance Troubleshooting performance To resolve issues with server performance, review the following: Troubleshoot slow performance on Windows Server 2008 Optimize SolarWinds server performance with Temporary Directories Troubleshoot slow performance on Windows Server 2008 SAM supports Windows Server 2008 R2 only. Previous versions of Windows Server 2008 are no longer supported. If SolarWinds is installed on Windows Server 2008 or Windows Vista and there are any devices on your network, between your SolarWinds server and your database server, that do not support RFC 1323, TCP Extensions for High Performance. The TCP window size auto-tuning feature of Windows Server 2008 and Windows Vista may prevent your SolarWinds server from successfully connecting with your Orion database. This TCP auto-tuning feature is intended as a network-sensitive restriction, applied by the receiver-your SolarWinds server-on the amount of data it is allowed or able to receive. If any devices along the network path between your SolarWinds server and your Orion database do not support the TCP window scaling detailed in RFC 1323, the allowed size of the TCP window in packets sent to your SolarWinds server may not match the TCP window size reported by packets sent from your SolarWinds server. This mismatch may lead to failed connections between your SolarWinds server and your Orion database. The following procedure disables this TCP auto-tuning feature, resetting the TCP receive window to 64kB. You should disable TCP Auto-tuning: 1. Click Start > All Programs > Accessories. 2. Right-click Command Prompt, and then click Run as Administrator. 8

9 Optimize SolarWinds server performance with Temporary Directories 3. If you are prompted by User Account Control, click Continue to open the elevated command prompt. In some cases, having User Account Control (UAC) enabled on Windows Server 2008 and Windows Vista (evaluation-only) can lead to installation errors. For more information about disabling UAC, see this KB article Disabling User Account Control (UAC). 4. At the prompt, enter the following: netsh interface tcp set global autotuninglevel=disabled 5. Close the command prompt window, and then restart your SolarWinds server. Optimize SolarWinds server performance with Temporary Directories The following sections provide procedures for moving Windows and SQL Server temporary directories to optimize SolarWinds server performance and resources. Move the SQL server temporary directory The SQL Server temporary directory, tempdb, where temporary database objects generated during table creation and sorting are stored, is typically created in the same location on your Orion database server as the master, model, and msdb databases. Moving the tempdb database to a physical drive separate from your Orion database can significantly improve overall system performance. For more information about moving the SQL Server 2008 temporary directory, tempdb, see Moving System Databases - Example A: Moving the tempdb Database. Redefine Windows system temporary directories Following established Windows standards, the SolarWinds installer may use Windows User and System TEMP and TMP variable directories as temporary scratch spaces for file expansion and execution. If you do not have the required scratch storage space available in the default User or System TEMP and TMP directories, use the following procedure to redefine your default locations. Regardless of the directory you installed SolarWinds product, some common files may be installed where the operating system of your SolarWinds server are located. 9

10 Troubleshooting performance 1. Log on to your SolarWinds server as a user with administrative rights. 2. Locate your environment variables. 3. Select the variable name representing the directory you want to redefine, and then provide a new path as the Variable value for the selected temporary directory variable. 10

11 Troubleshooting WMI Troubleshooting WMI This section provides troubleshooting options for the Windows WMI polling method. SAM WMI monitors show a status of Unknown Test WMI services using WBEMTest Manually reset the WMI counters Test remote WMI connectivity Verify administrator credentials Enable Remote Procedure Call (RPC) Configure Distributed Component Object Model (DCOM) and User Account Control (UAC) Enable DCOM permissions for your SAM credentials Enable Account Privileges in WMI Disable Remote User Account Control for workgroups Add a Windows firewall exception for remote WMI connections Determine if an additional polling engine is needed Verify SAM component configuration Service reporting Invalid Class The following Microsoft article describes how to set permissions in a workgroup after an upgrade from Microsoft Windows 2000 Professional to Microsoft Windows XP Professional. Use the flowchart to help determine the issues you are encountering and resolution steps. 11

12 Troubleshooting WMI 12

13 SAM WMI monitors show a status of Unknown SAM WMI monitors show a status of Unknown To monitor SAM applications containing WMI component monitors, the following conditions must be true: WMI on the remote server is enabled and functioning properly. The remote server is accessible through a RPC connection in order to run the WMI queries. If these conditions cannot be met, the WMI component monitors in SAM show an Unknown status. Examples of some issues that can prevent these conditions from being met include, but are not limited to the following: Trying to connect to a remote computer where you do not have local Administrator rights. A firewall blocking the WMI traffic. An operating system that is not configured for WMI. Mistyping the credential password in the SAM component monitor. To help diagnose and fix these issues and others, we can test the WMI services, the remote WMI connections, and the SolarWinds SAM component configuration to discover and correct the issues that can prevent your WMI component monitors from functioning correctly. Test WMI services using WBEMTest You can test WMI services to help isolate any faults on the target server. The testing program is a Microsoft program named WBEMTest that comes already installed on Microsoft Windows operating systems. You can use the application on the local server or login to the target server to run the application. You need to login with an Administrator account to run WBEMTest. 1. Click Start > Run, enter wbemtest.exe and then click OK. 2. Click Connect on the Windows Management Instrumentation Tester window. 13

14 Troubleshooting WMI 3. Enter root\cimv2 in the field at the top of the dialog box next to the Connect button. 4. Connect, then click Enum Classes. 14

15 Manually reset the WMI counters 5. Select the Recursive radio button without entering a superclass name, and then click OK. 6. If the list does not appear or does not contain the desired WMI class, WMI is not functioning correctly. 7. Click Close, and click Exit. Manually reset the WMI counters At times, the WMI performance counters may not get transferred to WMI because services were delayed or started out of order ( 1. Stop the Windows Management Instrumentation service. 2. Click Start, click Run, type cmd, and then click OK. 3. At the command prompt, type winmgmt /resyncperf, and then press Enter. 4. At the command prompt, type wmiadap.exe /f, and then press Enter. 5. Type exit, and then press Enter to close the command prompt. 6. Start the Windows Management Instrumentation service. After resetting the WMI counters, retest WMI. 15

16 Troubleshooting WMI Test remote WMI connectivity Testing the remote WMI connectivity of the target server helps isolate faults that could prevent the target server from receiving or responding to remote WMI requests. The testing program is a Microsoft program named WBEMTest that comes already installed on Microsoft Windows operating systems. You can test WMI remotely on the target server. Complete the following procedure to check whether the target server is responding appropriately to remote WMI requests that originate from the SolarWinds SAM server. 1. Log on to the SolarWinds SAM server with an administrator account, then click Start > Run, enter wbemtest.exe and then click OK. 2. Click Connect on the Windows Management Instrumentation Tester window. 3. Enter \\Target_Primary_IP_Address\root\cimv2 in the field at the top of the dialog box. Replace Target_Primary_IP_Address in the above example with the actual Hostname or Primary IP Address of the target server. 16

17 Verify administrator credentials 4. Enter the user name in the User field, the password in the Password field, and NTLMDOMAIN: NameOfDomain in the Authority field. Replace NameOfDomain with the domain of the user account specified in the User field. 5. Click Connect, then click Enum Classes. 6. Select the Recursive radio button without entering a superclass name, and then click OK. 7. If the list does not appear, remote WMI is not functioning correctly. Click Close, and then click Exit. Verify administrator credentials Only a credential that has administrator rights on the target server has the necessary permissions to access the target server s WMI services. Make sure that the username and password you are using belongs to an administrator on the target server. If the administrator credential is a domain member, be sure to specify both the user name and the domain in the standard Microsoft syntax. For example: DOMAIN\Administrator. Enable Remote Procedure Call (RPC) Remote WMI connections use RPC as a communications interface. If the RPC service is disabled on the target server, remote WMI connections cannot be established. You should log into the target server and enable RPC. 1. Log on to the target server with an administrator account. 2. Click Start > Run, then type services.msc, and then press Enter. 3. Scroll the list to Remote Procedure Call (RPC) 4. Right-click Remote Procedure Call (RPC), and then click Start on the shortcut menu. 17

18 Troubleshooting WMI Configure Distributed Component Object Model (DCOM) and User Account Control (UAC) If the target computer is running Windows Vista or Windows Server 2008, you may be required to make settings changes to allow remote WMI requests ( DCOM ITEM NEED Default and Limits permissions edited to allow the following actions: Local launch (default permission) Remote launch (default permission) Local activation (limits permission) Remote activation (limits permission) WMI Namespaces User Account Control Modify the CIMV2 security to enable and remote enable the account used to access the server or workstation through WMI. You must ensure the security change applies to the current namespace and sub-namespaces. Remote UAC access token filtering must be disabled when monitoring within a workgroup environment. Enable DCOM permissions for your SAM credentials WMI uses DCOM to communicate with monitored target computers. You need to enable and configure DOM on SAM to use WMI. 1. Log on to the target server with an administrator account. 2. Click Start > Control Panel > Administrative Tools > Component Services > Computers. 18

19 Enable Account Privileges in WMI 3. Right-click My Computer, and then select Properties. 4. Select the COM Security tab, and then click Edit Limits in the Access Permissions grouping. 5. Ensure the user account you want to use to collect WMI statistics has Local Access and Remote Access, and then click OK. 6. Click Edit Default, and then ensure the user account you want to use to collect WMI statistics has Local Access and Remote Access, then click OK. 7. Click Edit Limits in the Launch and Activation Permissions grouping. 8. Ensure the user account you want to use to collect WMI statistics has Local Launch, Remote Launch, Local Activation, and Remote Activation, and then click OK. 9. Click Edit Default, and then ensure the user account you want to use to collect WMI statistics Local Launch, Remote Launch, Local Activation, and Remote Activation. 10. Click OK. Enable Account Privileges in WMI The account you specify in the Credentials Library must possess security access to the namespace and sub-namespaces of the monitored target computer. 1. Log on to the computer to monitor with an administrator account. 2. Click Start > Control Panel > Administrative Tools > Computer Management > Services and Applications. 3. Click WMI Control, and then right-click and select Properties. 4. Select the Security tab, and then expand Root and click CIMV2. 5. Click Security and then select the user account used to access this computer and ensure you grant the following permissions: Enable Account Remote Enable 6. Click Advanced, and then select the user account used to access this computer. 7. Click Edit, select This namespace and subnamespaces in the Apply to field, and then click OK. 8. Click OK on the Advanced Security Settings for CIMV2 window. 9. Click OK on the Security for Root\CIMV2 window. 10. Click Services in the left navigation pane of Computer Management. 19

20 Troubleshooting WMI 11. Select Windows Management Instrumentation in the Services result pane, and then click Restart. Disable Remote User Account Control for workgroups If you are monitoring a target in a workgroup, you need to disable remote User Account Control (UAC). This is not recommended but it is necessary when monitoring a workgroup computer. Disabling remote user account control does not disable local user account control functionality. Warning: The following procedure requires the modification or creation of a registry key. Changing the registry can have adverse effects on your computer and may result in an unbootable system. Consider backing up your registry before making these changes. 1. Log on to the computer you want to monitor with an administrator account. 2. Click Start > Accessories > Command Prompt, then enter regedit. 3. Expand HKEY_LOCAL_ MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\ System. 4. Locate or create a DWORD entry named LocalAccountTokenFilterPolicy and provide a DWORD value of 1. To re-enable remote UAC, change this value to 0. Add a Windows firewall exception for remote WMI connections If the target computer has Windows Firewall enabled, it must have a Remote WMI exception to allow remote WMI traffic through ( 20

21 Determine if an additional polling engine is needed 1. Click Start, click Run, type cmd and then press Enter. 2. At the command prompt, type netsh advfirewall firewall set rule group="windows Remote Management" new enable=yes, and then press Enter. 3. At the command prompt, type exit, and then press Enter. Determine if an additional polling engine is needed You may need an additional polling engine for your environment. If you encounter the following issues or performance, you may need to add one or more additional pollers: Polling completion rates are lower than expected which leads to gaps in data. Sporadic component monitor timeouts usually resulting in component monitors showing an "Unknown" status, reporting the error, "Connection timeout. Job canceled by scheduler." Resource exhaustion on the SQL server. Installing additional pollers and Web Consoles help you extend your SolarWinds Server & Application Monitor (SAM) implementation. You can install additional polling engines to aid you in load balancing and configure additional websites to ensure redundant access through more than one web server. By sharing the same database, you can also share a unified user interface, making the addition of polling engines transparent to your staff. These components are licensed and purchased separately from your main SolarWinds Server & Application Monitor install and require the installation of the SolarWinds SAM Additional Polling Engine. For more information about purchasing licenses, contact your sales representative (sales@solarwinds.com) or customer service. Verify SAM component configuration The SAM component should use the credentials used for remote WMI. Verify you have these credentials set for the SAM components. 21

22 Troubleshooting WMI 1. Edit the SAM component and verify the credentials. 2. To save changes, click Set Component Credentials, then click Submit. Service reporting Invalid Class You may receive an error of Invalid Class for SAM services. The issue could be due to a number of issues including the incorrect polling method or issues with the WMI repository. If the following information does not resolve the issue, review information in our Success Center. Verify the polling type Services can be polled using RPC or WMI monitors. Verify the correct service is running for polling by navigating to Edit Application > RPC or WMI. If the correct monitor is used for polling, you may need to troubleshoot the monitors. 22

23 Repair the WMI Repository Repair the WMI Repository If the WMI monitor is used for polling and returning an error of Invalid Class, you need to repair the WMI repository. Use this KB article to resolve the issue: /WMI_Monitor_error%3A_Invalid_class Vista and Windows 2008 have a built-in method for repairing the WMI repository. 1. Open an Administrator command prompt and run: winmgmt /salvagerepository/ 2. You may need to resync the counters if you continue receiving the error. Resync the counters by opening a command prompt and typing: winmgmt /resyncperf 3. You should also verify the performance counters are not disabled. You may also need to switch the polling method from WMI to RPC, or vice versa, by editing the monitoring node. WMI is an operating system component. If the previous steps do no work, you may need to contact Microsoft for further information: WMI Troubleshooting. Microsoft Developer Network. 23

24 Troubleshoot hardware health Troubleshoot hardware health This following sections describe possible causes and solutions concerning hardware resources either not being reported or being reported incorrectly: Hardware prerequisite checklist SNMP monitored node not returning hardware information Troubleshoot a WMI node Troubleshoot a VMWare node Troubleshoot ESX hardware monitoring You can use the decision tree to help determine the cause and solution to your issue, provided in the links above. Dell does not make array and hard disk health information visible from WMI managed nodes. To monitor storage health on Dell servers, use SNMP. 24

25 Hardware prerequisite checklist Hardware prerequisite checklist Verify the following requirements to correctly display Hardware Health resources used to monitor hardware in SAM: 25

26 Troubleshoot hardware health The monitored node must be HP Proliant, Dell PoweEdge, IBM X-Series, HP C7000, HP C3000, or Dell M1000e. The node must be monitored using one of the following protocols: SNMP WMI ICMP nodes are allowed for VMWare when the Poll for VMware option is selected. The Hardware Monitoring Agent software, (provided by the vendor), is installed on the remote server. This applies for both SNMP and WMI. For VMware, the minimum requirements are as follows: ESX server version 3.5, 4.0, 4.1 ESXi version 5.0 vcenter version 4.0, 4.1, 5.0. SAM supports the following systems for hardware monitoring: Dell PowerEdge M610, R210, R610, R710, R900, 1950, 2850, 2950, 2970, 6850 HP ProLiant DL320 G4, DL360 G3, DL360 G4, DL380 G4, DL380 G6, ML570 G3 IBM IBM System x3550, System x3550 M2, System x3550 M3, System x3650, System x3650 M2, System x3650 M3, x3850, eserver 306m HP C7000, HP C3000 Dell M1000e IBM ServeRAID Manager software must be installed on IBM X-Series servers for storage hardware health information to be displayed in SolarWinds SAM. HP WBEM providers are required for HP servers polled via WMI. Hardware troubleshooting flowchart The following flowchart provides information for troubleshooting issues. You can use the decision tree to help determine the cause and solution to your issue. Dell does not make array and hard disk health information visible from WMI managed nodes. To monitor storage health on Dell servers, use SNMP. 26

27 27 Hardware troubleshooting flowchart

28 Troubleshoot hardware health SNMP monitored node not returning hardware information Error: SNMP monitored nodes may not return hardware health data if the software was installed improperly. You must install SNMP software prior to installing or deploying an Agent to the node. If you install the Agent first, you will encounter installation and configuration issues with MIBs. Resolution: Uninstall the Hardware Monitoring Agent software, install SNMP, then reinstall the Agent on the node. Review the Hardware Health resources for updated health data for the node. If you continue to encounter issues, you may need to review the following troubleshooting sections to verify the node and check OIDs. Verify the node was successfully added using SNMP Use the following steps to verify the monitored node has an SNMP polling method. 1. Verify the polling method on the Node Details page. 2. Verify the Hardware Monitoring Agent software is installed on the remote server and running. 3. Determine if SNMP responds for the proper OID. Below are the correct OIDS for each vendor: For HP: For Dell: For IBM:

29 Check OIDs using SNMPWalk.exe To determine if the remote server responds to the correct OID, you can use the MIB browser from SolarWinds Engineer s Toolset, which can be downloaded from Additionally, you can use other applications capable of making SNMP requests. Check OIDs using SNMPWalk.exe If you do not have a tool for checking OIDs on the remote server, you can create an SNMP walk by using the SNMPWalk.exe installed with SAM, normally located at C:\Program Files (x86)\solarwinds\orion\snmpwalk.exe. SNMPWalk.exe is used in this demonstration. 1. Start SNMPWalk.exe and type in the IP address of the remote server and the community string for SNMP. 2. Click Scan then save the SNMP walk in a text file. 3. Open the text file and manually search for the OIDs. If the Remote Server does not respond on this OID, the Hardware Monitoring Agent software may not be properly configured. Check to see if the Hardware Monitoring Agent software has imported the correct MIBs as outlined in the following table. HP Dell IBM CPQSTDEQ-MIB MIB-Dell IBM-SYSTEM-HEALTH-MIB CPQSINFO-MIB StorageManagement-MIB IBM-SYSTEM-ASSETID-MIB 29

30 Troubleshoot hardware health CPQIDA-MIB IBM-SYSTEM-LMSENSOR-MIB CPQHLTH-MIB IBM-SYSTEM-MEMORY-MIB CPQSTSYS-MIB IBM-SYSTEM-POWER-MIB CPQIDE-MIB IBM-SYSTEM-PROCESSOR-MIB IBM-SYSTEM-RAID-MIB ADAPTEC-UNIVERSAL-STORAGE-MIB Troubleshoot a WMI node The following conditions must be met before you can proceed troubleshooting WMI nodes: The node has successfully been added via WMI. WMI is working properly on the remote server. The Hardware Monitoring Agent software is installed on the remote server and running. When ready, use Wbemtest.exe to troubleshoot WMI: 1. Open wbemtest.exe, usually located at C:\Windows\System32\wbem\wbemtest.exe. 2. Connect from the problematic node (either the SAM server or the additional poller server) to the remote server using wbemtest.exe. 30

31 Troubleshoot a WMI node 3. Click Connect. In the Namespace field enter: For IBM and HP enter: \\RemoteServerIpAddress\root For Dell enter: \\RemoteServerIpAddress\root\cimv2 4. Enter administrator credentials and click Connect. 5. Once connected, click Query from the main screen. The Query dialog appears. 6. Enter: select * from Namespace 7. Replace Namespace with the following: For HP nodes, replace Namespace with HPQ For Dell nodes, replace Namespace with Dell For IBM nodes, replace Namespace with IBMSD 31

32 Troubleshoot hardware health 8. If the proper Namespace is found, connect to this Namespace. \\RemoteServerIpAddress\root\IBMSD for IBM. \\RemoteServerIpAddress\root\HPQ for HP. \\RemoteServerIpAddress\root\cimv2\Dell for Dell. 9. Run a Query for specific information: Select Manufacturer, Model, SerialNumber from CIM_Chassis 10. If the test was not successful, re-install the platform or Hardware Monitoring Agent software on the remote server with the latest release. Troubleshoot a VMWare node VMWare nodes can be polled for Hardware information either through the vcenter or directly by using the CIM protocol. Polling through the vcenter uses VMWare's native API interface. Polling the ESX server directly uses the CIM protocol to get Hardware information. You may need to log in with an administrator account to perform this action. Do the following to determine if a node is polled through the vcenter or directly: 32

33 Troubleshoot a VMWare node 1. On the Web Console, click Settings > Virtualization Settings. 2. Listed is the table of all the currently polled VMWare nodes. This table contains the Polling Through column. This column may be hidden. If the column is hidden, unhide it by clicking the drop down menu of an adjacent column and check the Polling. Through option: 3. Use the illustration below to determine how your VMWare is being polled. 33

34 Troubleshoot hardware health Troubleshoot ESX hardware monitoring The following error messages are associated with ESX hardware monitoring. Included are possible causes and solutions. Enable to connect to a remote server Error message: Unable to connect to remote server. Resolution: Ensure CIM is enabled on the ESX server. Ensure port 5989 is opened on the firewall. Enable to establish sessions Error message: The ESX server returns the message "Unable to establish session with all provided credentials." Resolution: Verify the VMware credentials in the VIM are correct. Verify the account belongs to the root user group. 34

35 Troubleshoot AppInsight for Exchange Troubleshoot AppInsight for Exchange The following sections will help you identify and correct AppInsight for Exchange errors concerning configuration and performance counters: Troubleshoot permissions Troubleshoot Exchange performance counters Troubleshoot error codes in AppInsight for Exchange Important: The following log file contains all the information and errors related to the WinRM configuration process: C:\ProgramData\Solarwinds\Logs\APM\RunWinRMConfigurator.lo g Troubleshooting Exchange Performance Counters Occasionally, you may encounter an Exchange server that is missing some of the expected performance counters. If this happens, you need to verify whether the counters are simply disabled or if they are completely missing. The simplest way to do this is through the registry. 1. Navigate to the service with the missing performance counters. Individual services are listed under HKLM\SYSTEM\CurrentControlSet\Services. 35

36 Troubleshoot AppInsight for Exchange 2. Expand the service and click on the Performance key. The important values we want to ensure exist are listed below. [HKEY_LOCAL_ MACHINE\SYSTEM\CurrentControlSet\Services\<ServiceName>\P erformance] Example: [HKEY_LOCAL_ MACHINE\SYSTEM\CurrentControlSet\Services\ESE\Performance] "Close"="ClosePerformanceData" "Collect"="CollectPerformanceData" "Library"="<Path to performance counter DLL file>" Example: "Library"="C:\\Program Files\\Microsoft\\Exchange Server\\V15\bin\\perf\\%PROCESSOR_ ARCHITECTURE%\\eseperf.dll" "Open"="OpenPerformanceData" "PerfIniFile"="<Name of performance counter INI file>" Example: "PerfIniFile"="eseperf.ini" The "Library" file path is typically "C:\Program Files\Microsoft\Exchange Server\%ExchangeVersion%\Bin\perf\%Processor_ Architecture%\%DLLFileName%" 3. Verify the counters have not been disabled by expand the service and then clicking on the Performance key. 36

37 Troubleshooting Exchange Performance Counters 4. Check for the value Disable Performance Counters If this value exists, ensure the data value is 0. Any other value disables the counters. 5. Once the value is confirmed to be set to 0, close all PerfMon windows and then reopen them. The performance counters should be visible at this time. If the performance counters are not visible, proceed to the next step. 6. If the values First Counter, First Help, Last Counter, and Last Help are listed, it is highly recommended to unload the performance counters prior to reloading them. 37

38 Troubleshoot AppInsight for Exchange Unload Performance Counters 1. Close all PerfMon windows and stop any services which may be using these counters. 2. Open the Exchange Management Shell (EMS) in the Run as Administrator context. 3. Type: Add-PSSnapin Microsoft.Exchange.Management.PowerShell.Setup and then press Enter. 4. Type: Remove-PerfCounters -DefinitionFileName "<Path to counter definition XML file> and then press Enter. The default location for Exchange counter definition files is: C:\Program Files\Microsoft\Exchange Server\%ExchangeVersion%\Setup\perf\%XMLFileName% Reload Performance Counters 1. Close all PerfMon windows and stop any services which may be using these counters. 2. Open the Exchange Management Shell (EMS) in the Run As Administrator context. 3. Type: Add-PSSnapin Microsoft.Exchange.Management.PowerShell.Setup and then press Enter. 4. Type: New-PerfCounters -DefinitionFileName "<Path to counter definition XML file> and then press Enter. a. The default location for Exchange counter definition files is: C:\Program Files\Microsoft\Exchange Server\%ExchangeVersion%\Setup\perf\%XMLFileName% 5. Check the application log to verify the counters were properly loaded and no PerfLib errors exist. Reopen PerfMon to ensure the counters are available. 38

39 Troubleshoot error codes in AppInsight for Exchange Troubleshoot error codes in AppInsight for Exchange The following error messages are associated with AppInsight for Exchange. Included are possible causes and solutions. The following log file contains all the information and errors related to the WinRM configuration process: C:\ProgramData\Solarwinds\Logs\APM\RunWinRMConfigurator.lo g For information about Microsoft error codes, see: Deployment service executable not found Error message: Remote configuration was unsuccessful due to the following: "The deployment service executable was not found on the Orion server." For details, see the log on the Orion server [ALLUSERSPROFILE] \ProgramData\Solarwinds\Logs\APM\RemoteExecutableStarter.log. Description: Internal error. The Remote Installation Service.exe file was not found. The default file location is C:\Program Files (x86) \SolarWinds\Orion\APM. This can be caused by an incorrect SAM installation or Remote Installation Service.exe was deleted or modified. Resolution: Add the Remote Installation Service.exe file to the following folder: C:\Program Files (x86)\solarwinds\orion\apm. Failed to start Windows Remote Management Error message: Remote configuration was unsuccessful due to the following: "Failed to start Windows Remote Management" For details, see the log on the Orion server [ALLUSERSPROFILE] \ProgramData\Solarwinds\Logs\APM\RemoteExecutableStarter.log. Resolution: Start the Windows Remote Management service: 1. On the target Exchange server, navigate to the Windows Services console and find the Windows Remote Management (WS-Management) service. 39

40 Troubleshoot AppInsight for Exchange 2. Set the Startup type to Automatic. 3. Start the Windows Remote Management (WS-Management) service. Unable to access remote admin Error: Unable to access remote administrator share. Cause: The remote administrator share cannot be accessed. Resolution: You can enable this feature manually through the following steps: 1. Ensure that both computers belong to the same Workgroup. 2. Specify which user(s) can access the Administrator Shares (Disk Volumes). 3. Enable File and print sharing through the Windows firewall. 4. Check to see if you can access the Administrator share from another computer. 5. When complete, rerun the AppInsight for Exchange Automatic Configuration Wizard. Configurator exe not found on the Orion server Error message: Remote configuration was unsuccessful due to the following: "The configurator executable was not found on the Orion server." For details, see the log on the Orion server [ALLUSERSPROFILE] \ProgramData\Solarwinds\Logs\APM\RemoteExecutableStarter.log. Description: Internal error. SolarWinds.APM.RemoteWinRmConfiguratorFull.exe file was not found. The default file location is C:\Program Files (x86)\solarwinds\orion\apm. This can be caused by an incorrect SAM installation or SolarWinds.APM.RemoteWinRmConfiguratorFull.exe was deleted or modified. Resolution: Add SolarWinds.APM.RemoteWinRmConfiguratorFull.exe to the following folder: C:\Program Files (x86)\solarwinds\orion\apm. Access denied Error message: Remote configuration was unsuccessful due to the following: "Access denied." For details, see the log on the Orion server [ALLUSERSPROFILE] \ProgramData\Solarwinds\Logs\APM\RemoteExecutableStarter.log) 40

41 Configurator exe has an invalid signature Description: The provided user account does not have access to the Administrator share on the remote Exchange server. Resolution: Connect to the Administrator share with \\ExchangeComputerName\<drive letter>$\ from the Orion server. Configurator exe has an invalid signature Error message: Remote configuration was unsuccessful due to the following: "The configurator executable has an invalid signature." For details, see the log on the Orion server: [ALLUSERSPROFILE] \ProgramData\Solarwinds\Logs\APM\RemoteExecutableStarter.log. Description: The configuration executable cannot be started on the remote Exchange server due to any of the following: The high level of User Account Control settings The provided user account does not have privileges high enough to run *.exe files SolarWinds.APM.RemoteWinRmConfiguratorFull.exe is not signed Resolution: with a Solarwinds certificate. Reduce User Account Control settings on the remote Exchange server; Ensure the user's account has privileges to run *.exe files Network path not found Error message: Remote configuration was unsuccessful due to the following: "The network path was not found." For details, see the log on the Orion server: [ALLUSERSPROFILE] \ProgramData\Solarwinds\Logs\APM\RemoteExecutableStarter.log. Description: This is a Windows error indicating that a networking component is malfunctioning. Resolution: Ensure you have access to the Shared Storage drive. In Windows Explorer, navigate to My Network Places > Entire Network > Microsoft Windows Network > (workgroup name) > (Shared Storage drive name). 41

42 Troubleshoot AppInsight for Exchange Firewalls can block traffic to the network which could generate this message. Try temporarily disabling any software or hardware firewalls to isolate the problem. Anti-spyware tools can block network traffic. Try temporarily disabling any antispyware programs and restart the computer to isolate the problem. PowerShell 2.0 not detected on the Exchange server Error message: Remote configuration was unsuccessful due to the following: "PowerShell 2.0 was not detected on the Exchange server." Learn how to correct this. For details, see the log on the remote computer. ([ALLUSERSPROFILE] \ProgramData\Solarwinds\Logs\APM\RunWinRMConfigurator.log). Description: PowerShell 2.0 is not installed on the remote Exchange server. Resolution: PowerShell 2.0 should be installed on Exchange server. To install PowerShell 2.0 on Server 2008, download the installation file for PowerShell 2.0 at and install it on the server. PowerShell 2.0 is automatically installed on Server 2008 R2 and therefore no additional installation is required. To install PowerShell 2.0 on Server 2012, complete the following: 1. Open Server Manager. 2. Click on the Manage menu, and the select Add Roles and Features. 3. After the wizard opens, click Next until you get to the Installation Type page. 4. Select Role-based or feature-based installation. 5. Click Next until you reach the Features page. 6. Scroll down to Windows PowerShell. It likely shows itself as partially installed (square inside box). 7. Check the box next to Windows PowerShell 2.0 Engine. 8. Click Next and then Install. 9. When the installation finishes, click Close. 42

43 Account must be an Active Directory account with local admin privileges Account must be an Active Directory account with local admin privileges Error message: Remote configuration was unsuccessful due to the following: "This account must be an Active Directory account with local administrative privileges." Learn how to correct this. For details, see the log on the remote computer: [ALLUSERSPROFILE] \ProgramData\Solarwinds\Logs\APM\RunWinRMConfigurator.log. Description: The provided user account does not have Local Administrative privileges. Resolution: All local administrator privileges to the Active Directory account: 1. On the server where you wish to grant local administrative privileges, open a Computer Management console. On Windows 2012, add this privilege using the Active Directory console. 2. Navigate to System Tools > Local Users and Groups > Groups and double click the Administrators group. 3. Click Add and type in the Active Directory user name of the account you want to grant administrative privileges, and then press Enter. Ensure the location is set to either the domain where the account is located or Entire Directory. 4. Click Apply and then click OK. Alternatively, you can add an Active Directory group to the local administrators group and add the Active Directory user accounts to that group. Account must be an Active Directory account with organization wide Exchange access Error message: Remote configuration was unsuccessful due to the following: "This account must be an Active Directory account with organization wide Exchange access (View-Only-Organization Management and Mailbox Search Management Role)." Learn how to correct this. For details, see the log on the remote computer: [ALLUSERSPROFILE] \ProgramData\Solarwinds\Logs\APM\RunWinRMConfigurator.log. Description: The provided user account does not have View-Only-Organization Management or Mailbox Search Management privileges. Resolution: You need to grant view-only-organization management privileges and mailbox search access to resolve. 43

44 Troubleshoot AppInsight for Exchange Granting Least Privilege access to the Exchange Organization using Active Directory Users and Computers (ADUC): 1. From the Start Menu, open ADUC and navigate to the Microsoft Exchange Security Groups OU. 2. Double click on the View-Only Organization Management group. 3. After the window opens, click the Members tab and then click Add. 4. Type the user name of the account you want to grant access to the Exchange organization, then click OK. 5. Click Apply, then click OK. 6. Close the ADUC window. Mailbox Search access is required to determine attachment counts and sizes. Grant the access using the Exchange Management Shell (EMS): 1. From the Start Menu, open the EMS. 2. Type New-ManagementRoleAssignment -Role "Mailbox Search" - User <Username of account being granted access> and press Enter. 3. To verify the management role has been properly assigned, enter the following command: Get-ManagementRoleAssignment -RoleAssignee <Username of account> Error occurred during Exchange Server Configuration Error message: Remote configuration was unsuccessful due to the following: "An error occurred during Exchange Server Configuration. Multiple connections to a server or shared resource by the same user, using more than one user name, are not allowed." Cause: This is a Windows Limitation when trying to map multiple drives to the same computer using different user accounts. Resolution: 1. Open the Computer Manager on the Exchange server 2. Expand Shared Folders and select Sessions. 3. Right-click on the users in the list and click Close Session. 44

45 PowerShell web site testing failed: Connecting to remote server 4. Once the session list is empty, retry the Configure Server button in AppInsight for Exchange. For more information, see this KB article: AppInsight for Exchange error: Remote configuration failed due to multiple connections PowerShell web site testing failed: Connecting to remote server Error message: Remote configuration was unsuccessful due to the following: "WinRM test was successful. PowerShell Exchange web site testing failed with the following error: Connecting to remote server..." Description: There are no default HTTPS bindings for all unassigned ports, (i.e. HTTPS 443 *.) Resolution: Establish the missing default HTTPS binding with an Exchange certificate. 45

46 Troubleshooting AppInsight for IIS Troubleshooting AppInsight for IIS The following sections help you identify and correct AppInsight for IIS errors concerning configuration and performance counters. Access is denied when configuring AppInsight for IIS Changing between 32-bit and 64-bit polling methods Credentials test failed Polling fails due to a missing certificate IIS Polling Failed Node unreachable IIS version failed WinRM testing failed Error code 1367 incorrect credentials Error code 1726 server configuration Error code deployment service exe not found Error code configuration exe not found Error code access denied Error code configuration exe has an invalid signature Error code network path not found Error code configuration file not found Error code unable to access remote admin share Error code HTTPS listener exists on port 5986 Error code unable to create self-signed certificate Error code unable to create wsman listener Error code failed to start Windows Remote Management Error Code unknown error IIS server configuration Error code PowerShell 2.0 or higher required Other errors Event IDs 46

47 Access is denied when configuring AppInsight for IIS Access is denied when configuring AppInsight for IIS Issue: You may encounter access denied errors when you choose not to use the builtin administrator account for IIS polling. This can occur if you create a new user and then add this user to the Administrator's group. Cause: If the group policy or the local policy has Administrator Approval Mode set to Enable, the polling will fail with an error. This is usually because the administrator account does not have enough permissions. Resolution: 1. In your local security policy, click Local Policies > Security Options > User Account Control: Run all administrators in Admin Approval Mode. This should be set to Disabled to let administrators poll properly. 2. Click Local Policies > Security Options > User Account Control: Admin Approval Mode for the Built-in Administrator Account. This should be Disabled for the built-in administrator to poll properly. 3. When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. 47

48 Troubleshooting AppInsight for IIS Change between 32-bit and 64-bit polling You should use 64-bit polling on 64-bit OS systems. Using AppInsight applications with 32-bit polling on 64-bit computers via an agent prevents certain performance counters from collecting information. You may need to log in with an administrator account to perform these changes. Change to 64-bit polling at the application level: 1. On the Web Console, click My Dashboards > Applications > SAM Summary. 2. Select an AppInsight Application (Exchange, SQL, IIS) and then click Edit Application Monitor. 3. Expand Advanced, and then click Override Template. 4. In the Platform to run polling job field, change the value to x Click Submit. Change to 64-bit polling at the template level: 1. On the Web Console, click Settings > All Settings > SAM Settings > Manage Templates. 2. Select an AppInsight application and click Edit. 48

49 Credentials test failed 3. Expand Advanced, and in the Platform to run polling job on field, change the value to x Click Submit. Credentials test failed Error: The credentials given are not valid. Please provide the correct credentials for an account with WMI permissions. Resolution: Enter valid credentials. For more information, see Inherit Credentials from Node. Polling fails due to a missing certificate Issue: You receive an error similar to the following: The SSL connection cannot be established. 49

50 Troubleshooting AppInsight for IIS Cause: A PowerShell listener was created but its certificate is missing. Resolution: Verify that the service on the remote host is properly configured to listen for HTTPS requests. Manually remove the existing PowerShell listener and re-run the IIS configuration wizard. This process creates a new listener along with the appropriate certificate needed. IIS Polling Failed Error: Application polling failed because IIS information is currently unavailable. Cause: This is a general error message returned by Windows when a networking component is malfunctioning. Resolution: For details, see the log on the Orion server located at: [ALLUSERSPROFILE] \ProgramData\Solarwinds\Logs\APM\RemoteExecutableStarter.log. Restart the remote registry service, check that TCP port 445 is open between the Orion server and the remotely monitored IIS server. When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. Node unreachable Error: Credentials test failed. Node is unreachable. Resolution: Ensure the node is currently being managed. For more information, see Unmanage Scheduling Utility. When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. IIS version failed Error: The current version of IIS is 6.0. AppInsight requires version 7.0 or higher. Resolution: Install IIS version 7.0 or higher. When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. WinRM testing failed Error: WinRM testing failed with the following error: Connecting to remote server failed with the following error message: Access is denied. 50

51 Error code 1367 incorrect credentials Resolution: Set the appropriate permissions for PowerShell on the target server. When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. Set the appropriate permissions for PowerShell on the target server: 1. On the remote computer, open the PowerShell console. 2. Execute the following PowerShell command: Set-PSSessionConfiguration Microsoft.PowerShell -ShowSecurityDescriptorUI -force When entered, a permissions dialog should open. 3. Enable Full Control for everyone. 4. Verify the the group the polling user belongs to has access to Microsoft PowerShell. Error code 1367 incorrect credentials Error: Incorrect user name or password. Resolution: Enter a valid user name and password combination. When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. For more information, see Inherit Credentials from Node. Error code 1726 server configuration Error: An error occurred during Server configuration. Cause: This can be due to a temporarily slow network or a heavily taxed computer at the moment of configuration causing the remote installer not to receive the necessary exit codes. Configuration is most likely successful at this point and the displayed error message is simply a response to the exit codes not being returned. Resolution: Running the configuration wizard again or clicking, I've already manually configured this server under Advanced should result in a successful configuration. 51

52 Troubleshooting AppInsight for IIS Error code deployment service exe not found Error: The deployment service executable was not found on the Orion server. Cause: The Remote Installation Service.exe file was not found. This could be caused by an incorrect SAM installation or the Remote Installation Service.exe was deleted or modified. Resolution: Add the Remote Installation Service.exe file to the following directory: C:\Program Files (x86)\solarwinds\orion\apm. For details, see the log on the Orion server located at: ([ALLUSERSPROFILE] \ProgramData\Solarwinds\Logs\APM\RemoteExecutableStarter.log). When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. Error code configuration exe not found Error: The configuration executable was not found on the Orion server. Cause: The SolarWinds.APM.RemoteIISConfigurator.exe file was not found. This can be caused by an incorrect SAM installation or the SolarWinds.APM.RemoteIISConfigurator.exe file was deleted or modified. 52

53 Error code access denied Resolution: Add the SolarWinds.APM.RemoteIISConfigurator.exe file to the following directory: C:\Program Files (x86)\solarwinds\orion\apm. For details, see the log on the Orion server located at: ([ALLUSERSPROFILE] \ProgramData\Solarwinds\Logs\APM\RemoteExecutableStarter.log). When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. Error code access denied Error: Access denied. Cause: The provided user account does not have access to the Administrator share on the remote IIS server. Resolution: Ensure the provided user account has access to the Administrator share by trying to connect to the Administrator share with the following: \\IISComputerName\<drive letter>$\ from the Orion server. When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. Error code configuration exe has an invalid signature Error: The configuration executable has an invalid signature. Cause: The configuration executable cannot be started on the remote IIS server due to one or more of the following: The high level of User Account Control settings; The provided user account does not have privileges to run executable files; The SolarWinds.APM.RemoteIISConfigurator.exe file is not signed with a Solarwinds certificate. Resolution: Try to lower the User Account Control settings on the remote IIS server. Also check that the user account has privileges to run executable files. When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. Error code network path not found Error: The network path was not found. 53

54 Troubleshooting AppInsight for IIS Cause: This is a general error message returned by Windows when a networking component is malfunctioning. This may be caused by not having the appropriate root or intermediate certificate on the remote IIS server. Resolution: For details, see the log on the Orion server located at: ([ALLUSERSPROFILE] \ProgramData\Solarwinds\Logs\APM\RemoteExecutableStarter.log). Following are some of the more common reasons why this error message might be returned by Windows: Ensure you have access to the shared storage drive. When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. Firewalls can block traffic to the network which could generate this message. Try temporarily disabling any software or hardware firewalls to isolate the problem. When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. Like firewalls, some anti-spyware tools have the same effect. Try temporarily disabling any anti-spyware programs and restart the computer to isolate the problem. When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. Error code configuration file not found Error: The configuration file for the configuration executable was not found on the Orion server. Cause: This is an internal error. This can be caused by either an incorrect SAM installation, or the SolarWinds.APM.RemoteIISConfigurator.exe.config file was deleted or modified. Resolution: Add the SolarWinds.APM.RemoteIISConfigurator.exe.config file to the following directory: C:\Program Files (x86) \SolarWinds\Orion\APM. For details, see the log on the Orion server located at: [ALLUSERSPROFILE] \ProgramData\Solarwinds\Logs\APM\RemoteExecutableStarter.log. When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. 54

55 Error code unable to access remote admin share Error code unable to access remote admin share Error: Unable to access remote administrator share. Cause: The remote administrator share cannot be accessed. Resolution: You can enable this feature manually through the following steps: 1. Ensure that both computers belong to the same Workgroup. 2. Specify which user(s) can access the Administrator Shares (Disk Volumes). 3. Enable File and Print Sharing through the Windows firewall. 4. Check to see if you can access the Administrator share from another computer. 5. When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. Error code HTTPS listener exists on port 5986 Error: An HTTPS listener currently exists on port Cause: This error means that an HTTPS listener already exists on port 5986 and cannot be reused because it is not configured to listen for the target IIS server IP address. Resolution: For details, see the log on the target computer located at: [ALLUSERSPROFILE] \ProgramData\Solarwinds\Logs\APM\RemoteIISConfigurator.log and RemoteIISConfiguratorPowerShell.log. For configuring WinRM using a different port, complete the following steps. When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. Configure WinRM on a server: Important: To configure the listener, change 5986 to an available port number throughout these steps. 55

56 Troubleshooting AppInsight for IIS 1. Open a command prompt with the option Run as Administrator. 2. Type: winrm create {Port="5986";CertificateThumbprint="<Thumbprint value of certificate>";hostname="<ip Address of Server>_Solarwinds_ IIS_Zero_Configuration"} and press Enter. 1. Verify the configuration by typing the following: winrm get winrm/config/listener?address=*+transport=https. Next, create a firewall rule: 56

57 Error code HTTPS listener exists on port Open PowerShell in the Run as Administrator context. 2. Create a function for adding firewall rules using the following code: function Add-FirewallRule { param( $name, $tcpports, $appname = $null, $servicename = $null ) $fw = New-Object -ComObject hnetcfg.fwpolicy2 $rule = New-Object -ComObject HNetCfg.FWRule $rule.name = $name if ($appname -ne $null) { $rule.applicationname = $appname } if ($servicename -ne $null) { $rule.servicename = $servicename } $rule.protocol = 6 #NET_FW_IP_PROTOCOL_TCP $rule.localports = $tcpports $rule.enabled = $true $rule.grouping = "@firewallapi.dll,-23255" $rule.profiles = 7 # all $rule.action = 1 # NET_FW_ACTION_ALLOW $rule.edgetraversal = $false $fw.rules.add($rule) } 57

58 Troubleshooting AppInsight for IIS 3. Run the function to create the firewall exception for WSMAN using the following command: Add-FirewallRule "Windows Remote Management" "5986" $null $null 4. Verify the rule was created by navigating to Control Panel > Windows Firewall > Advanced Settings. Error code unable to create self-signed certificate Error: Unable to create a self-signed certificate. Cause: An unexpected error occurred during the certificate creation process. Resolution: Check the configuration script log file located on target the IIS server, located at [ALLUSERSPROFILE] \ProgramData\Solarwinds\Logs\APM\RemoteIISConfigurator.log. When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. 58

59 Error code unable to create wsman listener Error code unable to create wsman listener Error: Unable to create wsman listener. Cause: This error means that an unexpected error occurred during wsman listener creation. Resolution: Check the configuration script log file located on the target IIS server at, [ALLUSERSPROFILE] \ProgramData\Solarwinds\Logs\APM\RemoteIISConfigurator.log. When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. Error code failed to start Windows Remote Management Error: Failed to start Windows Remote Management. Cause: This error means that the Windows Remote Management (WS-Management) service cannot be started remotely. Resolution: Ensure the Windows Remote Management service is running on the target IIS server. Error Code unknown error IIS server configuration Error: An unknown error occurred during IIS server configuration. Resolution: For more information, check the errors listed in the Other errors information. After isolating and resolving the issue, rerun the AppInsight for IIS Automatic Configuration Wizard. Error code PowerShell 2.0 or higher required Error: PowerShell 2.0 or higher is required. PowerShell is used for scripting in SAM. 59

60 Troubleshooting AppInsight for IIS Cause: PowerShell 2.0 or higher is not installed on the target server. Resolution: Ensure that PowerShell 2.0 or higher is installed on the target server. When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. For Windows Server 2008, download the installation file for PowerShell 2.0 at and install it on the target server. PowerShell 2.0 is automatically installed on Windows Server 2008 R2 and therefore no additional installation is required. Other errors The following compiled errors provide additional information and troubleshooting AppInsight for IIS. PowerShell 2.0 not installed Issue: PowerShell 2.0 is not Installed Resolution: Install PowerShell. See the following instructions for installing PowerShell andset the permissions. To install PowerShell 2.0 on Server 2008, download the installation file for PowerShell 2.0 at and install it on the server. PowerShell 2.0 is automatically installed on Server 2008 R2 and therefore no additional installation is required. Set the appropriate permissions for PowerShell on the target server: 1. On the remote computer, open the PowerShell console. 2. Execute the following PowerShell command: Set-PSSessionConfiguration Microsoft.PowerShell -ShowSecurityDescriptorUI -force. The permissions dialog should appear. 3. Enable Full Control. 4. Ensure that the group to which the polling user belongs has access to Microsoft PowerShell. 5. When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. 60

61 PowerShell WebAdministration snapin not installed PowerShell WebAdministration snapin not installed Issue: PowerShell WebAdministration snapin is not installed. Information: For legacy systems with IIS 7.0 (Windows Vista, Windows 2008) PowerShell WebAdministration snapins are not part of IIS. Resolution: Download and install the PowerShell WebAdministration snapins found at the following link: When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. Polling IIS unsuccessful Issue: Polling IIS servers is unsuccessful. Possible Errors: OutOfMemoryException, Not enough storage, WinRM, WinRM not functioning properly, and so on. Cause: IIS servers hosting many sites may cause polling to be unsuccessful. Resolution: Increase the polling frequency and Memory per Shell Limit on the target node as needed. When complete, rerun the AppInsight for IIS Automatic Configuration Wizard. Complete the following to increase the polling frequency: The default value is set at 300 seconds (five minutes). If you see more than a five minute gap in a graphed resource, this value can be increased, as in any other SAM application. 1. Click My Dashboards > SAM Summary, and then select an application to be taken to the Application details page. 2. Click Edit Application Monitor > Override template (next to Polling Frequency). 3. Change to the desired value, and then click Submit. Increase the memory per shell limit 1. Run PowerShell via Run as Administrator and execute the following command to get the actual value: winrm get winrm/config/winrs 61

62 Troubleshooting AppInsight for IIS 2. Examine MaxMemoryPerShellMB. If the value is less than 512, increase this value by executing the following command: Set-Item WsMan:\localhost\Shell\MaxMemoryPerShellMB NewValue -WarningAction SilentlyContinue; where the NewValue is the desired memory limit. If the error persists, the value should be increased again. Event IDs The matrix below outlines and defines Microsoft event IDs to help aid in troubleshooting: EVENT ID SOURCE LOG SEVERITY MESSAGE 2281 Microsoft- IIS- WMSVC 2297 Microsoft- IIS- W3SVC- WP 5039 Microsoft- WAS 5015 Microsoft- WAS 5127 Microsoft- WAS Application Warning Worker process failed to initialize communication with the World Wide Web Publishing Service and therefore could not be started Application Warning Worker process for application pool encountered an error while trying to read global module configuration System Warning Worker process serving an application pool reported a failure System Warning Worker process serving application pool was orphaned System Warning Worker process serving the application pool is no longer trusted by Windows Process Activation Service (WAS) 62

63 Event IDs EVENT ID SOURCE LOG SEVERITY MESSAGE 1173 Microsoft- IIS-W3SVC 1026 Microsoft- IIS-W3SVC 1135, 1134 Microsoft- IIS- WMSVC 1020 Microsoft- IIS- WMSVC 1062 Microsoft- IIS- WMSVC 1126 Microsoft- IIS-W3SVC 1175 Microsoft- IIS-W3SVC 1071, 1073 Microsoft- IIS-W3SVC System Error World Wide Web Publishing Service did not initialize the HTTP driver and was unable start System Warning World Wide Web Publishing Service encountered an error when it tried to secure the handle of the application pool System Warning World Wide Web Publishing Service failed to configure the centralized World Wide Web Consortium (W3C) logging properties System Warning World Wide Web Publishing Service failed to configure the HTTP.SYS control channel property System Warning World Wide Web Publishing service failed to configure the logging properties for the HTTP control channel System Warning World Wide Web Publishing Service failed to copy a change notification for processing System Warning World Wide Web Publishing Service failed to enable end point sharing for the HTTP control channel System Warning World Wide Web Publishing Service failed to enable global bandwidth throttling 63

64 Troubleshooting AppInsight for IIS EVENT ID SOURCE LOG SEVERITY MESSAGE 1087 Microsoft- IIS-W3SVC 1086 Microsoft- IIS-W3SVC 5067 Microsoft- WAS 1037 Microsoft- IIS-W3SVC 2208 Microsoft- IIS- WMSVC 2206 Microsoft- IIS- WMSVC 2201 Microsoft- IIS- WMSVC 2203 Microsoft- IIS- WMSVC System Warning World Wide Web Publishing Service failed to properly configure the application pool queue length System Warning World Wide Web Publishing Service failed to properly configure the load balancer capabilities on application pool System Warning World Wide Web Publishing Service property failed range validation System Warning HTTP control channel for the World Wide Web Publishing Service did not open Application Warning HTTP Server could not create a client connection object for user Application Warning HTTP Server could not create the main connection socket Application Warning HTTP Server could not initialize its security Application Warning HTTP Server could not initialize the socket library 64

65 Event IDs EVENT ID SOURCE LOG SEVERITY MESSAGE Application Warning HTTP Server was unable to initialize because of a shortage of available memory 2274, 2268, 2220, 2219, , 5061, Microsoft- IIS- WMSVC Microsoft- IIS- WMSVC Microsoft- WAS Application Warning ISAPI application error detected System Warning Job object associated with the application pool encountered an error 2227 Microsoft- IIS- WMSVC 2233 Microsoft- IIS- WMSVC Application Warning The server was unable to acquire a license for an SSL connection Application Warning The server was unable to allocate a buffer to read a file 2226, 2230, 2231, , 5179, 5180 Microsoft- IIS- WMSVC Microsoft- WAS Application Warning The server was unable to read a file System Warning Windows Process Activation Service (WAS) detected invalid configuration data 5085 Microsoft- WAS System Warning Windows Process Activation Service (WAS) did not apply configuration changes to application pool 65

66 Troubleshooting AppInsight for IIS EVENT ID SOURCE LOG SEVERITY MESSAGE 5054, 5091 Microsoft- WAS 5063 Microsoft- WAS 5058 Microsoft- WAS 5066 Microsoft- WAS 5153 Microsoft- WAS 5172, 5173 Microsoft- WAS 5014 Microsoft- WAS 5101 Microsoft- WAS System Error Windows Process Activation Service (WAS) did not run the automatic shutdown executable for application pool System Warning Windows Process Activation Service (WAS) encountered a failure requesting IIS configuration store change notifications System Warning Windows Process Activation Service (WAS) encountered a failure while setting the affinity mask of an application pool System Error Windows Process Activation Service (WAS) encountered an error attempting to configure centralized logging System Warning Windows Process Activation Service (WAS) encountered an error attempting to look up the built-in IIS_IUSRS group System Warning Windows Process Activation Service (WAS) encountered an error trying to read configuration System Warning Windows Process Activation Service (WAS) encountered an internal error while managing a worker process System Error Windows Process Activation Service (WAS) failed to create application pool 66

67 Event IDs EVENT ID SOURCE LOG SEVERITY MESSAGE 5070, 5093 Microsoft- WAS 5005 Microsoft- WAS 5053 Microsoft- WAS 2296 Microsoft- IIS- W3SVC- WP 2295 Microsoft- IIS- W3SVC- WP 2293 Microsoft- IIS- WMSVC 1133 Microsoft- IIS-W3SVC 2261 Microsoft- IIS- WMSVC System Warning Windows Process Activation Service (WAS) failed to issue recycle request to application pool System Error Windows Process Activation Service (WAS) is stopping because it encountered an error System Warning Windows Process Activation Service (WAS) received a change notification but was unable to process it correctly Application Warning Module has an invalid precondition Application Warning Module registration error detected (failed to find RegisterModule entry point) Application Warning Module registration error detected (module returned an error during registration) System Warning Only one type of logging can be enabled at a time Application Warning SF_NOTIFY_READ_RAW_DATA filter notification is not supported in IIS 67

68 Troubleshooting AppInsight for IIS EVENT ID SOURCE LOG SEVERITY MESSAGE 5036 Microsoft- WAS 2264 Microsoft- IIS- WMSVC 2298 Microsoft- IIS- W3SVC- WP 2218 Microsoft- IIS- WMSVC 2258 Microsoft- IIS- WMSVC 5152 Microsoft- WAS System Error The configuration manager for Windows Process Activation Service (WAS) did not initialize Application Warning The directory specified for caching compressed content is invalid Application Warning The Global Modules list is empty Application Warning The HTTP server encountered an error processing the server-side include file Application Warning The server failed to close client connections to URLs during shutdown System Warning Application pool has an IdleTimeout equal to or greater than the PeriodicRestart time 5010, 5011, 5012, 5013 Microsoft- WAS System Warning Application pool worker process is unresponsive 5009 Microsoft- WAS System Warning Application pool worker process terminated unexpectedly 68

69 Event IDs EVENT ID SOURCE LOG SEVERITY MESSAGE 2216 Microsoft- IIS- WMSVC 2221 Microsoft- IIS- WMSVC 2222 Microsoft- IIS- W3SVC- WP Application Warning A script has not responded within the configured time-out period Application Warning A server-side-include file has included itself, or the maximum depth of server-side include files has been exceeded Application Warning An attempt was made to load an ISAPI filter on a server instance, but because it requires the SF_ NOTIFY_READ_RAW_DATA filter notification, it must be loaded as a global filter 500, 499, 23, 22, 21, 20, 19, 18, 17, 16, 9, 8, 7, 6, 5 Active Server Pages Application Warning ASP application error occurred WAS System Error Windows Process Activation Service (WAS) terminated unexpectedly, and the system was not configured to restart it. 2279, 2278, Microsoft- Microsoft- IIS- W3SVC- WP Application Warning Worker process encountered an error during initialization 69

70 Troubleshooting AppInsight for IIS EVENT ID SOURCE LOG SEVERITY MESSAGE 1005 Microsoft- IIS-W3SVC 8001 Microsoft- IIS- WMSVC The World Wide Web Publishing Service (WWW Service) is stopping because it encountered an error. The data field contains the error number The Web Management Service was stopped because it did not respond 70

71 Troubleshoot permissions Troubleshoot permissions You may encounter permissions issues with the following account and service types. These accounts include 3rd party applications and services, which may require administration credentials to review and solve. Non-Domain Accounts Adding Local Administrative privileges to Active Directory Account Exchange Access Mailbox Exchange Access Mailboxes with an empty primary SMTP address can be polled; however, their Sent and Received statistics are not available. Non-Domain Account Local accounts (Non-Domain) cannot access Exchange Management interfaces and therefore are not supported by AppInsight for Exchange. Please select an Active Directory account or create a new one to use with AppInsight for Exchange. Add Local Administrative privileges to Active Directory Account 1. On the server you need to grant local administrative privileges, open a Computer Management console. On Windows 2012, add this privilege using the Active Directory console. 2. Navigate to the Administrators group. 3. Add the type in the Active Directory user name of the account you want to grant administrative privileges. Ensure the location is set to either the domain where the account is located or Entire Directory. 4. Save your changes. Alternatively, you can add an Active Directory group to the local administrators group and add the Active Directory user accounts to that group. 71

72 Troubleshoot permissions To verify the account and local group membership has been configured properly, run the following code in a PowerShell session: $Recurse = $true $GroupName = 'Administrators' Add-Type -AssemblyName System.DirectoryServices.AccountManagement $ct = [System.DirectoryServices.AccountManagement.ContextType]::Mac hine $group = [System.DirectoryServices.AccountManagement.GroupPrincipal]:: FindByIdentity($ct,$GroupName) $LocalAdmin = $group.getmembers($recurse) {N='Domain'; E={$_.Context.Name}}, {N='ObjectType'; E={$_.StructuralObjectClass}} -Unique $LocalAdmin = $LocalAdmin Where-Object {$_.ObjectType -eq "user"} 72