EMC IT Compliance Analyzer Storage Edition Version 1.1

Transcription

1 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide P/N REV A04 EMC Corporation Corporate Headquarters: Hopkinton, MA

2 Copyright 2009 EMC Corporation. All rights reserved. Published December 2009 EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All other trademarks used herein are the property of their respective owners. 2 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

3 Contents Preface Chapter 1 Chapter 2 Chapter 3 Chapter 4 Chapter 5 Initial Setup Administration overview Access and Log in to the console Create a new Security Administrator account User Management Overview Manage Users Manage Roles Assign Role Edit Users Manage Authentication Change Lockbox Password System Administration System Notifications Manage User-defined Groups User-defined groups System groups Static user-defined groups Dynamic user-defined groups Subgroups User-Defined Groups view Discovery Overview Manage access profiles Profile types Manage initial discovery policies Creating initial discovery policies Discovery policy execution results Discovery view Manage rediscovery policies EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide 3

4 Contents Configure host resolution Configure non-discovered objects Manage object inventory Manage Discovery Manager status Chapter 6 Chapter 7 Appendix A Database Administration Database Administration overview Back-up repositories Manually back-up discovered objects, Sybase and Lockbox repositories 47 Restore repositories Restore only Sybase repository Restore all repositories Retention policy Install Providers Overview EMC SMI-S Provider for Symmetrix and CLARiiON Windows installation Linux installation Adding CLARiiONs to the EMC SMI-S Provider for discovery In-band CLARiONs Using the TestSMiProvider application to add Symmetrix and CLARiiON arrays to the EMC SMI-S Provider Brocade SMI Agent Brocade SMI Agent for EOS (McDATA) HiCommand for Hitachi arrays Troubleshooting Overview General provider configuration issues Brocade and McDATA provider McDATA guides No elements detected with Brocade and McDATA CIMOMs Brocade provider not sending events Brocade provider stops sending events Brocade provider consuming memory and CPU Error message: Operation Not Supported SMI-S Provider for Symmetrix and CLARiiON Provider versions Identifying SMI-S Provider version numbers Logging troubleshooting data Disabling statistical information gathering Solutions Enabler configuration error regarding RA Directors Discovery errors related to timing out VCMDB devices Starting the cimserver on Linux Database errors in the log Connection failed message Error opening gatekeeper device Array DM errors caused by EMC SMI Provider/Solutions Enabler related issues HiCommand for HDS arrays HiCommand Device Manager does not support an automatic refresh of array 4 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

5 Contents data Troubleshooting Host discovery Supported HBA, driver, and firmware versions Discovering HBA path data Failure to open WMI sessions Common reasons for discovery failure with Windows hosts Issues on UNIX hosts with sudo accounts General SMI issues Error message: HTTP Error (401 Unauthorized) Error message: Cannot connect to <IP address>. Connection failed Error message: Unable to reach element Troubleshooting the gdpm DM Guidelines for analyzing troubleshooting data Troubleshooting an offline array Domain Manager Using SNMP Cisco SNMP Rediscoveries not invoked on configuration changes Troubleshooting user interface conditions Default value for the UDG refresh period Discovery status permanently stays in running state Discovery status: one or more element discoveries failed Discovery status: no elements detected Out of memory errors: the user interface hangs, stalls, or slows down UI logs for troubleshooting Unable to open UI Browser to the appliance Unable to see any data in the UI even after creating profiles and policies 87 Progress bar for loading data hangs for a long time in any view Unable to sort or filter by 'Last Successful' Time of Discovery in the Discovered Objects view Object Details and Maps not showing the real-time environment Paths shown in maps do not match the Path Details tab Useful commands and utilities Utilities Commands Using discovery logs Sending logs and repository files to Customer Support Rediscovery scenarios for user-defined groups Re-imaging the appliance Acquire network settings and log in Changing the time zone Linux-based time zones EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide 5

6 Contents 6 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

7 Figures Title Page 1 Login screen Administration screen System Notifications dialog Access Profiles view Back-up Repository dialog box Historical data retention policy dialog Interface window Network configuration window Edit Interface eth0 popup Time zone selection EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide 7

8 Figures 8 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

9 Tables Title Page 1 ITCA SE role privileges Host resolution parameters Useful utilities Useful commands Domain Manager logs Logs and repository files for diagnosis Document Type Variable Footer Draft version 9

10 Tables 10 Document Type Variable Footer Draft version

11 Preface As part of an effort to improve and enhance the performance and capabilities of its product lines, EMC periodically releases revisions of its hardware and software. Therefore, some functions described in this document may not be supported by all versions of the software or hardware currently in use. For the most up-to-date information on product features, refer to your product release notes. If a product does not function properly or does not function as described in this document, please contact your EMC representative. Audience Related documentation This document is part of the EMC IT Compliance Analyzer Storage Edition documentation set, and is intended for use by system administrators and analysts. Related documents include: EMC IT Compliance Analyzer Storage Edition 1.1 Administration Guide EMC IT Compliance Analyzer Storage Edition 1.1 Non-EMC Software EMC IT Compliance Analyzer Storage Edition 1.1 Hardware Replacement Guide EMC IT Compliance Analyzer Storage Edition 1.1 Product Security Defaults EMC IT Compliance Analyzer Storage Edition 1.1 Support Matrix EMC IT Compliance Analyzer Storage Edition 1.1 Online Help EMC IT Compliance Analyzer Storage Edition 1.1Release Notes EMC IT Compliance Analyzer Storage Edition 1.1 Performance and Scalability Guidelines EMC IT Compliance Analyzer Storage Edition 1.1 Installation Quick Start EMC IT Compliance Analyzer Storage Edition 1.1 Installation Guide EMC IT Compliance Analyzer Storage Edition 1.1 User Guide EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide 11

12 Preface Conventions used in this document EMC uses the following conventions for special notices. Note: A note presents information that is important, but not hazard-related.! CAUTION A caution contains information essential to avoid data loss or damage to the system or equipment.! IMPORTANT An important notice contains information essential to operation of the software. Typographical conventions EMC uses the following type style conventions in this document: Normal Used in running (nonprocedural) text for: Names of interface elements (such as names of windows, dialog boxes, buttons, fields, and menus) Names of resources, attributes, pools, Boolean expressions, buttons, DQL statements, keywords, clauses, environment variables, filenames, functions, utilities URLs, pathnames, filenames, directory names, computer names, links, groups, service keys, file systems, notifications Bold Italic Courier Courier bold Used in running (nonprocedural) text for: Names of commands, daemons, options, programs, processes, services, applications, utilities, kernels, notifications, system call, man pages Used in procedures for: Names of interface elements (such as names of windows, dialog boxes, buttons, fields, and menus) What user specifically selects, clicks, presses, or types Used in all text (including procedures) for: Full titles of publications referenced in text Emphasis (for example a new term) Variables Used for: System output, such as an error message or script URLs, complete paths, filenames, prompts, and syntax when shown outside of running text Used for: Specific user input (such as commands) Courier italic Used in procedures for: Variables on command line User input variables < > Angle brackets enclose parameter or variable values supplied by the user [ ] Square brackets enclose optional values Vertical bar indicates alternate selections - the bar means or { } Braces indicate content that you must specify (that is, x or y or z)... Ellipses indicate nonessential information omitted from the example 12 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

13 Preface Where to get help Your comments EMC support, product, and licensing information can be obtained as follows. Product information For documentation, release notes, software updates, or for information about EMC products, licensing, and service, go to the EMC Powerlink website (registration required) at: Technical support For technical support, go to EMC Customer Service on Powerlink. To open a service request through Powerlink, you must have a valid support agreement. Please contact your EMC sales representative for details about obtaining a valid support agreement or to answer any questions about your account. Your suggestions will help us continue to improve the accuracy, organization, and overall quality of the user publications. Please send your opinion of this document to: techpubcomments@emc.com EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide 13

14 Preface 14 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

15 1 Initial Setup This chapter describes the necessary steps for setting up and maintaining the ITCA SE environment. Topics include: Administration overview Access and Log in to the console Create a new Security Administrator account Initial Setup 15

16 Initial Setup Administration overview After installation of ITCA SE hardware and software initialization as described in the EMC IT Compliance Analyzer Storage Edition Installation Guide, you can begin to administer ITCA SE for your environment.! IMPORTANT The Admin user with Security Administrator privilege is the highest privilege in ITCA SE. Only Security Administrator role allows you to manage users including user roles. An Admin user with Administrator role can manage all other tasks except for User Management. ITCA SE allows only one Admin with Security Administrator role per ITCA SE implementation. ITCA SE allows you, the Admin user, to set up and manage the following tasks: System notifications Discovery management including management of: Access Profiles Object inventory Initial discovery policies Rediscovery policies Host resolution parameters Non-discovered hosts Database administration including administration of: Retention policies Back-up repository schedules Repository restore User-defined group management create static and dynamic user-defined groups User management (Security Administrator role only), includes management of: User authentication Users add, delete, modify users User roles assign privileges 16 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

17 Initial Setup Access and Log in to the console This section describes the necessary steps to access and log in to ITCA SE. ITCA SE is provided with a default Admin with Security Administrator privileges for initial log in. The Security Administrator is the only administrative role that can add and modify users, as well as perform normal administrative tasks. After logging in to the system, ensure that you create other users with administrative and nonadministrative roles as described in Chapter 2, Manage Roles on page 24. Initial log in To access and log in to ITCA SE: 1. Open Windows Internet Explorer (IE), or Firefox browser. Note: Make sure your screen resolution is set to at least 1024 x 768. For IE users, launch the browser from the Windows Desktop icon or Start, Programs menu each time you access an ITCA SE console. Performance issues can occur when using either the File > New Window menu option, or the Ctrl + N shortcut key to launch a browser in which to run an ITCA SE console. In addition, make sure you disable popup blockers. 2. Type the URL for the ITCA SE appliance: IP address>/itcase For example: The EMC IT Compliance Analyzer Storage Edition Login screen opens, as shown in Figure 1 on page 17. Figure 1 Login screen Access and Log in to the console 17

18 Initial Setup 3. Type the username and password of the default Security Administrator: Username: admin Password: qw 4. Click Login to display the ITCA SE Dashboard. Note: If the login fails, click? in the dialog box title bar for a list of possible errors, causes, and solutions.! CAUTION After initial login, it is highly recommended that you log in to the Admin with Security Administrator role account, and change the default admin password to protect the security of your system. Create a new Security Administrator account on page 19 provides instructions. 18 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

19 Initial Setup Create a new Security Administrator account The Admin user with Security Administrator role is the default user provided with ITCA SE and cannot be edited, disabled, or deleted. However, the admin password can be changed an Admin user with a Security Administrator user role, thereby creating an Security Administrator account for that user. To create a new Security Administrator account: 1. Log into ITCA SE with username: admin and password: qw 2. From the ITCA SE Dashboard, click the Administration tab to display the Administration menu functions, as shown in Figure 2 on page 19. Figure 2 Administration screen 3. Click Manage Users. 4. Click Create Users in the Manage Users screen. 5. In the Create User dialog, type: First Name: <First name of Security Administrator owner> Last Name: <Last name of Security Administrator owner> <Security Administrator s address> For User Role, select Security Administrator from the drop-down menu. 6. Enter User Authentication information. a. Select desired Authentication Method from the drop-down menu. b. Type User Name. c. Type the password and then retype the password. Create a new Security Administrator account 19

20 Initial Setup! IMPORTANT Ensure that you make a note of the username and password you provide here. After you type it, there is no other way to retrieve the information. 7. Click OK to create the user and return to the Manage Users screen. 8. Log out of ITCA SE. 9. Ensure that you can: Log in again with the new Security Administrator credentials. If you cannot log in, use the backup username admin and password qw, and check whether the username and password for the new Security Administrator are correct. See the Administration options on the Administration screen, as shown in Figure 2 on page 19. Select User Administration. If you cannot see the Administration options or if the User Administration button is disabled, then you may not have selected the User Role and Permissions correctly. To fix the mistake: a. Log out of the system. b. Log in again as admin with password qw. c. Go to Manage Users. d. Select the new administrator from the list and click Edit. e. Select Security Administrator for User Role. f. Select Create Policies and Security Administrator for Permissions. g. Click OK to save the changes. h. Repeat steps 5, 6, and EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

21 2 User Management This chapter discusses creating and managing users including assigning user roles, and managing user authentication. Topics include: Overview Manage Users Manage Roles Manage Authentication Change Lockbox Password User Management 21

22 User Management Overview EMC IT Compliance Analyzer Storage Edition allows the Security Administrator user to perform User Management functions. User Management consists of managing Users, User Authentication, and User Roles. ITCA SE provides a default User and User Role. Additional Users and User Roles can be created by the Security Administrator user and are managed from the ITCA SE Console Administration tab. This section provides an overview of the Manage User features. The ITCA SE online help provides field descriptions and definitions for User Management features. The default user is secadmin, and password is qw. The secadmin user is assigned the Security Administrator role, and therefore has all necessary permissions for user management and administration. The secadmin user cannot be deleted or disabled.! CAUTION It is highly recommended that you do not use the default secadmin account. It is best to create a new Security Administrator account and change the secadmin password to protect the security of your system. Create a new Security Administrator account on page 19 provides instructions. 22 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

23 User Management Manage Users To create additional users for the EMC IT Compliance Analyzer Storage Edition software: 1. Log in with Security Administrator permissions. 2. Click Administration to display the Administration tab. 3. Under User Management, click Manage Users. 4. In the Manage Users view, click Create User. Complete the Create User dialog to add a new user. The online help provides field information. In addition you can perform the following user assignments: Delete user When you delete a user, all policies and rules the user created, which are assigned to the user, are automatically assigned to users in the Administrator role. Edit users refer to Edit Users on page 25. Enable and disable users. Manage user roles and permissions refer to Manage Roles on page 24. Manage Users 23

24 User Management Manage Roles All users must be assigned to one user role. Only one role per user is supported.user roles are used to group users with similar permissions and functions. ITCA SE provides four levels of users including: Security Administrator A super user with administrator permissions who can manage users, user roles, user authentication, and view the audit log. (Similar to root in UNIX and Administrator in Windows.) Administrator A super user with full administrator permissions to the entire application, with the exception of User Management. User A regular user with full access to the entire application, but without administrator permissions or the ability to manage user-defined groups. Guest A user with view-only permissions who cannot make modifications or perform administrator tasks. The user role defines the possible permissions a user can have. User permissions can include all user role permissions, or a subset of the permissions, but a user cannot be granted more permissions than those available for a single user role. Table 1 on page 24 lists role privileges for each role. Table 1 ITCA SE role privileges Role privileges Role Security Admin Administrator User Guest User & Role Management Yes No No No Discovery Management Yes Yes No No UDG Management Yes Yes Yes No System Yes Yes No No DB Administration Yes Yes No No Policy Management Yes Yes Yes No Configuration Changes Annotation Yes Yes Yes No Breach Management Yes Yes Yes No View Only Operations Yes Yes Yes Yes To access Manage Roles view a, select Manage Roles from the User Management options in the Administration tab. Assign Role To assign a role to a user: 1. Log in with Security Administrator permissions. 2. Click Administration to display the Administration tab. 3. Under User Management, click Manage Users. 4. In the Manage Users view, if assigning a role to a new user, click Create User. If assigning a role to an existing user, select the user from the list and click Edit. 24 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

25 User Management 5. In the Create/Edit User dialog box, for Role, select the desired user role. 6. Click OK to assign the role to the user. The ITCA SE online help provides full field information and step-by-step procedures for assigning roles and permissions. Edit Users When creating a user, data is required for all fields. When editing user credentials, the Edit User dialog box option from the Admin Edit User menu. This dialog opens with the Password and Re-enter Password fields blank. These fields are not marked as required fields. If you type any characters in either of these fields, the system considers this action as a password change request. If a password change is not required, remove any characters previously entered in the password fields leaving the fields them empty. Proceed with editing any additional user changes, and then click OK to save the changes. Manage Roles 25

26 User Management Manage Authentication The Manage Authentication feature allows you, the Security Administrator to specify how users are authenticated when they log in. You can select either the Lightweight Directory Access Protocol (LDAP), or Local as options for types of authentication. You set or edit user authentication options when either creating or editing a user. Manage LDAP Settings To manage LDAP settings for your application: 1. Select Manage Authentication from the User Management section of the Administrator tab to display the LDAP Authentication options. 2. Enter the following information: Domain Name Type the domain name of the LDAP server. For example, corp.yourcompany.com Server Type the IP address, fully-qualified name, or hostname of the LDAP server. LDAP Port Type the port number of the LDAP server. The default is typically 389. Base Domain Name Type the base domain name of the LDAP server. For example, dc=corp,dc=emc,dc=com. Patterns Enter the full domain controller information for parsing the formatted LDAP entries. For example, uid=jdoe; ou=it; dc=ldap; dc=corp; dc=yourcompany; dc=com. The ITCA SE online help provides full field information and step-by-step procedures for assigning and managing User Authentication. 26 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

27 User Management Change Lockbox Password An additional responsibility for the ITCA SE Admin Security Administrator is to manage the Lockbox password. The ITCA SE Lockbox contains the passwords for all users defined as local, and used for local user. The Lockbox password can only be changed by the Security Administrator. This responsibility makes the Security Administrator responsible for company security policy with regard to server configuration. The Lockbox passcode is used for Lockbox administration such as resetting the Lockbox in case of relevant changes made to the server configuration. The Lockbox verifies that the application trying to access it is running on the same machine. It does this by comparing various configurations of the machine on which the lockbox is running. As a result, an application running on the same machine as the lockbox does not need the passcode to manage users (Security Admin function). Note: The lockbox is created byitca SE during the first run (and verified in consecutive runs). It is not created during setup authentication. To display the Lockbox password dialog, select Change Lockbox Password from the User Management options in the Administration tab. The default lockbox password is: 1Passphrase!! IMPORTANT It is the responsibility of the Security Administrator to change the Lockbox password to avoid security risks. Change the password the first time you access ITCA SE after you assign the Security Administrator user role. Change Lockbox Password 27

28 User Management 28 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

29 3 System Administration This chapter discusses the following topic: System Notifications System Administration 29

30 System Administration System Notifications You can complete the System Notifications dialog box to set up user notification of policy breaches. You can notify users of breaches through or SNMP traps or both. To access the System Notifications dialog, select System Notifications from the System Administration options in the Administration tab. Figure 3 System Notifications dialog The ITCA SE online help provides field descriptions and specific instructions for using this System Notification feature. 30 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

31 4 Manage User-defined Groups This chapter discusses the following topics: User-defined groups Manage User-defined Groups 31

32 Manage User-defined Groups User-defined groups The user-defined group feature provides the ability to create groups. A group is a logical collection of managed objects such as hosts, arrays, switches, and fabrics. Logical grouping of managed objects allows you to perform operations on large sets of like or related managed objects. In addition to system groups, there are two types of user-defined groups: static and dynamic. System groups System groups are groups automatically created by the system to contain all objects of a particular type. For example, All Hosts, All Arrays, All Switches, and All Fabrics are system groups that contain all discovered objects of that type. The All Groups group is also a group created by the system as a container for all groups in the group tree. You cannot delete system groups and can only change the membership of a system group programmatically. Static user-defined groups A static user-defined group can contain managed objects (members) and other static and dynamic user-defined groups (subgroups). A static group changes only when you explicitly add or remove members and subgroups. An object can be a member of more than one static group. Static groups are intended for collections of managed objects that rarely change. Dynamic user-defined groups A dynamic group contains objects that meet user-defined criteria based on object attributes. At regular intervals (either the default interval of 60 minutes or some other user-specified interval), the system scans your Repository for discovered objects that meet the user-defined criteria and automatically adds and removes objects. Subgroups Both static and dynamic groups can be subgroups of a static group. However, a group cannot be a subgroup of two parent groups. A subgroup can only have one parent. User-Defined Groups view The User-Defined Groups view provides information and options that allow you to manage static and dynamic user-defined groups. The panel on the left displays all system and user-defined groups in the system. When you select a group in the panel, all objects and subgroups in that group appear in the table on the right. Additionally, Users can create and implement user-defined groups in policy creation. Only Admin users can access and manage user-defined groups from the Administration tab of the console. For complete information on user-defined groups, refer to the EMC IT Compliance Analyzer Storage Edition User Guide. 32 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

33 5 Discovery This chapter discusses the following topics: Overview Manage access profiles Manage initial discovery policies Manage rediscovery policies Configure host resolution Configure non-discovered objects Manage object inventory Manage Discovery Manager status Discovery 33

34 Discovery Overview The Discovery functions of EMC IT Compliance Analyzer Storage Edition include managing and creating: Access profiles Initial discovery policies Rediscovery policies Host resolution configuration Object inventory This chapter provides an overview of ITCA SE discovery. The ITCA SE online help provides field descriptions and specific instructions for using these management features. 34 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

35 Discovery Manage access profiles This section provides an overview of ITCA SE access profile management features. An access profile is a set of user-specified properties that define the way in which an application makes a connection to objects in the network through a management interface. The access profile specifies the protocol and the required credentials for connecting to the management interface. For example, an SSH access profile contains the port number, username, and password for connecting to a secure shell (SSH) server. To open Access Profiles view, Figure 4 on page 35, select Manage Access Profiles from the Discovery options of the Administration tab. Figure 4 Access Profiles view One use of an access profile is to associate the profile with an initial discovery policy. When the initial discovery policy runs, the protocol and credentials in the access profile are used to initiate communication with the specified object types at the IP addresses specified in the initial discovery policy. You can designate any access profile as a global profile. A global access profile can be used with any initial discovery policy that does not specify specific access profiles. This means that when running initial discovery policies that use global profiles, the system tries each global profile until it finds one that is appropriate for the object specified in the initial discovery policy. The Discovered Objects tab lists all discovered objects and user-defined groups associated with the selected access profile. Manage access profiles 35

36 Discovery Access Profile view contains a list of the access profiles created in the system, and provides the following information and options for creating and managing access profiles. Options include: Profile Name Profile Type Global Profile Description Create Profile Edit Profile Delete Profile Copy Profile Profile types When you click Create Access profile, a profile type popup dialog box appears. Select the type of access profile you want to create. Profile types include: HiCommand SMI SNMP V1/V2 SNMP V3 SSH WMI The ITCA SE online help provides complete information and field definitions. 36 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

37 Discovery Manage initial discovery policies Initial discovery is the process by which the objects in your environment are detected. An initial discovery policy specifies the IP addresses you want the system to scan to detect objects that are not yet in the Repository. Creating initial discovery policies! Before you can create an initial discovery policy and discover objects, you must create at least one access profile. A discovered object is associated with the access profile used to discover it. Other management services, such as provisioning, use the object's associated access profile. You can designate any access profile as a global profile. A global access profile can be used with any initial discovery policy that does not specify specific access profiles. This means that when an initial discovery policy that uses global profiles runs, the system tries each global profile until it finds one that is appropriate for the objects specified in the initial discovery policy. As Admin you can perform the following tasks for discovery policies: create, delete, edit, enable/disable, copy, delete, manually run, view discovered objects, and view policy history details. The Create Initial Discovery Policy wizard includes the following steps. Policy Information Enter IP Addresses Set Schedule Select Access Profiles IMPORTANT Do not run an initial discovery policy for the first time the night of a time change to or from Daylight Savings Time. Discovery policy execution results This dialog provides detailed information about a specific execution of a specific initial discovery policy. The table contains one row for each object discovered when the policy was run. Information includes: Run Date Result Message IP Address Access Profile Object Name Object Type Status Message Manage initial discovery policies 37

38 Discovery Discovery view Once you create initial discovery polices, Discovery view contains a list of these policies, and provides users the following information and options for creating and managing them: Information: Name Status Last Started Next Scheduled Last Modified Schedule enabled Description Commands: Create Policy Edit Run Policy Enable Disable Delete You access the Manage Initial Discovery Policies view, from the Discovery options on the Administration tab. The ITCA SE online help provides field descriptions and specific instructions for using the Manage Initial Discovery Policies features. 38 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

39 Discovery Manage rediscovery policies Rediscovery is the process by which the information about objects in the Repository is updated. Rediscovery policies specify what to rediscover (the policy scope) as well as the conditions under which you want the application to rediscover managed objects. Only discovered objects can be rediscovered. The policy scope consists of user specified user-defined groups. When the rediscovery policy runs, each managed object in the specified user-defined groups is evaluated to see if rediscovery of that managed object is necessary. The Administrator function of the ITCA SE allows you to: Add a user-defined group to a rediscovery policy Create a rediscovery policy Delete a rediscovery policy Edit a rediscovery policy Enable and disable a rediscovery policy Manually rediscover specific objects Remove a user-defined group from a rediscovery policy! IMPORTANT Do not run rediscovery on the day or evening that Daylight Savings Time occurs. This includes running discovery when time changes to or from Daylight Savings Time. Select Manage Rediscovery from the Discovery options of the Administration tab to display the Manage Rediscovery dialog box. The ITCA SE online help provides field descriptions and specific instructions for using this management features. Manage rediscovery policies 39

40 Discovery Configure host resolution Host resolution is a feature that applies only to hosts that have not been discovered. It creates the instances of the hosts and their associated HBA port information based on zoning information. During host zoning, this feature maintains a naming convention in which the zone name contains the hostname. Host resolution uses templates to parse for the hostname. Zone members are checked for host ports, and the system creates the host with the specified ports. By default, the system attempts to use the fully qualified hostname and its IP addresses when creating the host. Hostnames that do not resolve through DNS are by default not created. Host Resolution configuration parameters Host resolution can be configured using the parameters in Table 2 on page 40. Table 2 Host resolution parameters Parameter Description Default value Comments tes.hostresolutionisenabled Enables and disables the feature. If set to false, no host will be created based on host resolution of hostnames based on zoning. true tes.validatehostwithdns Specifies that TES should try to get the host IP address using DNS resolvers. true If set to "false," host resolution is disabled. tes.createhostonlyifdnsresolve s Specifies that the host is created only if DNS resolves IP address of the host. true tes.zonenametemplates The templates for deducing the hostnames from a given zone name. N/A The consumer must configure this value to suit their environment. tes.allowedvendorcodes Specifies which WWN vendor codes can be considered host ports. 0000C9 000A33 000E B 0090FA 00C0DD 00E08B 001B32 The consumer is not expected to change the default, unless instructed otherwise. To set these parameters, log in to ITCA SE as admin and edit the following file in this location: /opt/dmt/local/conf/storage-path-query-service/ tes.properties After making the changes, stop and restart the discovery manager as follows: 1. Stop the Discovery Manager, type: sm_service stop SAN_server 2. Start the Discovery Manager, type: sm_service start SAN_server This forces the appliance to refresh the topology using the new configuration. Configuring the tes.zonenametemplates parameter Zonename templates specify where to extract the hostname from the zonename. The tes.properties file is delivered with the following value for the tes.zonenametemplates parameter: tes.zonenametemplates = z_%h% z_%h%_* %h%_* z_%h%_*_* z_%h%_*_*_* 40 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

41 Discovery Example 1 Example 2 You must configure this parameter for your environment. Edit the value for the tes.zonenametemplates parameter using the following examples as a guideline: %h% Specifies where to extract the hostname from the zone name. * This wildcard matches any string, up to a delimiter. tes.zonenametemplates = Zone_%h%_array101_port1 Zone_server1_array101_port1 matches the template and "server1" is extracted as the hostname. Zone_server2_array101_port2 does not match the template because "port2" is part of the zone name but not part of the template. No hostname is extracted. tes.zonenametemplates = Zone_%h%_array101_* Zone_server1_array101_port1 matches the template and "server1" is extracted as the hostname. Zone_server2_array101_port2 matches the template and "server2" is extracted as the hostname. Note that in this example the template specifies a wildcard character as the last element of the template. Multiple templates can be specified. Each template is separated by a pipebar ( ) character. Each template is tried until the first successful match; the hostname is then extracted based on this match. Note: Host resolution and TES configured hosts are identified in the ITCA SE Console with separate icons to indicate non-discovered hosts. Configure host resolution 41

42 Discovery Configure non-discovered objects You can manage non-discovered hosts by creating a host for discovery, and defining it by name, IP address and operating system using the Configure non-discovered objects option in the Administrator menu on the console. After defining the host you can select paths that you want to associate with the host such as: WWN Fabric Connects to Switch Connects to Slot Connects to Port The online help shows step-by-step procedures for creating a user-defined host and defining the host with the Create Host wizard. 42 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

43 Discovery Manage object inventory The Manage Object Inventory view allows you to view and manage discovered objects. A tab in the Manage Object Inventory view appears for the following: Hosts Fabrics Switches Arrays You can then select any object from one of the object tabs and display the object properties in the object details dialog. In addition you can select to rediscover the object from the view options. You access the Manage Object Inventory view by selecting it from the Discovery option of the Administration tab. The ITCA SE online help provides field descriptions and specific instructions for using these management features.! IMPORTANT With this release of ITCA SE, you must refresh Object Details manually. Manage object inventory 43

44 Discovery Manage Discovery Manager status The Discovery Manager Status dialog allows you to check the health (status) of Discovery Managers. Discovery Managers are responsible for discovery of: SANs Discovers fabrics, switches and zones. Hosts Discovers hosts and HBAs. Paths Discovers paths. Arrays Discovers arrays, volumes, and LUNs. You can reach the Discovery Manager Status dialog box by clicking Discovery Managers Status in the Administration view, Discovery menu. The status is updated manually so you must click Check on the dialog box to get an updated view. Note: The first time you access the dialog, the update is performed before the dialog is displayed. A green status indicates that the Discovery Manager is connected. A yellow triangle indicates that the Discovery Manager not connected. If the Discovery Manager is not connected, Troubleshooting Appendix A, on page 63 offers helpful information. 44 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

45 6 Database Administration This chapter describes the necessary steps for maintaining the EMC IT Compliance Analyzer Storage Edition Sybase database. Topics include: Database Administration overview Back-up repositories Restore repositories Retention policy Database Administration 45

46 Database Administration Database Administration overview ITCA SE uses two object repositories: Discovered Objects repository The discovered objects in ITCA SEare saved to a set of files. Sybase database Compliance policies, configuration changes, breaches, and general product configuration information are stored in the Sybase database. The Database Administration option of the ITCA SE Administration tab allows you to manage the Sybase database, including options to: Back-up Repository back-up the Sybase repository database, including scheduling routine back-ups of the database, for archiving and recovery. The database is in a Sybase file format. Manage Historical Data Retention Purge or delete policy breaches and configuration changes. To back-up the Discovered Objects repository, you must perform a manual procedure from the API. To restore either repository, you must perform a manual procedure from the API as shown in Restore repositories on page 49. This chapter provides an overview of database administration features. The EMC IT Compliance Analyzer Storage Edition online help provides field descriptions and specific instructions for using these administrative features. 46 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

47 Database Administration Back-up repositories Sybase back-up Use the Back-up Repository dialog box as shown in Figure 5 on page 47 to back-up and schedule routine back-ups of the Sybase database that contains: Compliance and configuration change data. Lockbox encrypted user passwords You can use these back-ups for archiving or disaster recovery. Back-up now Select to execute the back-up immediately. Back-up later Select to execute the back-up on a future date. Select the day of the week and time for a single back-up or to execute back-ups on a recurring schedule. Figure 5 Back-up Repository dialog box To access the Back-up Repository dialog, select Back-up Repository from the Manage Database option of the ITCA SE Administration tab. Manually back-up discovered objects, Sybase and Lockbox repositories You can also manually back-up the Discovered Objects and Sybase repository. This action is necessary if, for example, you need to replace the ITCA SE hardware appliance, or replace a hard drive or power supply. (Refer to the EMC IT Compliance Analyzer Storage Edition Hardware Replacement Guide for instructions.) To back-up the Discovered Objects and Sybase repository, follow these steps: 1. Log in to the appliance through SSH using root user. 2. Stop ITCASE services, type: /opt/dmt/local/conf/sdastopall.sh 3. Stop the Sybase service, type: service sybase stop Note: Stopping all services can take from 15 to 25 minutes. 4. From a remote machine, SCP to the system using root user. 5. Copy existing /opt/dmt/local/repos/icf/*.rps files to external location. Back-up repositories 47

48 Database Administration 6. Copy /opt/sqlanywhere10/itcase.db and /opt/sqlanywhere10/itcase.log files to external location. 7. Copy the following files from the /opt/cst/xml/ directory: csp.clb csp.clb.bak LocalDirectoryData.xml 48 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

49 Database Administration Restore repositories You cannot restore repositories with the user interface, you must use the API. Restore only Sybase repository To restore just the Sybase repository, follow these steps: 1. Login to appliance though SSH as Admin user. 2. Stop Tomcat service, type: service tomcat stop. 3. Stop Sybase service, type: service sybase stop. 4. back-up existing /opt/sqlanywhere10/itcase.db and /opt/sqlanywhere10/itcase.log 5. Copy new ITCA SE database, and itcase.log files to /opt/sqlanywhere10. Overwrite existing files is necessary. 6. Start Sybase service, type: service sybase start. 7. Start Tomcat service, type: service tomcat start. Restore all repositories To restore both the Sybase, Discovered Objects and Lockbox repositories follow these steps: 1. Log in to the appliance through SSH using root user. 2. Stop ITCASE services, type: /opt/dmt/local/conf/sdastopall.sh 3. Stop Sybase service, type: service sybase stop 4. back-up existing /opt/dmt/local/repo/icf/*.rps files 5. Delete all *.rps file under /opt/dmt/local/repo/icf directory 6. Copy previously backed-up ITCA-SE discovered object repository files (*.rps files) to /opt/dmt/local/repo/icf 7. Copy the files: csp.clb, csp.clb.bak, LocalDirectoryData.xml to the CST config folder. 8. Run the following: cstadmin sign-file csp.clb -passphrase=<passphrase> -cstdir=<cst config dir> cstadmin sign-file csp.clb.bak -passphrase=<passphrase> -cstdir=<cst config dir> cstadmin sign-file "*.xml" -passphrase=<passphrase> -cstdir=<cst config dir> 9. Back-up /opt/sqlanywhere10/itcase.db and /opt/sqlanywhere10/itcase.log files. 10. Copy previously backed-up ITCA-SE database, and itcasse.log files to /opt/sqlanywhere10. Overwriting existing files is necessary 11. Start Sybase service, type: service sybase start 12. Start ITCASE services, type: /opt/dmt/local/conf/sdastartall.sh Note: Rebooting the appliance after replacing a defective appliance restarts all services. Restore repositories 49

50 Database Administration Retention policy Managing data retention allows you to purge or resolve policy breaches and configuration changes after a period of time. To manage data retention, select Retention policy from the Database Administration option of the Administration tab to display the dialog box as shown in Figure 6 on page 50. Select the checkbox next to each retention setting you want to enable. Then, type the number of weeks or months to retain the policy breaches or configuration changes before they are purged. Specify the day and time to run the retention policy. Figure 6 Historical data retention policy dialog 50 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

51 7 Install Providers This chapter discusses the following topic: Overview EMC SMI-S Provider for Symmetrix and CLARiiON Adding CLARiiONs to the EMC SMI-S Provider for discovery Brocade SMI Agent HiCommand for Hitachi arrays Install Providers 51

52 Install Providers Overview The ITCA SE discovery use the following agents and providers to discover the SAN: Brocade/McDATA SMI Provider EMC SMI-S Provider for Symmetrix and CLARiiON (Solutions Enabler) EMC Host Proxy HiCommand for HDS arrays (available starting with ITCA SE version 1.1 Update Bundle 1) It is very important that these data providers are properly installed and configured for discovery to function properly. Ensure that the provider versions match the versions qualified by ITCA SE. The providers are separate products and they change rapidly. It is very essential that the correct version is installed for discovery to work. Note: Refer to the EMC IT Compliance Analyzer Storage Edition Support Matrix for information on supported devices and provider versions. Also, you should obtain installation documentation from the provider manufacturer as needed. 52 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

53 Install Providers EMC SMI-S Provider for Symmetrix and CLARiiON To discover Symmetrix, you must install the SMI-S Provider for Symmetrix and CLARiiON (with Solutions Enabler kit) on a host connected to the Symmetrix. Note: SE is an ECOM-based provider rather than a Pegasus-based provider. As a result several things have been deleted or moved around. Several things in the symcli/storbin directory (ex. TestSmiProvider) have been moved to the ecim/ecom/bin directory. There is no cimserver daemon any more. Default Provider Username: LocalDir/admin Default Provider Password: #1Password Windows installation The Windows installer with the correct build of Solutions Enabler is available on Powerlink. To install EMC SMI-SProvider with the Solutions Enabler kit: 1. Uninstall previous versions of Solutions Enabler by navigating to: Start > Control Panel > Add/Remove Programs 2. Select EMC Solutions Enabler and remove it 3. Delete following directories: C:\Program Files\EMC\SYMAPI C:\Program Files\EMC\SYMCLI 4. Download the Windows installer from Powerlink. 5. Double-click the <filename_windows-x86-smi>.msi file. Example: double-click the se65132-windows-x86-smi.msi file. 6. After the installation is complete, create the following file: <SE directory>\symcli\storbin\emcprovider_user.conf 7. Add the following line to the file in step 6 and save it: OslProv/com.emc.se.osls.osl.StatisticsService.enable = false 8. Restart the cimserver: a. Stop the CIM Server using following commands: For 6.5 SMI provider: From the SYMCLI/storbin directory: cimserver -stop EMC_SMI_Provider For 7.0 SMI provider: From the ECIM/ECOM/bin directory: sm service stop ecom.exe From the ECIM/Slp/lib directory: slpd -stop b. Restart the CIM Server using the following commands EMC SMI-S Provider for Symmetrix and CLARiiON 53

54 Install Providers For 6.5 SMI provider: From the SYMCLI/storbin directory: cimserver -start EMC_SMI_Provider For 7.0 SMI provider: From the ECIM/ECOM/bin directory: sm service start ecom.exe From the ECIM/Slp/lib directory: slpd -start Linux installation The Linux version of the Symmetrix SMI-S Provider with Solutions Enabler is available on Powerlink. Follow these steps for installation: 1. Download the Linux installer with the correct build of Solutions Enabler from Powerlink. Example: download the se65132-linux-i686-smi.tar.z file from Powerlink. 2. Untar the installer by typing the following command: gunzip <filename>.tar.z 3. Extract the installer by typing the following command: tar -xvf <filename>.tar 4. Execute the installation script by typing the following command:./<filename_install>.sh -install Example:./se650294_install.sh -install 5. After the installation is complete, create the emcprovider_user.conf file. For example: /opt/symcli/storbin/emcprovider_user.conf for SE or /opt/emc/symcli/v7.0.0/storbin/emcprovider_user.conf for SE Add the following line to the file in step 5 and save it: OslProv/com.emc.se.osls.osl.StatisticsService.enable = false 7. Restart the cimserver: a. Stop the CIM Server using the following commands: For 6.5 SMI provider: From the SYMCLI directory:./storbin cimserver -s 54 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

55 Install Providers For 7.0 SMI provider: From the ECIM/ECOM/bin directory: 1. Find the PID of the running ECOM daemon: ps -ef grep "ECOM -d" 2. Kill the ECOM daemon: kill -s TERM <ecom-pid> 3. Find the PID of the running SLP daemone: ps -ef grep slpd 4. Kill the SLP daemon: kill -s TERM <slpd-pid> b. Restart the CIM Server using the following command from the SYMCLI directory: For 6.5 SMI provider: From the SYMCLI directory:./storbin/cimserver For 7.0 SMI provider: From the ECIM/ECOM/bin directory:./ecom -d From the ECIM/slp/lib directory: slpd EMC SMI-S Provider for Symmetrix and CLARiiON 55

56 Install Providers Adding CLARiiONs to the EMC SMI-S Provider for discovery In-band CLARiONs In-band arrays have a direct connection the provider s host system via a Fibre Channel or a SCSI port. Out-of-band arrays are not connected to the host system via a Fibre Channel or SCSI port. Generally, the out-of-band systems have a network connection to the host system. In-band CLARiiON arrays are discovered automatically by the EMC SMI Provider if all of the Service Processor credentials have been supplied using the "symcfg auth" command:. Windows: cd C:\Program Files\EMC\SYMCLI\bin UNIX: cd /usr/symcli/bin symcfg auth add -host <SPA Address> -username <username> -password <password> symcfg auth add -host <SPB Address> -username <username> -password <password> To verify what you added, type: symcfg auth list Using the TestSMiProvider application to add Symmetrix and CLARiiON arrays to the EMC SMI-S Provider Example: Starting the TestSMiProvider application and supplying connection details In-band and out-of-band Symmetrix and CLARiiON arrays may also be added to the EMC SMI-S Provider using the TestSmiProvider application via the "addsys" command. The TestSmiProvider application is a command-line based CIM Client that can be used to invoke generic CIM operations as well as some special operations supported only by the EMC SMI-S Provider. For example, the TestSmiProvider provides convenience methods for invoking EMC_SystemRegistrationService operations used to add, remove, discover, and refresh EMC arrays contained in the EMC SMI Provider database. Start the TestSmiProvider application by going to the SYMCLI\storbin directory and entering "TestSmiProvider". For example: The TestSmiProvider for SE is located at symcli/storbin/ and the TestSmiProvider for SE is located at ecim/ecom/bin/testsmiprovider. Connect to a running EMC SMI-S Provider by entering the requested information when prompted (defaults are listed in brackets and may be accepted by pressing ENTER). The following is an example of how to start the TestSMiProvider application. C:\Program Files\EMC\SYMCLI\storbin>TestSmiProvider.exe Host [localhost]: Connection Type (ssl,no_ssl) [no_ssl]: Logfile path [Testsmiprovider.log]: Port [5988]: Username []: Password []: Connecting to localhost: EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

57 Install Providers ######################################################################## ## ## ## EMC SMI Provider Tester ## ## This program is unsupported and intended for use by EMC Support ## ## personnel only. At any time and without warning this program may ## ## be revised without regard to backwards compatibility or be ## ## removed entirely from the Solutions Enabler kit. ## ######################################################################## slp - slp urls slpv - slp attributes cn - Connect dc - Disconnect rc - RepeatCount st - Set timeout value ns - NameSpace ens - EnumerateNamespaces ec - EnumerateClasses ecn - EnumerateClassNames ei - EnumerateInstances ein - EnumerateInstanceNames a - Associators an - AssociatorNames r - References rn - ReferenceNames gi - GetInstance gc - GetClass lc - Log control dv - Display version info addsys - EMC AddSystem refsys - EMC RefreshSystem disco - EMC Discover remsys - EMC RemoveSystem im - InvokeMethod tms - TotalManagedSpace q - Quit h - Help ##################################################################### Example: disco command Example: addsys command! To force the EMC SMI Provider to discover all in-band arrays known to it, use the TestSmiProvider "disco" command: (EMCProvider:5988)? disco ++++ EMCDiscoverSystem ++++ Output : 0 Legend:0=Success, 1=Not Supported, 2=Unknown, 3=Timeout, 4=Failed 5=Invalid Parameter 4096=Job Queued, 4097=Size Not Supported To add in-band or out-of-band arrays to the EMC SMI Provider, use the "addsys" command and supply information as requested by the prompts. In-band arrays may be specified by array name. Out-of-band arrays may be specified by host name or IP address. IMPORTANT All CLARiiON Service Processors must be listed in order for example, SP_A followed by SP_B in the input file. Failure to specify all service processors may create array discovery problems. (EMCProvider:1234)? addsys Add System {y n} [n]: y ArrayType (1=Clar, 2=Symm) [1]: One or more IP address or Hostname or Array ID Elements for Addresses IP address or hostname or array id 0 (blank to quit): IP address or hostname or array id 1 (blank to quit): IP address or hostname or array id 2 (blank to quit): Address types corresponding to addresses specified above. (1=URL, 2=IP/Nodename, 3=Array ID) Adding CLARiiONs to the EMC SMI-S Provider for discovery 57

58 Install Providers Address Type (0) [default=2]: Address Type (1) [default=2]: User [null]: xyz Password [null]: ********** ++++ EMCAddSystem ++++ Output : 0 Legend:0=Success, 1=Not Supported, 2=Unknown, 3=Timeout, 4=Failed 5=Invalid Parameter 4096=Job Queued, 4097=Size Not Supported System : Clar_StorageSystem.CreationClassName="Clar_StorageSystem",Name="CLARi ion+apm " Example: ein and remsys commands To remove an array you must first get the array's instance name using the "ein" (enumerate instances) command and specify the "EMC_StorageSystem". Then use the "remsys" command and provide the array's instance name. (localhost:5988)? ein Class: EMC_StorageSystem ++Testing EnumerationInstanceNames: EMC_StorageSystem++ Instance 0: Symm_StorageSystem.CreationClassName="Symm_StorageSystem",Name="SYMME TRIX " Enumerate 1 instance names; repeat count 1;return data in seconds Retrieve and Display data - 1 Iteration(s) In Seconds Please press enter key to continue... repeat count: 1 (localhost:5988)? remsys Remove System {y n} [n]: y System's ObjectPath[null]: Symm_StorageSystem.CreationClassName="Symm_StorageSystem",Name="SYMME TRIX " About to delete system Symm_StorageSystem.CreationClassName="Symm_StorageSystem",Name=="SYMM ETRIX " Are you sure {y/n} [n]: Example: refsys command To refresh the EMC SMI Provider's view of an array with the actual array hardware, use the TestSmiProvider "refsys" command: (EMCProvider:1234)? refsys Refresh Systems {y n} [n]: y Refresh all systems {y n} [n]: One or more System's ObjectPath. System 0 (blank to quit): Clar_StorageSystem.CreationClassName="Clar_StorageSystem",Name="CLARi i ON+ APM " System 1 (blank to quit): ++++ EMCRefreshSystem ++++ Output : 0 Legend:0=Success, 1=Not Supported, 2=Unknown, 3=Timeout, 4=Failed 5=Invalid Parameter 4096=Job Queued, 4097=Size Not Supported 58 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

59 Install Providers Brocade SMI Agent To discover Brocade switches, you must install and run SMI providers on a host that manages those switches. Refer to the Brocade website for installation documentation and provider information. Brocade SMI Agent 59

60 Install Providers Brocade SMI Agent for EOS (McDATA) To discover McData switches, you must install and run SMI providers on a host that manages those switches. Refer to the Brocade website for installation documentation and provider information. 60 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

61 Install Providers HiCommand for Hitachi arrays Note: Hitachi array support is available starting with ITCA SE version 1.1 Update Bundle 1. ITCA SE supports Hitachi arrays that are supported by the HiCommand Device Manager. To discover Hitachi arrays you must install and configure the HiCommand Device Manager. The Hitachi HiCommand Device Manager Server XML API provides discovery of Hitachi arrays, however it does not support array health and operational status updates or alerts. Refer to the Hitachi website for HiCommand installation and configuration documentation. HiCommand for Hitachi arrays 61

62 Install Providers 62 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

63 A Troubleshooting This appendix explains several methods of troubleshooting issues with EMC IT Compliance Analyzer Storage Edition. Topics include: Overview Brocade and McDATA provider SMI-S Provider for Symmetrix and CLARiiON HiCommand for HDS arrays Troubleshooting Host discovery General SMI issues Troubleshooting the gdpm DM Troubleshooting an offline array Domain Manager Using SNMP Useful commands and utilities Using discovery logs Rediscovery scenarios for user-defined groups Re-imaging the appliance Troubleshooting 63

64 Troubleshooting Overview This Appendix provides additional information for best practices and troubleshooting an ITCA SE implementation. Review the EMC IT Compliance Analyzer Storage Edition Release Notes for information on known Issues and Limitations, and the EMC IT Compliance Analyzer Storage Edition Support Matrix for complete support data for this release. Refer also to the EMC IT Compliance Analyzer Storage Edition User Guide, available from the ITCA SE online help or from the EMC Powerlink website.! IMPORTANT If you review this document and still need assistance troubleshooting your application, contact your EMC Support Representative to check relevant log files. 64 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

65 Troubleshooting General provider configuration issues Follow these guidelines when configuring providers and troubleshooting provider configuration issues: The ITCA SE discovery services use the following agents and providers to discover the SAN: Brocade/McDATA SMI Provider EMC SMI-S Provider for Symmetrix and CLARiiON (Solutions Enabler) EMC Host Proxy It is very important that these data providers are properly installed and configured for discovery to function properly. Ensure that the provider versions match the versions qualified by ITCA SE. The providers are separate products and they change rapidly. It is very essential that the correct version is installed for discovery to work. Note: Refer to the EMC IT Compliance Analyzer Storage Edition Support Matrix for information on supported devices and provider versions. Learn how to enable logging on providers. (Refer to manufacturer s documentation.) Have knowledge of starting and stopping provider services. (Refer to Useful commands and utilities on page 89.) Keep a list of provider known issues at hand. (Refer to this Troubleshooting section and manufacturer s provider documentation.) Brocade providers only need a single proxy switch to manage a large fabric. Since ITCA SE discovery management is lost if the proxy becomes unavailable, select a high availability director class switch. To receive events from Cisco, add the IP address of the ITCA SE appliance to trap recipient table on Cisco. For all other providers, the appliance automatically subscribes to events following a discovery. Use the EMC IT Compliance Analyzer Storage Edition Performance and Scalability Guidelines to allocate managed elements per providers. There are no restrictions for host or Cisco. Host discovery requires Host-HBA, drivers, and firmware listed in the EMC Support Matrix. Use the E-Lab Interoperability Navigator tool, available through this URL: Discovery issues typically arise when new models, software, and firmware are introduced to the discovery environment. Identifying where the issue lies, with ITCA SE or provider and managed element is key to troubleshooting. General provider configuration issuesoverview 65

66 Troubleshooting Brocade and McDATA provider This section describes issues that may arise with SAN discovery when Brocade and McDATA providers are used. McDATA guides A McDATA provider troubleshooting guide is available on their corporate website. No elements detected with Brocade and McDATA CIMOMs Follow these steps: 1. Verify that instances of root/brocade1:brocade_switch or root/mcdata:mcdata_switch exist in the CIMOM using a CIM/XML client such as cimcli. If there are no errors but no instances are returned for one of these, then it is possible that the CIMOM was not configured correctly or it cannot connect to its proxy or proxies. Both CIMOMs have configuration GUIs that you can use to verify this. The Brocade configuration GUI lists all proxy switches used and whether or not it is currently connected to each. The McDATA GUI tests the credentials used for EFCM as listed under Test Connection under Management Platform > Update Login ID Password 2. Verify that the correct CIMOM is installed on the expected IP. If the detection probe can connect to the CIMOM but it is managing arrays instead of switches, it returns No Elements Detected without failures or errors in the probe log. 3. If the SMI Access Profile does not have SSL enabled, make sure that the provider is configured with HTTP access enabled. The default setting for the McDATA provider is to have HTTP disabled and only HTTPS (SSL) enabled. Alternatively, modify the SMI Access Profile to enable SSL and set the port number accordingly (default port for HTTPS is 5989). 4. Verify that SSL certificate authentication is disabled. In the Brocade SMI Agent disable the "Mutual Authentication (Clients)" and "Mutual Authentication (Indications)" features. Brocade provider not sending events The Brocade provider may not send events if either the "Mutual Authentication (Client)" or "Mutual Authentication (Indications)" are enabled. A workaround for this issue is to verify that the "Mutual Authentication (Client)" and "Mutual Authentication (Indications)" features are disabled. If either of them is enabled, disable it using Brocade's configuration tool. After restarting the provider, run Brocade's SubscriptionCleanup tool (<install>\agent\server\jserver\bin\subscriptioncleanup.bat on Windows) to remove any failed subscription instances. Finally, run a Discovery Policy or Rediscovery Policy using that provider. 66 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

67 Troubleshooting Note: The SubscriptionCleanup tool is not included with Brocade SMI Agent for EOS (McData) version 2.5; if needed you can use SubscriptionCleanup tool included with Brocade SMI Agent for FOS (Brocade) version a Brocade provider stops sending events The Brocade provider may stop sending events due to a large number of event subscriptions. A workaround for this issue is to schedule rediscoveries periodically and disable indication subscriptions. To disable indication subscriptions: 1. Open the following file: /opt/dmt/local/conf/cmp-fabric/fabric-smi.import 2. Change TRUE to FALSE in the following line: parameters += {"INDICATIONS_SUBSCRIBE", "TRUE"} 3. Restart the ITCA SE. Brocade provider consuming memory and CPU Sometimes the Brocade provider consumes large amounts of memory and CPU. To determine CPU and memory consumption: 1. Start Windows Task Manager 2. Look for the following provider process with several hundred threads active and large amounts of memory allocated to it: jserverd.exe 3. If this is occurring, restart the provider to recover from this state. A workaround for this issue is to avoid indication subscriptions. If indication subscriptions are enabled, disable them by following the procedure in Brocade provider not sending events on page 66. Error message: Operation Not Supported While adding a proxy switch to the Brocade SMI Agent on a host that has multiple IP addresses, you may notice an error "Operation Not Supported". A workaround for this issue is to modify the file <SMIAgent>/agent/server/jserver/bin/SMIAgentConfig.xml as described in the SMI Agent User Guide under "Brocade SMI Agent", "Enable multi-homed support". General provider configuration issuesbrocade and McDATA provider 67

68 Troubleshooting SMI-S Provider for Symmetrix and CLARiiON Provider documentation contains information about installation, starting, stopping, and configuring the SMI provider. EMC SMI-S Provider for Symmetrix and CLARiiON on page 53 provides information on the installation for SMI-S Provider for Symmetrix and CLARiiON. Provider versions Array discovery can fail or there may be partial discovery if the provider version is not qualified. Identifying SMI-S Provider version numbers To verify the versions of the SE kit and the SMI-S provider do the following: 1. Start symcli/storbin/testsmiprovider(.exe) 2. At the command prompt, type the following: dv This command displays version information for the Pegasus CIMOM, SMI-S Provider, Solutions Enabler, and all the attached Symmetrix and CLARiiON systems. Logging troubleshooting data To log additional troubleshooting information for the array discovery, enable debug mode: 1. Open the following file: /opt/dmt/local/conf/cmp-storage-arrays/array.import 2. Set the DEBUG parameter to TRUE as in the following example: CMP_ConfigurationParameters::Array-ConfigurationParameters { parameters += {"DEBUG", "TRUE"} } 3. Restart the DM after updating the parameter. Disabling statistical information gathering The EMC SMI-S Provider collects statistical information from the arrays every 5 minutes. This information is not used by ITCA SE at this time. Disable this collection if there are no other applications that utilize the statistical information. 68 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

69 Troubleshooting Solutions Enabler configuration error regarding RA Directors A Solutions Enabler configuration error as in Example 1 may appear in the Array DM log file regarding RA Directors. Sometimes this can be addressed by cleaning up the Solutions Enabler database and restarting it. Example 1 Error for RA Director from array1.log file =========================================== WBEM-EUNKNOWNERROR-Unknown error occurred: C:ERROR_CLASS_SOFTWARE F:ERROR_FAMILY_FAILED R:1 R: L: : "The load of a remote storage array configuration failed" : "Unable to initialize Mask Session for specified handle" : 2 : 152 : "Could not find any valid (online and connected) RA Director to use for remote call" =========================================== If this occurs, try cleaning the Solutions Enabler repository. To clean Solutions Enabler on a Windows host: 1. Stop the cimserver by entering the following commands: cd C:\Program Files\EMC\SYMCLI\storbin cimserver stop EMC_SMI_Provider 2. Delete the following file by entering the command: del C:\Program Files\EMC\SYMAPI\db\symapi_db.bin 3. Restart the cimserver by entering the following commands: cd C:\Program Files\EMC\SYMCLI\storbin cimserver start EMC_SMI_Provider To clean Solutions Enabler on a UNIX host: 1. Stop the cimserver by entering the following commands: cd /usr/symcli./storbin/cimserver s 2. Delete the following file by entering the command: rm /var/symapi/db/symapi_db.bin 3. Restart the cimserver by entering the following commands: cd /usr/symcli./storbin/cimserver Discovery errors related to timing out The provider does not discover the arrays until it receives the first client request. When a large number of arrays are managed by a single SMI-S provider instance, the initial discovery may take several minutes and may sometimes result in time-outs during detection. You may see the following errors in the logs: WBEM-EUNKNOWNERROR-Unknown error occured: C:ERROR_CLASS_SOFTWARE General provider configuration issuessmi-s Provider for Symmetrix and CLARiiON 69

70 Troubleshooting F:ERROR_FAMILY_FAILED R:1 connection timed out In order to avoid this, do the following after starting the provider for the first time: 1. Start symcli/storbin/testsmiprovider(.exe) 2. At the command prompt, type the following: ein 3. When prompted for the Classname, force the provider to discover the arrays by typing the following: EMC_StorageSystem Some array discovery time-out issues may be related to network latency (such as discovering over a WAN). If this is the case, then the array DM has a configuration parameter that can be adjusted to work around the latency. An example of an array discovery time-out during the discovery of storage volumes over a WAN: [25-Jun :42:47 AM+659ms UTC] t@ Discovery #2 ASL-W-ERROR_RULE_SOURCE-While executing rule set '/opt/dmt//local/rules/cmp-storage-rules/array/volumes_discovery_smi. asl' ASL-ERROR_ACTION-While executing action at: ASL-CALL_STACK_RULE- RuleName: DISCOVER, Line: 1766 ASL-CALL_STACK_RULE- RuleName: INITIATE_DISCOVERY, Line: 1616 ASL-CALL_STACK_RULE- RuleName: START, Line: 1734 ASL-ERROR_INVOKE-While attempting to invoke operation 'createassociators' of object 'CIMXML_Session::CIMXML_Session access-profile-3-root/ emc' WBEM-ECREATE_ASSOCIATED_FAILED-Create associated instance failed with remote instance root/emc:symm_devicestoragepool%instanceid=symmetrix c and association CIM_AllocatedFromStoragePool WBEM-EUNKNOWNERROR-Unknown error occured: C:ERROR_CLASS_SOFTWARE F:ERROR_FAMILY_FAILED R:1 connection timed out To increase the default time-out period while discovering CIMOMs across a WAN where latency may be an issue, follow this procedure: 1. Edit /opt/dmt/local/conf/cmp-storage-array/array.import 2. Find the following section: CMP_ConfigurationParameters::SMI-ConfigurationParameters { # Timeout values are in seconds parameters += {"DETECTION_TIMEOUT", "900"} parameters += {"DISCOVERY_TIMEOUT", "240"} } 3. Change the value of "DISCOVERY_TIMEOUT" to a larger value. 4. Save the "array.import" file. 5. Restart the Array DM (or entire ITCA SE). 6. Initiate discovery again. 70 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

71 Troubleshooting There may be time-outs because of dropped packets too. In that case, the time-out errors in the log file happen randomly at different places. If the time-outs are consistently in the same place (especially while discovering volumes), as in this case, it may be an indication of network latency related problems. In such cases, increasing the time-out alleviates the problem. However, if you have network issues where packets are dropped randomly, a large time-out value keeps discovery going for a long time because each time there is a problem, the probe waits much longer before moving on. VCMDB devices A VCMDB device should also be visible to the host where the provider is running. Starting the cimserver on Linux Use the full path of the cimserver when starting the provider, such as the following: /usr/symcli/storbin/cimserver Otherwise, a default cimserver that ships with Linux may be started. Database errors in the log An error message such as the following indicates that some changes have been made to the array outside of the CIMOM, and the provider is unable to synchronize its cache with the new state of the array: WBEM-EUNKNOWNERROR-Unknown error occured: C:ERROR_CLASS_SOFTWARE F:ERROR_FAMILY_FAILED R:1 R: : "The internal database is in an inconsistent state, and must be fully synchronized before performing this operation." To rectify this situation: 1. Shut down the cimserver. 2. Shut down all the stordaemons. 3. Delete the following files: /var/symapi/db/*.bin /var/symapi/db/*.xml 4. Restart all services. Connection failed message The following error in the array logs indicate that the CIMOM IP address is bad or the host is down, or the CIMOM is not running: ASL-W-ERROR_RULE_SOURCE-While executing rule set '/opt/dmt//local/rules/cmp-storage-rules/array/array_detection_smi.as l' ASL-ERROR_ACTION-While executing action at: ASL-CALL_STACK_RULE- RuleName: DETECT_CIMOM, Line: 1515 ASL-CALL_STACK_RULE- RuleName: ATTEMPT_ARRAY_DETECT, Line: 2192 ASL-CALL_STACK_RULE- RuleName: START, Line: 1756 General provider configuration issuessmi-s Provider for Symmetrix and CLARiiON 71

72 Troubleshooting ASL-ERROR_INVOKE-While attempting to invoke operation 'enumerateinstancenames' of object 'CIMXML_Session::OSLS-Session-losay ac cessprofile-1' WBEM-EUNKNOWNERROR-Unknown error occured: C:ERROR_CLASS_SOFTWARE F:ERROR_FAMILY_FAILED R:1 Cannot connect to :5988. Connection failed. Error opening gatekeeper device The provider host should be FC connected to the Symmetrix array with a minimum of four gatekeeper devices available for each Symmetrix system connected to the host. If you see the following errors in the logs, it means there are no gatekeeper devices available or that there is a problem with the host connectivity to the gatekeepers: ASL-W-ERROR_RULE_SOURCE-While executing rule set ASL-ERROR_ACTION-While executing action at: ASL-ERROR_INVOKE-While attempting to invoke operation 'createassociators' of WBEM-EUNKNOWNERROR-Unknown error occured: C:ERROR_CLASS_SOFTWARE F:ERROR_FAMILY_FAILED R:1 R: L: : "A gatekeeper open : 2 : 34 : "Error opening the gatekeeper device for communication to the Symmetrix" Array DM errors caused by EMC SMI Provider/Solutions Enabler related issues Entries in the SYMAPI device masking database point to a device that has been deleted. The following error is displayed in the logs: WBEM-ECREATE_REFERENCED_FAILED-Create referenced instance failed with remote instance root/emc:symm_lunmaskingscsiprotocolcontroller%creationclassname=symm _LunM askingscsiprotocolcontroller%deviceid=w e08b0b5c e08b0b5 C31+S YMMETRIX FA-3C+1%SystemCreationClassName=Symm_StoragePro cesso rsystem%systemname=symmetrix fa-3c and association CIM_ProtocolControllerForUnit WBEM-EUNKNOWNERROR-Unknown error occurred: C:ERROR_CLASS_SOFTWARE F:ERROR_FAMILY_FAILED R:1 R: L: : "A device attribute makes its use invalid for the requested operation" : 2 : 4659 : "The operation cannot be completed because one or more devices are unconfigured" In order to view the complete list of bad devices, log on to the provider (symapi\logs) and browse for a file EMCProvider<date> and find Device name = 01F11 as shown below. Thu Feb 05 10:34:45 Eastern Standard Time 2009 losak111 cimserver.exe global WARNING Device.cpp 231 : Received STORAPI return code STOR_C_ARRAY_INV_DEVICE for StorDevShow of array id: '<Array type="symmetrix" encoding="string" value=" " />' using device id: '<Device name="01f11" />'. 72 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

73 Troubleshooting It is possible that the storage array masking database may have devices assigned and configured that are no longer available for use. To overcome this issue: 1. Use "symmaskdb list database" to find the entries that use the device in question symmaskdb list database -wwn C94 186F3 Symmetrix ID : Database Type : Type6 Last updated at : 02:03:18 PM on Thu Sep 27,2007 Director Identification : FA-2A Director Port : 1 User-generated Identifier Type Node Name Port Name Devices c94186f3 Fibre losak c12345f C:003F F: Use the "symmask delete" command to remove those devices from the database. symmask delete -wwn c12345f3 -dir 2a -p 1 Masking and Mapping discovery should complete normally after all bad masking entries have been removed. General provider configuration issuessmi-s Provider for Symmetrix and CLARiiON 73

74 Troubleshooting HiCommand for HDS arrays This section describes issues that may arise with discovering HDS arrays. HiCommand Device Manager does not support an automatic refresh of array data The Hitachi Device Manager or HP Command View XP Device Manager must be configured to automatically refresh the array data contained in the Device Manager database. Otherwise, a manual refresh will need to be performed from the Device Manager user interface before each array discovery. To automatically refresh the array data in the Device Manager, execute the "RefreshStorageArrays" CLI command and specify a refresh interval. For example: HiCommandCLI RefreshStorageArrays -o "D:\logs\RefreshStorageArrays.log" "interval=1440" The Interval time option is specified in minutes from storage subsystem re-discovery completion to start of next storage subsystem re-discovery. You can specify a value from 0 to If it is not set, the default interval time is 0. The user must be a member of the "All Resources Resource Group" and have Modify permissions to execute this command. A user to whom a user-defined resource group is assigned, and who has only the View permission, cannot execute this command. Please refer to the Hitachi Storage Command Suite Hitachi Device Manager Software Command Line Interface (CLI) User s Guide or similar document for HP Command View XP for more information. To manually refresh using the Device Manager UI: 1. Login into Device Manager web-based UI 2. Select "Subsystems" in the Explorer pane 3. Select the array you wish to refresh in the "Subsystems" pane 4. Click "Refresh" button 74 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

75 Troubleshooting Troubleshooting Host discovery This section describes issues that may arise during host discovery. Know the difference between discovered host instances and host instances that are created by resolving their names based on zoning information. Supported HBA, driver, and firmware versions Host discovery fails if the target host does not have elab supported HBAs, drivers, and firmware versions. To verify that hosts use supported versions, obtain the version numbers from hosts and verify them against the elab supported versions: 1. Log in to the target host. 2. Go to /usr/symcli/bin 3. Enter the following command:./syminq fibre hba This command returns the versions on the host. Note: Use the steps 2 and 3 only if symcli is in target machine. If not, use vendor CLIs and UIs (in case of Windows) to find out about the versions. 4. Compare these versions to the supported versions. Refer to the Qlogic website for supported Qlogic HBAs, and refer to the Emulex website for supported Emulex HBAs. For additional supported versions, use the E-Lab Interoperability Navigator tool, available through this URL: Discovering HBA path data If HBA path information is not discovered for a target host, the easiest way to diagnose issues is to go to the target machine and run the inq command. Host discovery uses this command to discover the host. For UNIX hosts, inq is located in: /tmp/nl_dwd Type the following to run inq: inq mapinfo Note: If logged in as a sudo user, type the following to run the inq: sudo /tmp/nl_dwd/inq -mapinfo For example when inq is in target host. In case the ActiveProbe fails to push the inq, follow these steps: 1. Check whether inq is present. General provider configuration issuestroubleshooting Host discovery 75

76 Troubleshooting 2. If the inq is missing check whether HBAs are present. 3. If inq is not pushed even though the HBAs are present check If scp is installed in the host The host is a UNIX 4. If the host is a Windows host, check the ActiveProbeService.log to see why inq push failed. 5. If inq is present, try to push it and observe the output. 6. If the host is accessing host devices from arrays through HBA and paths are missing in inq output, then check the HBA driver and firmware versions. Failure to open WMI sessions The following prerequisites are required for successful Windows host discovery. Windows prerequisites: The user (defined in the access profile) through which the discovery is done must be an administrator, or should be part of an administrator group. Windows Management Instrumentation and Remote Registry services must be running on the target host. DCOM communication must be enabled between the target host and the ITCA SE appliance. On the target host make sure that the following registry key is set to Y : HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Ole ->EnableDCOM Brief detail about how APE works on Windows: The probe will try to check if it has the necessary permission to write into the folders. This check is done by pushing some javascript files (.js) into C:\Documents and Settings\<username>\Local Settings\Temp. The probe tries to run the javascript files using Cscript command. In case of no exceptions, the probe has permissions to access the file system. If the file system permissions are available, INQ will be pushed into C:\Documents and Settings\<username>\Local Settings\Temp\nl_dwd\ Based on the INQ output, the probe takes the information to the UI. This is followed by a range of probes for further discovery process. UNIX prerequisites: There are no pre-requisites other than file access permissions that are available through the root user. Common reasons for discovery failure with Windows hosts The following are the common reasons for discovery failure: Improper credentials Verify that all credentials in the WMI profile are correct. If they are not, the host1.log file displays the following error message: Credentials/Authentication Failure 76 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

77 Troubleshooting If the wrong credentials were passed for discovery from the access profiles), the host1.log file displays the following message for the IP with the credentials of the target host: :03:23 AM==> HostDetection.DiscoveryTaskToTomcatSubmitter.createTasksForIPsAndAddT otomcat: Got an unsuccessful result from ActiveProbe :03:23 AM==> HostDetection.DiscoveryTaskToTomcatSubmitter.createTasksForIPsAndAddT otomcat: The message is Authentication failed :03:23 AM==> HostDetection.DiscoveryTaskToTomcatSubmitter.createTasksForIPsAndAddT otomcat: The resolution is CHANGE_USER_OR_CREDENTIALS :03:23 AM==> HostDetection.setDetectionResults: detection result added is [, Host, SSH, 0,,,,,, 219 discovery ,,, ] If an unreachable IP was tried for detection or discovery or if there were network issues, the following message appears in the host1.log for the targeted IP address which was tried and unsuccessful: :19:34 AM==> HostDetection.DiscoveryTaskToTomcatSubmitter.createTasksForIPsAndAddT otomcat: The resolution is CHECK_NETWORK :19:34 AM==> HostDetection.setDetectionResults: detection result added is [, Host, SSH, 0,,,,,, invalid discovery ,,, ] Logging Active Probe data To capture data from Active Probe (APE) discovery attempts, set the debug flag to true in the following file: Using sudo /opt/dmt/local/conf/host/runtimesettings.properties The default setting is false. Setting it to true enables debugging invalid data originating with APE that may result in failure when updating data to the DM. Error symptoms for a host with partial sudo prerequisites If the target host does not have the sudo account, but the sudo feature is installed, then the policy fails because the account to log in to the host is not available. If the target host has a valid sudo account (without the sudo feature installed), then there is a partial discovery because of limited credentials. If the target host has a valid account with the sudo feature installed, but the credentials provided for sudo account are invalid then the discovery fails because of authentication failure. Verifying sudo feature prerequisites Log in to the respective host using sudo credentials. If login credentials do not work, resolve this with your local IT. Move to the /tmp/nl_dwd folder if it is present and execute sudo /tmp/nl_dwd/inq mapinfo. If sudo is installed, all HBAs and the respective host devices are visible. Otherwise, the following message is populated Sudo is not available. If the /tmp/nl_dwd folder is not present, type sudo. If you see the message sudo: not found or Sudo is not available, have your local IT install it. General provider configuration issuestroubleshooting Host discovery 77

78 Troubleshooting Using the administrative (root) account, check the settings for the host to be discovered in the sudoers file. The UNIX sudoers file is typically located at /etc/sudoers or /usr/local/etc/sudoers. On some UNIX hosts, the sudoers file can be edited using visudo, otherwise use locally available editors (vi, and so on). Issues on UNIX hosts with sudo accounts Sudo Messages repeatedly seen on UNIX hosts with improperly configured sudo accounts On all the UNIX hosts with Sudo accounts, the sudo messages get repeatedly populated. This issue is observed only when the discovery command fails, and a line is added to the console. As the application is designed to retry the same command at an interval of x seconds for 2 hours, this message gets added repeatedly. It is the absence of the sudoers file that causes the problem. Improperly configured sudo accounts could be either of the scenarios explained below: The credentials given to ITCA SE do not include the /etc/sudoers file. Message displayed on the UNIX console: user NOT in sudoers ; TTY=pts/4 ; PWD=/ ; USER=root ; COMMAND=/tmp/nl_dwd/inq -mapinfo /etc/sudoers file was never setup. Sudo accounts are not used. Message displayed on the UNIX console: can't stat /etc/sudoers: No such file or directory; s/2 ; PWD=/ ; USER=root ; COMMAND=/tmp/nl_dwd/inq 78 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

79 Troubleshooting General SMI issues There are some connection related errors that apply to fabric and array-based SMI discovery. During detection, some of these errors may not be a real issue. If several SMI access profiles and IP addresses are given, then the discovery attempts all combinations, and some are bound to fail. Error message: HTTP Error (401 Unauthorized) The following messages are a result of invalid credentials: Error message: Invalid Credentials (Detection) [25-apr :25:23 PM+961ms UTC] t@ Discovery #8 CMPP-W-EDIC-Invalid credentials specified for IP using access profile SMIProfile. Check provider configuration. Error message: Invalid Credentials (Discovery) [25-apr :27:20 PM+210ms UTC] t@ Discovery #9 ASL-W-ERROR_RULE_SOURCE-While executing rule set '/opt/dmt//local/rules/cmp-storage-rules/fabric-smi/switch_discovery_ smi.asl' ASL-ERROR_ACTION-While executing action at: ASL-CALL_STACK_RULE- RuleName: ELEMENT_REACHABLE, Line: 1489 ASL-CALL_STACK_RULE- RuleName: INITIATE_DISCOVERY, Line: 2442 ASL-CALL_STACK_RULE- RuleName: START, Line: 2604 ASL-ERROR_INVOKE-While attempting to invoke operation 'findinstance' of object 'CIMXML_Session::CIMXML_Session access-profile-1-root/ mcdat a' WBEM-ENONEXISTENTINSTANCE-Instance root/mcdata:mcdata_switch%creationclassname=mcdata_switch%name= CD does not exist WBEM-EUNKNOWNERROR-Unknown error occurred: C:ERROR_CLASS_SOFTWARE F:ERROR_FAMILY_FAILED R:1 HTTP Error (401 Unauthorized). [25-apr :27:20 PM+210ms UTC] t@ Discovery #9 CMPP-E-EIC-Invalid credentials specified for IP while discovering Switch test111. Check Access Profile and provider configuration. Error message: Cannot connect to <IP address>. Connection failed. Unreachable IP, SSL Certificates enabled on the CIMOM, wrong port number issues appear with the same Connection Failed message similar to the message in Example 2 and Example 3 To determine what the problem is, ensure the IP can be pinged and check the configuration on the CIMOM. The default port numbers are 5988 for HTTP and 5989 for HTTPS (SSL), but they can be changed on the CIMOM. Note: For ITCA SE and RME 1.0 SSL Certificates (Mutual Authentication) are not supported. Example 2 Sample Connection Failed message (Detection) [25-Feb :29:46 PM+815ms UTC] t@ Discovery #1 General provider configuration issuesgeneral SMI issues 79

80 Troubleshooting CMPP-W-EDUTCTI-Unable to connect to IP using access profile SMIProfile. Make sure provider is running. Example 3 Sample Connection Failed message (Discovery) [25-Feb :28:56 PM+445ms UTC] t@ Discovery #10 CMPP-N-NDISCO_SHORT-Probing Switch information about Switch test111 from IP using Access Profile SMIProfile [25-Feb :28:58 PM+466ms UTC] t@ Discovery #10 ASL-W-ERROR_RULE_SOURCE-While executing rule set '/opt/dmt//local/rules/cmp-storage-rules/fabric-smi/switch_discovery_ smi.a sl' ASL-ERROR_ACTION-While executing action at: ASL-CALL_STACK_RULE- RuleName: ELEMENT_REACHABLE, Line: 1489 ASL-CALL_STACK_RULE- RuleName: INITIATE_DISCOVERY, Line: 2442 ASL-CALL_STACK_RULE- RuleName: START, Line: 2604 ASL-ERROR_INVOKE-While attempting to invoke operation 'findinstance' of object 'CIMXML_Session::CIMXML_Session access-profile-1-root/ mcdat a' WBEM-ENONEXISTENTINSTANCE-Instance root/mcdata:mcdata_switch%creationclassname=mcdata_switch%name= CD does not exist WBEM-EUNKNOWNERROR-Unknown error occurred: C:ERROR_CLASS_SOFTWARE F:ERROR_FAMILY_FAILED R:1 Cannot connect to :5980. Connection failed. [25-Feb :28:58 PM+466ms UTC] t@ Discovery #10 CMPP-E-EUTCTI-Unable to connect to IP while discovering Switch test111. Make sure provider is running. Error message: Unable to reach element. There are two reasons this might happen: The CIMOM is no longer managing the device. For example, a switch may have disappeared after a fabric split. There was a connection error. If it is a connection error, then there should be another error message nearby in the element DM logs, similar to the following messages: CMPP-E-EENR-Unable to reach Switch element angels80 or CMPP-E-EENR-Unable to reach Array element angels80 In the UI, the error code CMPP-E-EENR- is stripped off and does not appear. 80 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

81 Troubleshooting Troubleshooting the gdpm DM Data path related issues may result from problems in the Synchronizer or with the discovered data. If you encounter path issues, use the following procedure to collect information necessary to investigate the problem: 1. Log in to the appliance and collect Translator.log and gdpm.log files in /opt/dmt/local/logs. 2. Run these commands to dump GDP-related data from all DMs: 5. Change current directory to /opt/dmt/local/conf 6. Run dm_dum.sh 7. Collect all generated files in the newly created dm_dump directory. 3. Include gdpm.log under Guidelines for analyzing troubleshooting data Guidelines for analyzing troubleshooting data Below are some guidelines for analyzing the data collected in the following files: Translator.log path.log path.txt array1.txt host1.txt san1.txt If you find problems in the paths between a host and an array: 1. Open host1.txt and locate the host, or port, if you know which host port to look for. 2. Open the path.txt and locate the same host using the instance name. 3. Check the attributes for paths under this host/port. If there is no path: 1. Open the array1.txt file and look for the host port WWN. 2. See if it is masked. If it is, then probably there was an issue during synchronization, and you should look at the Translator.log file. If not, then this indicates the discovery was not complete or there was an issue during discovery or an issue with the provider. If there are paths but the attributes are not as expected, for example, IsConnected and IsZoned are wrong, open the san1.txt file and search for the host port WWN or array port WWN. Check if they are in the same fabric and in the same zone. If yes, then probably it is a synchronization issue. If not, then check the discovery-side issues. General provider configuration issuestroubleshooting the gdpm DM 81

82 Troubleshooting Troubleshooting an offline array Domain Manager If an array DM crashes and/or stops, objects managed by it will not be populated or managed in the other array DM unless the offline DM is restarted. To diagnose whether a DM is offline: 1. If all rediscovery policies managed by a particular DM fail and the following error message is displayed in the UI, then that DM has gone offline. The error message is as follows: Discovery service related data for this element not found. Unable to proceed with discovery of the element <element Name>. 2. Login in to the appliance using SSH and type sm_service show To restart the offline DM: 1. Type sm_service start <ServiceName> Example: sm_service start Array_Server_1 82 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

83 Troubleshooting Using SNMP To use SMMPv3 or SNMPv2, it must be enabled on the switch. Using an incorrect version often causes problems. Cisco SNMP Rediscoveries not invoked on configuration changes Make sure that the appliance IP is set as the trap destination on each Cisco switch managed by the ITCA SE. To set up traps destination on the Cisco switch refer to General provider configuration issuesusing SNMP 83

84 Troubleshooting Troubleshooting user interface conditions This section describes the following user interface (UI) conditions and actions that you need to take to diagnose them: Default value for the UDG refresh period on page 84 Default value for the UDG refresh period on page 84 Discovery status: one or more element discoveries failed on page 85 Discovery status: no elements detected on page 85 Out of memory errors: the user interface hangs, stalls, or slows down on page 85 UI logs for troubleshooting on page 86 Unable to open UI Browser to the appliance on page 87 Unable to see any data in the UI even after creating profiles and policies on page 87 Progress bar for loading data hangs for a long time in any view on page 87 Unable to sort or filter by 'Last Successful' Time of Discovery in the Discovered Objects view on page 87 Object Details and Maps not showing the real-time environment on page 88 Paths shown in maps do not match the Path Details tab on page 88 Default value for the UDG refresh period The default value for the UDG refresh period to refresh dynamic groups' membership is 15 minutes. Every refresh consumes significant CPU and memory resources. Because of the minor memory leak issue in the UDG refresh process, lowering this value will cause GDPM DM memory usage to increase. This will therefore therefore degrade system performance, scalability, and availability. For small to medium-scale systems, the 15 minute default value is fine, and will not impact performance. For large scale systems, however, the recommendation for optimal performance is to set the refresh period to 30 minutes. Based on current memory leak analysis, increasing the refresh period to 30 min will double the lifetime of the GDPM DM. Discovery status permanently stays in running state Follow the procedure given below: 1. SSH to SDA and log in. 2. Verify that all services are running as expected by entering the following command: sm_service show 3. Use the dmctl geti command to determine if you can get instances from the DM by entering the following commands. In this example, the commands attach you to the Path DM: dmctl attach path geti 84 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

85 Troubleshooting The geti command retrieves discovery instances known to the Path DM. This command may hang with the Path DM. If the command hangs on Path DM, it is an indication of deadlock on DM processes that needs to be reported. If not, then it indicates a different error and the logs for UI and DMs need to be handed over to developers for troubleshooting. Discovery status: one or more element discoveries failed The status may not reflect an accurate count of discovered elements, which could be related to failure of event subscriptions. Determine if the point of failure lies with the SDA application or the provider. 1. Verify that all SDA services are running by entering the following command: sm_service show 2. Check the Path DM log as well as the logs for the Host, SAN, and Array DMs. Look for the following strings to see if something has failed: error exception fail asl 3. The sm_service show command shows all services run status. Except 'Path_Starter', all other services must be running. Discovery status: no elements detected Follow the procedure given below: 1. Check the Path DM log as well as the logs for the Host, SAN, and Array DMs. Look for the following strings to see if something has failed: error exception fail asl Table 5 on page 91 provides a list of log files to check. 2. Verify that the provider is up and running. 3. Ping the Cisco or provider switch from the SDA Telnet utility. Out of memory errors: the user interface hangs, stalls, or slows down If you are actively using the same web browser session for 3 hours or longer, you may notice that the UI starts to run slower, and buttons may dissapear or hang. This may be due to the browser sessions's Out Of Memory condition, and such delays depend on the level of activity or scale of data managed by the application. When there is an Out Of Memory condition, the following exceptions can be found in the Java Console: java.lang.outofmemoryerror: Java heap space with polling java.lang.outofmemoryerror: Java heap space In order to view the Java Console, click on the Java icon located on your system's taskbar. Troubleshooting user interface conditionsusing SNMP 85

86 Troubleshooting Example To overcome this issue: Regularly close browser and reopen the application in a new browser at your convenience. Using your operating system's Java Control Panel, increase the browser memory for the java process to a minimum of 256 MB, and maximum of 1 GB. To increase the browser memory on on Windows XP: 1. Open the Windows Control Panel (Start --> Settings --> Control Panel). 2. Double-click on the Java icon within the list. This will open the Java Control Panel. 3. Click the Java tab. 4. In the Java Applet Runtime Settings field, click the View button. (This will open the Java Runtime Settings dialog box). 5. Enter -Xmx1024m -Xms256m in the Java Runtime Parameters field. Note: These instructions may vary for different operating systems. UI logs for troubleshooting It is helpful to collect logs of UI activity when troubleshooting UI behavior. There are two sources that log UI activity: Java console in the browser for the client side Tomcat logs for the server side To obtain client logs from the Java console: 1. Start the browser. 2. From the Internet Explorer Tools menu, select Sun Java Console to start the Java console. 3. For Firefox, select Java Console from the Tools menu. 4. Browse to the appliance and use the SDA as you would under normal operations, navigating to the trouble spot. The console records all activity from the browser client as you work. 5. When you reach the trouble source, examine the console log for errors and exceptions. Click Copy to copy the log contents from the Java console. To check the server logs from the Tomcat server, go to the following locations and check the logs for errors and exceptions: /opt/apache-tomcat /logs/catalina.out /opt/apache-tomcat /webapps/<app-name>/web-inf/logs/litewave.tr c 86 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

87 Troubleshooting Unable to open UI Browser to the appliance If you have problems browsing to the appliance: Ensure that the URL is correct. 6. Verify that you are using a supported browser with a supported version of JRE. For supported browsers, refer to the EMC IT Compliance Analyzer Storage Edition Support Marix available at Unable to see any data in the UI even after creating profiles and policies If unable to see any data: 1. Verify that all DMs are running and alive by entering all of the following commands. Each command returns a DM-specific instance: dmctl --broker=<brokerhost> --server=path geti CMP_AccessProfile dmctl --broker=<brokerhost> --server=array1 geti EMC_SDMStorageArray dmctl --broker=<brokerhost> --server=san1 geti EMC_SDMPhysicalConnectivityDevice dmctl --broker=<brokerhost> --server=host1 geti EMC_SDMHost If the DMs are functioning properly, then each command returns data. 2. Verify the UI logs for 'domain disconnect' or 'lost' phrases to see for any DM connections failure. 3. If the network connection is up and running, click Refresh the view to force the UI to retrieve data again. If possible, restart the Tomcat server. 4. If data does not appear in the UI, check the logs located in the /opt/dmt/local/logs directory in the appliance for any errors in restoring data. Using discovery logs on page 91 provides more information on checking log files. Progress bar for loading data hangs for a long time in any view If queries for getting data are run for a long time or if the DMs are not responsive, the progress bar in the UI tends to hang. If this happens, you can try any of the following: Ensure that all DMs are running and alive. (Refer to Step 1 on page 87.) Check for 'DM disconnect' statements in UI logs. UI logs for troubleshooting on page 86 provides instructions on how to do this. Consider restarting the Tomcat Server. If the DMs are responsive, wait for the queries to finish. This may take a long time but they should finish eventually. If the queries are running, you will observe 100 percent CPU utilization. This indicates that the queries are getting data. Unable to sort or filter by 'Last Successful' Time of Discovery in the Discovered Objects view This is a known issue for which there is no workaround. Troubleshooting user interface conditionsusing SNMP 87

88 Troubleshooting Object Details and Maps not showing the real-time environment This may be due to a lag in discovery or rediscovery policy schedules. Rerun the policies to show the live data. Paths shown in maps do not match the Path Details tab Sometimes, the paths displayed in maps do not match the paths in the Path Details tab. This behavior could be due to the following reasons: 1. If this behavior pertains to a Windows host with PowerPath the pseudo devices will not show the links through all the initiators; it just shows through a single initiator. To overcome this issue use the Host Path Details tab to see all the paths by selecting the device. 2. For hosts with multipathing, if all of the paths from an array to devices are up and running, then they will be displayed correctly. If a path from an array to a device goes down, and then the host is rebooted, the path should no longer be displayed. However, the path is still displayed in this scenario and therefore the user cannot know for sure whether the path is really down or not. 88 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

89 Troubleshooting Useful commands and utilities The following sections describe utilities and commands that may be useful when troubleshooting problems and issues with the SDA. Utilities Table 3 on page 89 provides two useful utilities. Table 3 Useful utilities To Use More information about the utility SSH to the ITCA SE Putty PuTTY is a shareware Telnet/SSH client. Transfer files between a Windows machine and the appliance WinSCP WinSCP is a Windows freeware SFTP and FTP client. Commands Table 4 on page 89 provides a list of useful commands. You must be logged in to the appliance and use them from a command line in the appliance. Table 4 Useful commands To Show status of all required ITCA SE services Stop service Start service Start tomcat (UI and compliance engine) service Stop tomcat (UI and compliance engine) service Start sybase database service Stop sybase database service Start discovery service Stop discovery service Reboot ITCA SE List details of all processes Save list of running processes Collect memory and thread usages in all Domain Managers List files with timestamp Use sm_service show sm_service stop <service name> sm_service start <service name> service tomcat start service tomcat stop service sybase start service sybase stop service ic-serviced start service ic-serviced stop Reboot.sh ps -ef more ps -ef > itcaseprocesses.txt ps -eo thcount,rss,vsz,pid,cmd grep sm_server > dm_info.txt ls -lt more Troubleshooting user interface conditionsuseful commands and utilities 89

90 Troubleshooting Table 4 Useful commands (continued) To Collect TCP connections between all processes. Redirect output to a file for easy analysis. List CPU/memory usage Collect the version of the installed RME build Use netstat -p top conary q grep rme > rme_installed.txt 90 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

91 Troubleshooting Using discovery logs Key logs related to discovery issues are stored in /opt/dmt/local/logs. These logs show discovery progress and any issues. In general, look for the following strings: ERROR EXCEPTION FAIL ASL-ERROR Table 5 on page 91 lists and describes the logs that you can use in troubleshooting discovery issues. Table 5 Domain Manager logs Log Translator.log san1.log path.log indications.log gdpm.log host1.log array1.log / array2.log a sm_trapd.log discovery-service.log AMSDataManager Path_Java Scheduler TaskManager TaskScheduler Description Translate end-to-end path Switch discovery log Task execution Provider event indications This log file collects the information related to gdpm domain manager. This will show when the last time the gdp/udg data was saved to the gdpm RPS file. Lists Host discovery log Lists Array discovery log Cisco traps Discovery Services log Lists all logs from AMS Data expiration service Lists java logs from all the services (discovery service, task manager, task scheduler, scheduler, AMSDataManager and so on) Lists logs from the scheduler service Lists logs from the task manager Lists logs from the task scheduler a. Load balancing that has been introduced that evenly divides the number of arrays discovered in a discovery policy, across both the DMs. This is reflected in the data contained within array1.log and array2.log. All log files are named as follows: <DM name>.log Where <DM name> is the name of the Domain Manager. If you have renamed the DMs, the logs will always bear the name of the DM. Troubleshooting user interface conditionsusing discovery logs 91

92 Troubleshooting Sending logs and repository files to Customer Support Zip the directories in Table 6. Table 6 Logs and repository files for diagnosis Directory /opt/dmt/local/logs /opt/dmt/local/repos /opt/apache-tomcat /logs /opt/apache-tomcat /webapps/sdademoap p/web-inf/logs /opt/dmt/local/logs/logs Description Discovery logs; core file is indication of a service component crash. Repository files that contain details of discovered data GUI internals GUI transactions Task manager and task scheduler logs 92 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

93 Troubleshooting Rediscovery scenarios for user-defined groups An object such as an array or switch may belong to more than one user-defined group. Similarly, a user-defined group may belong to more than one rediscovery policy. Thus there are scenarios where the same object an array or switch, for example is part of multiple rediscovery policies. The system handles this scenario as follows: Assume user-defined group A (UDG A) contains several children UDGs, and a storage array belongs to each of these children UDGs. When the rediscovery policy executes for UDG A, it will create discovery tasks for all the unique members. Duplicate rediscoveries are therefore avoided. Assume there are two separate UDGs: UDG A and UDG B. A storage array belongs to each of them. When the rediscovery policy executes for each UDG, duplicate discovery tasks are created for the array. However, one discovery task will run, and the other will fail, stating that the object is already being discovered. Troubleshooting user interface conditionsrediscovery scenarios for user-defined groups 93

94 Troubleshooting Re-imaging the appliance To re-image your appliance, you need the installation CD or ISO image of the CD. Contact your EMC Customer Service Representative Acquire network settings and log in Before switching on the appliance, obtain the following required network information for your site. IP Address - Appliance Network mask Default Gateway Primary DNS Hostname If your are required to replace the ITCA SE hardware appliance, follow the steps in the Removing and replacing the appliance section of the EMC IT Compliance Analyzer Storage Edition Hardware Replacement Guide. Follow these steps to re-image the ITCA SE software: 1. Insert the software CD (or ISO image burned to a CD), into the CD driver and turn on the appliance. 2. When the Interface window appears, as shown in Figure 7 on page 94, select the To install or upgrade in Graphical mode, and then click Enter. Figure 7 Interface window The Network configuration window appears as shown in Figure 8 on page EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide

95 Troubleshooting Figure 8 Network configuration window 3. In the Network Devices box, in the Network Configuration window, uncheck eth1, eth2, and eth 3, and make sure that eth0 is checked. 4. Click the manually radio button to enable, and then click Edit. The Edit Interface eth0 popup dialog box appears as shown in Figure 9 on page 95. Figure 9 Edit Interface eth0 popup 5. Enter your IP Address and Netmask and then click OK to return to the Network Configuration window. 6. Click Next to display the Time Zone Selection window as shown in Figure 10 on page 96. Troubleshooting user interface conditionsre-imaging the appliance 95

96 Troubleshooting Figure 10 Time zone selection 7. Follow the directions in the left panel of the Time Zone selection window to restore time zone information for your site, and then click Next to begin the software installation. Refer to Linux-based time zones on page 97 to select your relevant Linux-specific time zone. 8. An Installing Packages progress window appears. When the software installation is complete, reboot, and then log in to the appliance home page, <appliance_ip>/itcase and change default password to your previously used admin password. Initial login password is ChangeMe. 9. Proceed to restore your data to the new appliance. Refer to Restore repositories on page 49. Changing the time zone If a time zone is entered incorrectly, you can change the existing time zone. To change time zone settings: a. Type nlclockcfg --getzone <timezone> to view a list of time zones. Note: To view active time zone type nlclockcfg --listzones. Note: You can also view list of supported time zones as shown in Linux-based time zones on page 97. b. Reboot the appliance after changing the time zone. 96 EMC IT Compliance Analyzer Storage Edition Version 1.1 Administration Guide