How to set up, Configure and Manage VCS Virtual Business Services across a Multi-tier Application environment

Transcription

1 SM L09 How to Set up, Configure and Manage VERITAS Cluster Server Virtual Business Services across a Multi-Tier Application environment How to set up, Configure and Manage VCS Virtual Business Services across a Multi-tier Application environment Hands-On Lab Description VERITAS Cluster Server (VCS) 6.0 introduces a new feature known as Virtual Business Service (VBS). VBS ensures high availability and manageability across multiple tiers, hardware architectures, virtualization technologies and operating systems. Gain hands-on experience how to use VCS and ApplicationHA together with VBS to ensure maximum availability protection for a complete business service. Attendees will learn how VBS integrates with different components in a datacenter, such as virtualization technologies. Attendees should have a working knowledge of VCS/ApplicationHA and preferably VERITAS Operations Manager (VOM). They should be able to install and configure both products. However, these components are already installed. At the end of this lab, you should be able to Environment Explain the relationship between VBS, VCS, ApplicationHA and VOM Install the VBS infrastructure from VOM Configure Virtual Business Services Understand VBS dependency types and fault management model The labs are based on 6 Virtual Machines (2-node VCS cluster on Solaris, 2-node VCS cluster on Linux Redhat, 1-node with ApplicationHA and a VOM Central Server). A VCS cluster (dbsolclus) running on Solaris simulates a database cluster with three Oracle service groups. The VCS cluster (applinuxclus) running on Redhat simulates a application cluster with three WebSphere service groups. The ApplicationHA node simulates a VMware Virtual Machine with Windows Server running an IIS Web Service

2 Feature Descriptions This section contains a short introduction of the features that we will configure in this lab session. Virtual Business Services Virtual Business Services (VBS) is an add-on to Symantec s leading clustering software VERITAS Cluster Server (VCS). VBS combines the proven High Availability technology from VCS, as well as ApplicationHA and the power of the VERITAS Operations Manager (VOM) management interface. VBS allows you to manage multiple Service Groups (applications), grouping them into a single entity. The service groups can be located in different clusters, with a different operating system and hardware architecture. Key features of VBS: One-click start/stop of applications in different tiers Fault propagation between tiers, providing maximum availability and automation One-stop shop for HA management, regardless of operating system, hardware platform or virtualization technology Start/Stop Virtual Machines as a part of a VBS start/stop operation (6.0 supports VMware only. Start/stop operations for other virtualization platforms may be implemented in future releases) Note: Virtual Machine start/stop is not covered in this lab. VBS architectural features: No central brain orchestrates HA operations. Each tier is managing its own High Availability. The VOM server is required for initial configuration. VOM can also visualization and management, but is not required for VBS to operate. Backward compatible with VCS 5.1 and 5.1 SP1 and ApplicationHA 5.1 SP2 Support all platforms that VCS and ApplicationHA supports Supports operations using the VOM Central Server or CLI. Start/stop of multiple tiers Coordination to start/stop a complete multi-tier application has been a pain point since the introduction of multi-tiered concepts. Usually, these procedures are carried out manually, which involves interaction from several different teams within an organization. These procedures are time consuming and there are a lot of room for human errors. VBS introduces the ability to start/stop a complete multi-tier application with a single operation. This is especially useful in Disaster Recovery scenarios. SM L09 - Lab Student s Guide / Last Update: Saturday, May 05,

3 Start/Stop of Virtual Machines VBS includes the capability to start and stop individual VMware Virtual Machines. In a disaster recovery scenario, VBS can spin up cold VMs running on a DR site. Fault Management If a failure occurs in one tier, this fault may need to be propagated to another tier. For example, an application in a middle tier may need to be restarted following a failure on a lower tier. VBS introduces fault propagation between different clusters running on different platforms. Symantec ApplicationHA 6.0 ApplicationHA was developed in partnership with VMware and was introduced at VMWorld in late VMwareHA provides infrastructure resilience and ApplicationHA providing application availability in VMware environments, together they provide protection for application failures, virtual machine failures and physical machine failures. The solution provides full support for vmotion, DRS and FT and supports storage interconnects such as NFS, iscsi and Fiber Channel. ApplicationHA is also available for other virtualization platforms, such as Oracle VM Server for SPARC (LDOMs), IBM AIX Micro partitions (DLPARs) and Redhat Kernel Virtual Machines. In this lab, a virtual machine running Windows Server 2008 and ApplicationHA is available. This VM simulates an IIS web server, and is located in the top tier. SM L09 - Lab Student s Guide / Last Update: Saturday, May 05,

4 Lab Exercises Deploying VBS infrastructure In this exercise, you will deploy the VBS packages to support this new feature. It is required to install specific packages on cluster nodes and on the VOM Central Management Server. In later releases of VCS and VOM, packages may be included in the base product. Installing the VBS packages is done in two steps. First, we will install the Virtual Business Services Add-on. This add-on is installed onto the VOM Central Management Server only. In the second step, we will deploy the VBS infrastructure to all VCS cluster nodes / ApplicationHA nodes. This package is known as VRTSvbs. NOTE 1: The Virtual Business Services Add-on and the VRTSvbs packages have already been uploaded to the VOM Central Server to save time. This will allow more time for labs. In a real-world environment, the VBS Add-On and the VRTSvbs package are required to be uploaded to the VOM Central Server. Installing the VBS Add-on onto the VOM Central Management Server 1. Click on the vomserver virtual machine. Click on the VOM Central Management Console short cut. The short cut will take you directly to the VOM splash screen. Login credentials are available on the presentation screen or in the last page of this document 2. In the VOM console, select Settings > Deployment Management 3. In the Deployment Management view, select the Virtual Business Services Availability add-on and click Actions, Install 4. In the Install Solution panel, you will get prompted to install the VBS Availability add-on on the vomserver Central Management Server. Click Install. In this step, certain parts of the VOM Central Management Server will be restarted automatically. Click OK. 5. Login to the VOM Central Management Server again using the browser (refresh the sesson) 6. After the session to the VOM Central Management Server is restored, the Deployment Summary is displayed. Verify that the add-on is installed correctly (This may take some time) Deploying the VRTSvbs package onto the cluster nodes The VRTSvbs package is required on ALL cluster nodes in clusters that participate in a VBS configuration. SM L09 - Lab Student s Guide / Last Update: Saturday, May 05,

5 NOTE: In VCS 6.0, the VRTSvbs package is included in the base product. Hence, there is no need to deploy VRTSvbs package onto nodes running VCS 6.0. In this lab, the following nodes are running VCS 6.0: dbsol1 dbsol2 However, earlier releases of VCS and ApplicationHA will require installation of the VRTSvbs package. In this lab, we will deploy the VRTSvbs package onto required nodes: 1. In the VOM console, select Settings > Deployment Management 2. In the Deployment Management view, select VRTSvbs and click Actions, Install 3. In the Install Solution panel, select all hosts that will participate in the VBS configuration. Select the following nodes: applinux1 applinux2 webwin1 Click Install NOTE: The VRTSvbs package should NOT be deployed on the VOM Central Management Server (vomserver) 4. In the Result panel, click OK 5. In the Deployment Summary view, verify that the VRTSvbs package installs correctly on all nodes. This may take some time Virtual Business Services Setup and Management Initial configuration of a VBS is performed from the VOM Console. After the VBS has been created, it can be managed and displayed using VOM or using the CLI. In this lab, we will configure three different Virtual Business Services, one for each dependency type. The dependency types are explained in a later lab. The above picture displays the lab environment. The following VBS names will be used: VBS_soft, VBS_firm and VBS_restart. SM L09 - Lab Student s Guide / Last Update: Saturday, May 05,

6 The table below describes the VBS configuration and dependencies. VBS Name Lowest Tier / Child Service Group Dependency Depending Tier 1 / Parent Service Group Dependency Depending Tier 2 / Parent Service Group VBS_soft VBS_firm VBS_restart oracle1 Soft websphere1 N/A N/A oracle2 Firm websphere2 N/A N/A oracle3 Restart websphere3 Soft IIS_SG (child SG for IIS_SG) The following picture shows an example of a parent/child relationship Virtual Business Services creation A VBS is created in two steps. First, a Business Entity is created that includes the service group you want to have in the VBS. Secondly, VBS functionality such as dependencies and fault propagation is configured and enabled. Repeat the following steps for each VBS using the details provided in the above table: 1. Click Manage, Business Entities 2. Click Actions, Create Business Entity (VBS/OE) 3. Type in a Name from the above table. Make sure that Virtual Business Service is selected as Type and click Next 4. In the Select Base Object Types panel, select Service Groups and click Next 5. In the Select Base Service Groups panel, select Service Groups according to the above table and click Next SM L09 - Lab Student s Guide / Last Update: Saturday, May 05,

7 6. In the Business Entity Summary panel, click Finish 7. Right-click on the newly created VBS, click VBS Availability, Configure Service Group Dependencies 8. In the Set Start order panel, configure Service Group dependencies according to the above table. Select Parent and Child, and click Link. (for the VBS_restart configuration, you will configure two service group dependencies) NOTE: For all dependency types except soft, the Fault Management checkbox is required to be true. If CLI support is desired, this checkbox should be set to true for the soft dependency as well. Click Next 9. Validate the dependencies and click Finish 10. In the Results panel, click OK 11. Repeat the above steps for each VBS (VBS_soft, VBS_firm and VBS_restart) Start/Stop and visualization of Virtual Business Services The objective in this lab is to visualize the state of a VBS and to test start/stop functionality. VBS can be managed using VOM or CLI. Start/stop using VOM: 1. Click Manage, Business Entities 2. Right-click on a VBS, select VBS Availability, Start VBS or Stop VBS 3. Click OK to confirm that you want to start up the VBS 4. The Service Group Order panel is displayed. Examine and validate start/stop of the VBS Start/stop using CLI: 1. Open a terminal session as root to any cluster node (the vomserver host is not part of a VBS configuration) 2. Use the following commands to start/stop and visualize a VBS from CLI: # vbssvc start/stop <vbs> # vbssvc state <vbs> <vbs> entry is optional. If not specified, all VBS configured in the cluster are displayed # vbssbc grpstate <vbs> Displays the consolidated status of all servicegroups within the VBS SM L09 - Lab Student s Guide / Last Update: Saturday, May 05,

8 # vbsgrp state <vbs> <vbs> entry is optional. If not specified, groups from all VBS configured on the cluster will be displayed Understanding VBS Fault Management High Availability decisions, such as failing over a service group from one node to another, are handled on the local cluster level. This is the default VCS behaviour, and VBS does not change this. However, VBS provides the ability to propagate an event from one tier to another (between clusters and/or ApplicationHA instances) Fault propagation works autonomously of the VOM Central Management Server. Even if the VOM Central Management Server is unavailable, fault propagation is still in effect. Note that VBS will propagate events upon failures only. If fault propagation is enabled for a VBS, and a service group fails, a fault propagation event will be triggered. However, if a service group is switched over manually, a fault will not be propagated to another tier. In our examples, we have one 3-tier VBS (VBS_restart) and two 2-tier VBS (VBS_soft and VBS_firm) The following table describes the VBS dependency types: Dependency Type Soft Firm Restart Explanation When a fault occurs, the event is not propagated to another tier When the child faults, the parent is taken OFFLINE. When the child recovers, the parent is brought ONLINE When the child faults, the parent ignores the event. When the child recovers, the parent is taken OFFLINE and then brought ONLINE. Use case Used for start/stop orchestration/ordering Some applications have a strict dependency on lower level components. For example, the firm dependency is useful when an application requires a NFS share. If the NFS share goes offline, the application will be brought offline Some applications requires a restart following a failure of let s say a database. Application may need to restart to reconnect to the database. The Restart dependency resolves this situation and will provide another layer of High Availability. Understanding the soft service group dependency 1. Open a Terminal Session to applinux1 and dbsol1 as root. Issue the following command in both sessions: # tail f /var/vrtsvcs/log/engine_a.log 2. Start the VBS_soft VBS using VOM. Make sure that the Service Group order panel is displayed. This panel is displayed automatically after VBS start has been initiated SM L09 - Lab Student s Guide / Last Update: Saturday, May 05,

9 3. Open another terminal session to dbsol1 4. Issue the following # hastatus sum command to locate the oracle1 service group. # hastatus -sum -- SYSTEM STATE -- System State Frozen A dbsol1 RUNNING 0 A dbsol2 RUNNING 0 -- GROUP STATE -- Group System Probed AutoDisabled State B ClusterService dbsol1 Y N ONLINE B ClusterService dbsol2 Y N OFFLINE B oracle1 dbsol1 Y N ONLINE Online on dbsol1 B oracle1 dbsol2 Y N OFFLINE B oracle2 dbsol1 Y N ONLINE B oracle2 dbsol2 Y N OFFLINE B oracle3 dbsol1 Y N ONLINE B oracle3 dbsol2 Y N OFFLINE 5. If the oracle1 service group is online on dbsol1, continue with the next step. If the oracle1 service group is online on dbsol2, login to dbsol2 and then continue with the next step 6. Simulate a failure of the oracle1 service group by removing the following lock file: # rm /testfiles/oracle1 Examine the output in the terminal sessions started in step 1, as well as the visualization in the VOM console. The oracle1 service group should be failed over from one system to another, and the fault should NOT be propagated to another tier, as the soft dependency doesn t perform fault propagation. 7. Clear the resource failure using the following command on dbsol1 or dbsol2: # hagrp clear oracle1 Understanding the Firm service group dependency To demo the Firm dependency type, a double failure is required. We will fail the oracle2 service group twice (once on node dbsol1 and once on node dbsol2) SM L09 - Lab Student s Guide / Last Update: Saturday, May 05,

10 1. Open a Terminal Session to applinux1 and dbsol1 as root. Issue the following command in both sessions: # tail f /var/vrtsvcs/log/engine_a.log 2. Start the VBS_firm Virtual Business Service using VOM. Make sure that the Service Group order panel is displayed. This panel is displayed automatically after VBS start has been initiated 3. Open a terminal session to dbsol1 4. Issue the following # hastatus -sum command to locate the oracle2 service group # hastatus -sum -- SYSTEM STATE -- System State Frozen A dbsol1 RUNNING 0 A dbsol2 RUNNING 0 -- GROUP STATE -- Group System Probed AutoDisabled State B ClusterService dbsol1 Y N ONLINE B ClusterService dbsol2 Y N OFFLINE B oracle1 dbsol1 Y N ONLINE B oracle1 dbsol2 Y N OFFLINE B oracle2 dbsol1 Y N ONLINE Online on dbsol1 B oracle2 dbsol2 Y N OFFLINE B oracle3 dbsol1 Y N ONLINE B oracle3 dbsol2 Y N OFFLINE 5. If the oracle2 service group is online on dbsol1, continue with the next step. If the oracle1 service group is online on dbsol2, login to dbsol2 and then continue with the next step 6. Simulate a failure of the oracle2 service group by removing the following lock file: # rm /testfiles/oracle2 Use # hastatus sum to determine that the service group has failed over from one system to another. Login to the other system and issue the above command again: # rm /testfiles/oracle2 Examine the output in the terminal sessions started in step 1, as well as the visualization in the VOM console. The oracle2 service group should be failed over from one system to another, and the fault is propagated to the application tier (applinuxclus). In the Firm case, this means that the dependent service groups (websphere2) is stopped. 7. Clear the resource failure and online the database (commands can be executed from dbsol1 or dbsol2) # hagrp clear oracle2 SM L09 - Lab Student s Guide / Last Update: Saturday, May 05,

11 # hagrp online oracle2 sys dbsol1 This step simulates that an administrator are resolving a non-recoverable issue with the database, and then starting up the database again. Examine the output in the terminal sessions started in step 1 and the visualization in the VOM console again. The dependent service groups are now started by VBS. Understanding the Restart service group dependency 1. Open a Terminal Session to applinux1 and dbsol1 as root. Issue the following command in both sessions: # tail f /var/vrtsvcs/log/engine_a.log 2. Start the VBS_restart Virtual Business Service using the VOM console. Make sure that the Service Group order panel is displayed. This panel is displayed automatically after VBS start has been initiated 3. Open a terminal session to dbsol1 4. Issue the following hastatus -sum command to locate the oracle3 service group # hastatus -sum -- SYSTEM STATE -- System State Frozen A dbsol1 RUNNING 0 A dbsol2 RUNNING 0 -- GROUP STATE -- Group System Probed AutoDisabled State B ClusterService dbsol1 Y N ONLINE B ClusterService dbsol2 Y N OFFLINE B oracle1 dbsol1 Y N ONLINE B oracle1 dbsol2 Y N OFFLINE B oracle2 dbsol1 Y N ONLINE B oracle2 dbsol2 Y N OFFLINE B oracle3 dbsol1 Y N ONLINE Online on dbsol1 B oracle3 dbsol2 Y N OFFLINE 5. If oracle3 service group is online on dbsol1, continue with the next step. If oracle3 service group is online on dbsol2, login to dbsol2 and then continue with the next step SM L09 - Lab Student s Guide / Last Update: Saturday, May 05,

12 6. Simulate a failure of the oracle3 service group by removing the following lock file: # rm /testfiles/oracle3 Examine the output in the terminal sessions started in step 1, as well as the visualization in the VOM console. The oracle3 service group should be failed over from one system to another, and the fault is propagated to the Websphere tier. When the service group is failed over in the database tier, the Websphere tier is restarted. 8. Clear the resource failure using the following command on dbsol1 or dbsol2: # hagrp clear oracle3 Creating shared VBS dependencies You have deployed the VBS infrastructure, learned how to manage VBS from VOM and from the command line, and also understood how VBS fault management works. VBS also support shared service groups. A shared service group is shared between two or more VBS entities. VBS applies a specific logic to shared service groups. Startup of a VBS with one or more shared service groups: VBS will start up all service groups in the configured order. Shutdown of a VBS with one or more shared service groups: VBS will stop all non-shared service groups. By default, shared service groups will not be stopped. However, there is an option to offline shared service groups from the VOM Console and from the CLI You will reconfigure the environment according to the picture below: SM L09 - Lab Student s Guide / Last Update: Saturday, May 05,

13 Removing the VBS_Firm VBS and configuring a new shared VBS In this lab, we will remove the VBS_Firm VBS from the configuration and create a new shared VBS. 1. In the VOM console, select Manage > Business Entities 2. Right-click on the VBS_Firm VBS and click Delete Business Entity 3. Click Yes to confirm that you want to remove the VBS. Click OK 4. Click Actions, Create Business Entity (VBS/OE) 5. In the Name field, type VBS_shared. Make sure that Virtual Business Service is selected as Type and click Next 6. In the Select Base Object Types panel, select Service Groups and click Next 7. In the Select Base Service Groups panel, select oracle3 and websphere2 8. In the Business Entity Summary panel, click Finish 9. Right-click on the VBS_shared VBS, click VBS Availability, Configure Service Group Dependencies 10. In the Set Start order panel, configure the oracle3 service group as child, and the websphere2 service group as parent. Select Dependency Type Soft. Click Next 11. Validate the dependencies and click Finish SM L09 - Lab Student s Guide / Last Update: Saturday, May 05,

14 Start/Stop of a VBS with shared service groups In this lab, we will validate start and stop behavior when one VBS has a shared service group configured. 1. Stop VBS_shared from the VOM Console. Note the warning about shared service groups. Do not select Offline shared service groups 2. Stop VBS_restart. Select Offline shared service groups (If you prefer CLI operations, you can use # vbssvc stop force VBS_restart 3. Start VBS_restart and VBS_shared again and confirm that they online correctly 4. Induce a failure on oracle3 (remove the lock file as instructed earlier). Examine the difference and note that fault propagation only is effective on the websphere3 service group. Lab Configuration Information SM L09 - Lab Student s Guide / Last Update: Saturday, May 05,

15 Hostname IP dbsol dbsol applinux applinux webwin vom Cluster Dbsolclus Dbsolclus Applinuxclus Applinuxclus ApplicationHA instance N/A Service Group IP ClusterService ClusterService Cluster Dbsolcluster Applinuxclus Credentials: vom: dbsol1/dbsol2 applinux1/applinux2 webwin1 root/symc4now root/symc4now root/symc4now administrator/symc4now SM L09 - Lab Student s Guide / Last Update: Saturday, May 05,