SA L07 Virtual Business Services Lab. Hands-On Lab

Transcription

1 SA L07 Virtual Business Services Lab Hands-On Lab Description VERITAS Cluster Server (VCS) includes a feature known as Virtual Business Service (VBS). VBS ensures high availability and manageability across multiple tiers, hardware architectures, virtualization technologies and operating systems. Gain hands-on experience on how to use VCS and ApplicationHA together with VBS to ensure 24x7 availability for a complete Multitier business service from the top tier to the bottom tier. Attendees will learn how VBS integrates with different components in a datacenter, such as virtualization technologies. Attendees should have a working knowledge of VCS and preferably VERITAS Operations Manager (VOM). They should be able to install and configure both products. At the end of this lab, you should be able to Environment Explain the relationship between VBS, VCS, ApplicationHA and VOM Install the VBS infrastructure from VOM Configure Virtual Business Services Understand VBS dependency types and fault management model Understand and configure Recovery Plans The current lab simulates two data centers, one in California (CA) and one on New York (NY). The labs are based on 8 Virtual Machines Database Tier: 3 nodes running Solaris and VCS (Global Cluster configuration). Oracle database Application Tier: 2 nodes running Linux and VCS (single node configurations) WebSphere application Web Tier: 2 VMware Virtual Machines running Windows and ApplicationHA. IIS web server

2 Feature Descriptions This section contains a short introduction of the features that we will configure in this lab session. Virtual Business Services Virtual Business Services (VBS) is an add-on to Symantec s leading clustering software Veritas Cluster Server (VCS). VBS combines the proven High Availability technology from VCS, as well as ApplicationHA and the power of the Veritas Operations Manager (VOM) management interface. VBS allows you to manage multiple Service Groups (applications), grouping them into a single entity. Service groups can be located in different clusters, with different operating systems and hardware architectures. Key features of VBS: One-click start/stop operations of applications in different tiers Fault propagation between tiers, providing availability on the service level One-stop shop for HA management, regardless of operating system, hardware platform or virtualization technology Start/Stop of VMware Virtual Machines as a part of VBS operations One click disaster recovery by using the Recovery Plan. VBS architectural features: No central brain orchestrates HA operations. Each tier is managing its own High Availability. The VOM server is required for initial configuration. Visualization and Management is also provided by VOM. However, VBS operations can be carried out independently of the VOM server by using the VBS CLI. VBS is supported from VCS 5.1/ApplicationHA 5.1 SP2 and onwards. All later releases of VCS/ApplicationHA supports VBS. Supports all hardware architectures and operating systems that VCS and ApplicationHA supports. Supports operations using the VOM Central Management Server or VBS CLI.

3 Start/stop of multiple tiers Coordination to start/stop a complete multi-tier application has been a pain point since the introduction of multi-tiered concepts. Usually, these procedures are carried out manually, which involves interaction from several different teams within an organization. These procedures are time consuming and there is a lot of room for human error. VBS introduces the ability to start/stop a complete multi-tier application with a single operation. Fault Management Keeping components in each tier highly available is still the foundation for service level availability. This is maintained by VCS and/or ApplicationHA. However, if a failure occurs in one tier, this fault may need to be propagated to another tier. For example, an application in a middle tier may need to be restarted following a failure on a lower tier. VBS introduces fault propagation between different clusters running on different platforms. By propagating failures, VCS/ApplicationHA, together with VBS can maintain the availability for the complete multi-tier business service. Start/Stop of Virtual Machines VBS includes the capability to start and stop individual VMware Virtual Machines. In a disaster recovery scenario, VBS can spin up cold VMs running on a DR site. Note: Virtual Machine start/stop is not covered in this lab. Disaster Recovery VBS includes extensive disaster recovery support. To carry out a successful disaster recovery event in a real-world disaster situation, automation is key. VBS together with the Recovery Plan provides this functionality. Bringing up a data center on the disaster recovery site can be done by executing a single command.

4 Lab Exercises Deploying VBS infrastructure In this exercise, you will deploy required packages to support the VBS functionality on some of the cluster nodes and on the VOM Central Management Server. Installing the infrastructure packages is done in two steps. 1. Install the VOM Virtual Business Services Availability Add-On. This add-on is installed onto the VOM Central Management Server only. 2. Deploy the VRTSvbs package. Note this is required for cluster nodes prior to VCS 6.0 and on ApplicationHA nodes. In this lab, you will deploy this package to the ApplicationHA nodes. All other clusters are running VCS NOTE: The VOM Virtual Business Services Availability Add-on and the VRTSvbs packages have already been uploaded to the VOM Central Management Server to save time. Eliminating this step will give you more time to complete labs. In a real-world environment, the VBS Availability Add-On and the VRTSvbs package are required to be uploaded to the VOM Central Management Server. Installing the VOM VBS Availability Add-on onto the VOM Central Management Server 1. Click on the vomserver virtual machine. 2. Login to the server by using the credentials on the presentation screen or on the last page of this document. (root/symc4now) 3. Click on the VOM Central Management Server Login Page short cut, located on the desktop. The short cut will take you directly to the VOM splash screen. Login credentials are available on the presentation screen or on the last page of this document. (root/symc4now) 4. In the Veritas Operations Manager console, select Settings > Domain Management > Deployment. 5. In the Deployment Management view, select the Virtual Business Services Availability add-on and click Actions, Install. 6. In the Install Solution panel, you will get prompted to install the VBS Add-On on the vomserver Central Management Server. Click Install.

5 7. Click on Goto Solutions Summary. NOTE: VOM will be restarted in the background. This may take some time. Refresh the session and login to the VOM Central Management Server again. 8. Select Settings > Domain Management > Deployment and verify that the Virtual Business Service Availability Add-on is installed and enabled successfully. Installing the VRTSvbs package 1. In the VOM UI, select Settings > Domain Management > Deployment. 2. In the Deployment Management view, select VRTSvbs and click Actions, Install. NOTE: Select the Windows version of the VRTSvbs package. 3. In the Install Solution panel, select all hosts that will participate in the VBS configuration. Select the following nodes: ca-webwin1 ny-webwin2 Click Install. NOTE: The VRTSvbs package is NOT required on the VOM Central Management Server. (vomserver) 4. Click on Goto Solutions Summary. In the Deployment Summary view, verify that the VRTSvbs package installs correctly on all nodes. This may take some time. Virtual Business Services Setup and Management Initial configuration of a VBS is performed from the VOM UI. After a VBS has been created, it can be managed and displayed using VOM or CLI. In this lab, we will configure three different Virtual Business Services, one for each dependency type. The dependency types will be explained in a later lab.

6 This picture displays how the lab environment will look like when the VBS configuration is completed. Initially, you will configure the California site. In the Disaster Recovery labs, you will configure the New York site. The table below describes the VBS configuration and dependencies. VBS Name HR_VBS Sales_VBS Finance_prod_VBS Lowest Tier / Child Service Group Dependency Type Depending Tier 1 / Parent Service Group Dependency Type Depending Tier 2 / Parent Service Group ca_oracle1 Soft ca_websphere1 N/A N/A ca_oracle2 Firm ca_websphere2 N/A N/A global_oracle3 Restart ca_websphere3 Soft CA_IIS_SG (child SG for CA_IIS_SG)

7 NOTE: Please disregard the DR VBS (Finance_DR_VBS) for the moment. We will configure that VBS in a later lab exercise. The following picture shows an example of a parent/child relationship: Virtual Business Services creation You will now create three VBS s specified in the table above. Repeat the following steps for each VBS. 1. In the VOM GUI, click Manage > Business Entities > Virtual Business Services. 2. Click Actions, Create Virtual Business Service. 3. Type in a Name from the table above and click Next. 4. In the Select Base Object Types panel, select Service Groups and click Next. 5. In the Select Base Service Groups panel, select Service Groups according to the table above and click Next. NOTE for Finance_prod_VBS: Two service groups with the same name (global_oracle3) is present, one for each site. Make sure to select the global_oracle3 service group from the ca-dbsolclus1 cluster. 6. In the Virtual Business Service Summary panel, click Finish. 7. Right-click on the newly created VBS, click VBS Availability, Configure Service Group Dependencies. 8. In the Virtual Business Service start order panel, configure Service Group Dependencies according to the table above. Select Parent and Child, and click Link.

8 NOTE: For the Finance_prod_VBS configuration, you will configure two service group dependencies. Make sure that the Configure Fault Management checkbox is selected. This checkbox should be set to true for new VBS deployments. Click Next. 9. In the Virtual Business Service configuration summary panel, validate the dependencies and click Finish. 10. In the Results panel, click OK. 11. Repeat the above steps for each VBS (HR_VBS, Sales_VBS and Finance_prod_VBS) Start/Stop and visualization of Virtual Business Services The objective in this lab is to visualize the state of a VBS and to test start/stop functionality. VBS can be managed using VOM or CLI. Start/stop using VOM: 1. Click Manage > Business Entities > Virtual Business Services. 2. Right-click on a VBS, select VBS Availability, Start VBS or Stop VBS. 3. Click OK to confirm that you want to start up the VBS. 4. The Service Group Order panel is displayed. Examine and validate start/stop of the VBS. Note that the progress of the operation is displayed on the right hand side of the VOM GUI. Start/stop using CLI: 1. Open a terminal session as the root user to any node participating in the VBS (cadbsol1, ca-dbsol2, ca-applinux1. Note that those commands cannot be executed from the VOM Central Management Server)

9 Click on the Terminal icon on the desktop of the vomserver virtual machine. Connect to the remote node by using SSH: # ssh ca-dbsol1 If prompted for a password, use symc4now NOTE: In remaining labs, it s assumed that the participant know how to use SSH to connect to the cluster nodes. This document does not specify how to connect to the cluster nodes from now on. 2. Use the following commands to start/stop and visualize a VBS from CLI: # vbssvc display ß Displays the VBS configuration # vbssvc start/stop <vbs> ß Start/stop a VBS # vbssvc state <vbs> ß <vbs> entry is optional. If not specified, all VBS configured in the cluster are displayed # vbssvc grpstate <vbs> ß displays the availability state for each group per VBS # vbsgrp state <vbs> ß <vbs> entry is optional. If not specified, groups from all VBS configured on the cluster will be displayed Understanding VBS Fault Management High Availability High Availability decisions, such as failing over a service group from one node to another, are handled by VCS/ApplicationHA on the local cluster level. In addition, VBS provides the ability to propagate an event from one tier to another. Fault propagation is not dependent on the VOM Central Management Server. Even if the VOM Central Management Server is unavailable, an event will still be propagated. Note that VBS will propagate events upon failures only. A service group has to fail to trigger fault propagation. If a service group is switched over manually, a fault will not be propagated to another cluster/tier. In our examples, we have one 3-tier VBS (Finance_prod_VBS) and two 2-tier VBS (HR_VBS and Sales_VBS)

10 The following table describes the VBS dependency types: Dependency Type Soft Firm Restart Explanation When a fault occurs, the event is not propagated to another tier When the child faults, the parent is taken OFFLINE. When the child recovers, the parent is brought ONLINE When the child faults, the parent ignores the event. When the child recovers, the parent is taken OFFLINE and then brought ONLINE. Use case Used for start/stop orchestration only. Some applications may cause issues such as crashes, unnecessary memory allocation etc. (e.g. a database is unavailable) Using a Firm dependency avoids this situation. Some applications require a restart following a failure. (e.g. a database failure) Application may need to restart to reconnect to the database. The Restart dependency resolves this situation and provides another layer of High Availability. Understanding the soft service group dependency 1. Open a Terminal Session to ca-applinux1 and ca-dbsol1 using SSH. Issue the following command in both sessions: # tail -f /var/vrtsvcs/log/engine_a.log 2. Start the HR_VBS Virtual Business Service using VOM. Make sure that the Service Group order panel is displayed. This panel is displayed automatically after VBS start has been initiated. 3. Open a terminal session to ca-dbsol1. 4. Issue the following # hastatus command to locate the ca_oracle1 service group: # hastatus -sum -- SYSTEM STATE -- System State Frozen A ca-dbsol1 RUNNING 0 A ca-dbsol2 RUNNING 0 -- GROUP STATE -- Group System Probed AutoDisabled State

11 B ClusterService ca-dbsol1 Y N ONLINE B ClusterService ca-dbsol2 Y N OFFLINE B ca_oracle1 ca-dbsol1 Y N ONLINE ß Online B ca_oracle1 ca-dbsol2 Y N OFFLINE B ca_oracle2 ca-dbsol1 Y N ONLINE B ca_oracle2 ca-dbsol2 Y N OFFLINE B global_oracle3 ca-dbsol1 Y N ONLINE B global_oracle3 ca-dbsol2 Y N OFFLINE -- WAN HEARTBEAT STATE -- Heartbeat To State M Icmp ny-dbsolclus2 ALIVE -- REMOTE CLUSTER STATE -- Cluster State N ny-dbsolclus2 RUNNING -- REMOTE SYSTEM STATE -- cluster:system State Frozen O ny-dbsolclus2:ny-dbsol3 RUNNING 0 -- REMOTE GROUP STATE -- Group cluster:system Probed AutoDisabled State P global_oracle3 ny-dbsolclus2:ny-dbsol3 Y N OFFLINE 5. If the ca_oracle1 service group is online on ca-dbsol1, continue with the next step. If the ca_oracle1 service group is online on ca-dbsol2, login to ca-dbsol2 and then continue with the next step. 6. Simulate a failure of the ca_oracle1 service group by removing the following lock file: # rm /testfiles/oracle1 Examine the output in the terminal sessions started in step 1, as well as the visualization in the VOM UI. The ca_oracle1 service group should be failed over from one system to another, and the fault should not be propagated to another tier.

12 7. Clear the resource failure of the ca_oracle1 service group. This command can be executed from ca-dbsol1 or ca-dbsol2, or alternatively, the fault can be cleared from VOM. # hagrp clear ca_oracle1 Understanding the Firm service group dependency To demo the Firm dependency type, the child (lower) service group must be faulted on all nodes. We will fail the ca_oracle2 service group twice (once on node ca-dbsol1 and once on node ca-dbsol2) 1. Open a Terminal Session to ca-applinux1 and ca-dbsol1 using SSH. Issue the following command in both sessions: # tail -f /var/vrtsvcs/log/engine_a.log 2. Start the Sales_VBS Virtual Business Service using VOM. Make sure that the Service Group order panel is displayed. This panel is displayed automatically after VBS start has been initiated. 3. Open a terminal session to ca-dbsol1. 4. Issue the following # hastatus command to locate the ca_oracle2 service group # hastatus -sum -- SYSTEM STATE -- System State Frozen A ca-dbsol1 RUNNING 0 A ca-dbsol2 RUNNING 0 -- GROUP STATE -- Group System Probed AutoDisabled State B ClusterService ca-dbsol1 Y N ONLINE B ClusterService ca-dbsol2 Y N OFFLINE B ca_oracle1 ca-dbsol1 Y N ONLINE B ca_oracle1 ca-dbsol2 Y N OFFLINE B ca_oracle2 ca-dbsol1 Y N ONLINE ß Online B ca_oracle2 ca-dbsol2 Y N OFFLINE

13 B global_oracle3 ca-dbsol1 Y N ONLINE B global_oracle3 ca-dbsol2 Y N OFFLINE -- WAN HEARTBEAT STATE -- Heartbeat To State M Icmp ny-dbsolclus2 ALIVE -- REMOTE CLUSTER STATE -- Cluster State N ny-dbsolclus2 RUNNING -- REMOTE SYSTEM STATE -- cluster:system State Frozen O ny-dbsolclus2:ny-dbsol3 RUNNING 0 -- REMOTE GROUP STATE -- Group cluster:system Probed AutoDisabled State P global_oracle3 ny-dbsolclus2:ny-dbsol3 Y N OFFLINE 5. If the ca_oracle2 service group is online on ca-dbsol1, continue with the next step. If the ca_oracle1 service group is online on ca-dbsol2, login to ca-dbsol2 and then continue with the next step. 6. Simulate a failure of the ca_oracle2 service group by removing the following lock file: # rm /testfiles/oracle2 Use hastatus -sum to determine that the service group has failed over from one system to another. Login to the other system and issue the above command again: # rm /testfiles/oracle2 Examine the output in the terminal sessions started in step 1, as well as the visualization in the VOM UI. The ca_oracle2 service group is now completely faulted and cannot go online. The Firm dependency propagates the failure and brings offline the ca_websphere2 service group in the tier above. Examine the log output as well as the VOM visualization.

14 7. Clear the resource failure and online the ca_oracle2 service group. This command can be executed from ca-dbsol1 or ca-dbsol2, or alternatively, the fault can be cleared from VOM. # hagrp -clear ca_oracle2 # hagrp -online ca_oracle2 -sys ca-dbsol1 This step simulates that an administrator resolved a non-recoverable issue with the database, and then starting up the database again. Examine the output in the terminal sessions started in step 1 and the visualization in the VOM UI again. The dependent service groups are automatically started by VBS. Understanding the Restart service group dependency 1. Open a Terminal Session to ca-applinux1 and ca-dbsol1 using SSH. Issue the following command in both sessions: # tail -f /var/vrtsvcs/log/engine_a.log 2. Start the Finance_prod_VBS Virtual Business Service using the VOM UI. Make sure that the Service Group order panel is displayed. This panel is displayed automatically after VBS start has been initiated. 3. Open a terminal session to ca-dbsol1. 4. Issue the following hastatus command to locate the global_oracle3 service group # hastatus -sum -- SYSTEM STATE -- System State Frozen A ca-dbsol1 RUNNING 0 A ca-dbsol2 RUNNING 0 -- GROUP STATE -- Group System Probed AutoDisabled State B ClusterService ca-dbsol1 Y N ONLINE

15 B ClusterService ca-dbsol2 Y N OFFLINE B ca_oracle1 ca-dbsol1 Y N ONLINE B ca_oracle1 ca-dbsol2 Y N OFFLINE B ca_oracle2 ca-dbsol1 Y N ONLINE B ca_oracle2 ca-dbsol2 Y N OFFLINE B global_oracle3 ca-dbsol1 Y N ONLINE ß Online B global_oracle3 ca-dbsol2 Y N OFFLINE -- WAN HEARTBEAT STATE -- Heartbeat To State M Icmp ny-dbsolclus2 ALIVE -- REMOTE CLUSTER STATE -- Cluster State N ny-dbsolclus2 RUNNING -- REMOTE SYSTEM STATE -- cluster:system State Frozen O ny-dbsolclus2:ny-dbsol3 RUNNING 0 -- REMOTE GROUP STATE -- Group cluster:system Probed AutoDisabled State P global_oracle3 ny-dbsolclus2:ny-dbsol3 Y N OFFLINE 5. If global_oracle3 service group is online on ca-dbsol1, continue with the next step. If global_oracle3 service group is online on ca-dbsol2, login to ca-dbsol2 and then continue with the next step. 6. Simulate a failure of the global_oracle3 service group by removing the following lock file: # rm /testfiles/oracle3 Examine the output in the terminal sessions started in step 1, as well as the visualization in the VOM UI. The global_oracle3 service group should be failed over from one system to another, and the fault is propagated to the application tier, in this case the ca_websphere3 service group running on the ca-applinux1 cluster node.

16 This service group will automatically be restarted when the global_oracle3 service group fails over. 7. Clear the resource failure of the global_oracle3 service group. This command can be executed from ca-dbsol1 or ca-dbsol2, or alternatively, the fault can be cleared from VOM. # hagrp clear global_oracle3 Configuring Virtual Business Services for Disaster Recovery Together with Veritas Cluster Server and Veritas Operations Manager, VBS implements endto-end disaster recovery management. Disaster Recovery can be managed per VBS, or for a set of VBS s, or for the complete data center. The data center can be recovered on the DR site by executing one single command. The following prerequisites exist for VBS DR operations: At least one tier needs to be configured with VCS/GCO Control of data replication should be managed by VCS/GCO. For tiers without data replication requirements, GCO is not mandatory. VBS DR operations are not supported together with VMware SRM. The service group on the primary site and the secondary site cannot be in the same VBS.

17 In this lab, the number of VBS tiers between production and DR sites is equal. While this is most common, it is not a requirement. Number of tiers, number of cluster nodes, OS etc, can differ. In this lab, two sites are simulated (California and New York). The database tier is configured with VCS/GCO, while the remaining two tiers are configured for local HA only (note that the number of nodes has been reduced to prevent lab machines from being overloaded) VCS on the database tier will take care about the availability of the database. The following flow is required to configure a VBS for Disaster Recovery: 1. Configure VCS/GCO (already done) 2. Configure replication and global service groups (already done, but simulated for the sake of the lab) 3. Configure the VBS on the primary site, California (done in earlier labs) 4. Configure the VBS on the secondary site, New York. (this will be done in the next lab)

18 Configuring the DR VBS. 1. In the VOM GUI, click Manage > Business Entities > Virtual Business Services. 2. Click Actions, Create Virtual Business Service. 3. Type in a Finance_DR_VBS as Name and click Next. 4. In the Select Base Object Types panel, select Service Groups and click Next. 5. In the Select Base Service Groups panel, select Service Groups the following service groups: global_oracle3 (Note that two service groups exist with this name. Make sure to select the global_oracle3 service group from the ny-dbsolclus2 cluster) ny_websphere3 NY_IIS_SG Click Next. 6. In the Virtual Business Service Summary panel, click Finish. 7. Right-click on the newly created VBS, click VBS Availability, Configure Service Group Dependencies. 8. In the Virtual Business Service start order panel, configure Service Group Dependencies according to the table below. Select Parent and Child, and click Link. VBS Name Lowest Tier / Child Service Group Dependency Type Depending Tier 1 / Parent Service Group Dependency Type Depending Tier 2 / Parent Service Group Finance_DR_VBS global_oracle3 Restart ny_websphere2 Soft NY_IIS_SG 9. Make sure that the Configure Fault Management checkbox is selected. This checkbox should be set to true for new VBS deployments. 10. Click Next. 11. In the Virtual Business Service configuration summary panel, validate the dependencies and click Finish. 12. In the Results panel, click OK.

19 VBS DR Operations for individual VBS objects Concurrency violation prevention is handled by VCS/GCO. VBS integrates with this feature and will not allow a VBS to be started if the corresponding site still is up. If the site is down/unreachable, VBS will allow start-up. 1. Make sure that Finance_prod_VBS is started. 2. At this point, the Finance_DR_VBS is partially started. For the stake of the lab, offline the NY_IIS_SG from the VOM UI: In the VOM GUI, click Manage > HA-DR > Service Groups. Right-click on the NY_IIS_SG and select Offline. Click OK. 3. Open a Terminal Session to ca-dbsol1 using SSH and issue the following command in each session: # vbssvc -showplan Finance_prod_VBS Note that this cluster has authority for the global service group. 4. Open a Terminal Session to ny-dbsol3 using SSH and issue the following command in each session: # vbssvc -showplan Finance_DR_VBS Examine the difference. The VBS on the NY site cannot go online due to the fact that the VBS is online on the CA site. 5. Offline the Finance_prod_VBS by using VOM GUI or by using CLI from ca-dbsol1/2: # vbssvc -stop Finance_prod_VBS 6. Now, execute the vbssvc -showplan command again: On ca-dbsol1/2: # vbssvc showplan Finance_prod_VBS

20 On ny-dbsol3: # vbssvc showplan Finance_DR_VBS The VBS is now ready to go online on any site. Online the Finance_DR_VBS site by using VOM or CLI, for example: On ny-dbsol3: # vbssvc start Finance_DR_VBS 7. To prepare for the next lab, please switch back the VBS to the California site by using VOM or CLI, for example: On ny-dbsol3: # vbssvc stop Finance_DR_VBS On ca-dbsol1/2: # vbssvc start Finance_prod_VBS Recovery Plan DR operations for the complete data center In the last lab, you learned how to manage DR for individual VBS s. While this is fine for isolated errors, it may not be feasible for a site-wide outage, possible affecting hundreds of VBS s. A VOM 5.0 feature known as the Recovery Plan addresses this use case. The Recovery Plan is a run-book that can be used for disaster recovery as well as server repurposing. In a Recovery plan, certain tasks can be configured. Tasks are executed in the order specified in the Recovery plan. The Recovery Plan supports three types of tasks:

21 1. VBS Start/Stop (can be used global failover) 2. Service Group Start/Stop (can be used for global failover) 3. Custom Script Execution (for example to automatically provision extra CPU/Memory in a disaster recovery scenario) Each task is configured with a set of parameters: Type: Task Name: Action: Critical: Type of object to operate on: VBS, Service Group or Custom Script. Name of the task. What kind of action to trigger: Start, Stop or Execute If a task is marked as critical and it fails, then the recovery plan is stopped during execution and the remaining tasks are not run. If the task is marked as not critical, then even if the task fails, the recovery plan execution is not stopped. Recovery Plan DR operations for the complete data center Creating a Recovery Plan 1. In the VOM UI, click Manage, Recovery Plan. 2. Click Actions, Create Recovery Plan. 3. In the Specify name and description for the Recovery Plan panel, specify Datacenter_Recplan as the name for the Recovery Plan and click Next. 4. In the Specify tasks for Recovery Plan wizard panel, you can add tasks for the Recovery Plan. The objective for this Recovery Plan is to switch over the Finance Multi- Tier Business Service from California to New York. Hence you need to add two tasks: Type VBS Name Action Critical VBS Finance_prod_VBS stop true VBS Finance_DR_VBS start true Note: If a task in a Recovery Plan is marked critical, and it fails, the execution of the rest of the Recovery Plan is aborted.

22 Click Add and select Type, VBS Name, Action and Criticality according to the table above. Repeat the step for each VBS. Click Finish. 5. In the Result panel, verify that the Recovery Plan has been successfully created. Click OK. Running a Recovery Plan In this exercise, we only have a single VBS DR pair. However, in a real-world configuration, the Recovery Plan may contain hundreds of VBS s, Service Groups and/or scripts. Executing all those tasks manually would take significant amount of time. Here, you will demonstrate the capability of bringing up the DR site by executing one single click. 1. Click Manage, Recovery Plan. 2. Right-click on the Datacenter_Recplan Recovery Plan and select Execute Recovery Plan. 3. In the Review tasks for the Recovery Plan panel, confirm the Recovery Plan that you want to run. Click Execute and then quickly OK to get to the Recovery Plan status view. 4. Sit back and relax while the Recovery Plan is bringing up your data center on the DR site! On the Results panel, verify that the Recovery Plan was executed successfully. Click OK. Bonus Lab: Creating shared VBS dependencies Welcome to the Bonus Lab! You have deployed the VBS infrastructure, learned how to manage VBS from VOM and from the command line, understood fault management and also VBS DR. VBS also support shared service groups. A shared service group is shared between two or more VBS entities. VBS applies a specific logic to shared service groups. Startup of a VBS with one or more shared service groups: VBS will start up all service groups in the configured order.

23 Shutdown of a VBS with one or more shared service groups: VBS will stop all non-shared service groups. By default, shared service groups will not be stopped. However, there is an option to offline shared service groups from the VOM UI and from the CLI You will reconfigure the environment according to the picture on the next page. Removing the Sales_VBS VBS and configuring a new shared VBS In this lab, we need to remove the Sales_VBS VBS from the configuration and create a new VBS that shares the bottom-most service group (ca_oracle1).

24 1. In the VOM UI, click Manage > Business Entities > Virtual Business Services. 2. Right-click on the Sales_VBS VBS and click Delete Virtual Business Service. 3. Click Yes to confirm that you want to remove the VBS. Click OK. 4. Click Actions, Create Virtual Business Service. 5. Type in a PR_VBS as Name and click Next. 6. In the Select Base Object Types panel, select Service Groups and click Next. 7. In the Select Base Service Groups panel, select Service Groups the following service groups: ca_oracle1 ca_websphere2 Click Next. Note that this will be the shared service group 8. In the Virtual Business Service Summary panel, click Finish. 9. Right-click on the newly created VBS, click VBS Availability, Configure Service Group Dependencies. 10. In the Virtual Business Service start order panel, configure Service Group Dependencies according to the table below. Select Parent and Child, and click Link. VBS Name PR_VBS Lowest Tier / Child Service Group ca_oracle1 (shared) Dependency Type Restart Depending Tier 1 / Parent Service Group ca_websphere2 11. Make sure that the Configure Fault Management checkbox is selected. This checkbox should be set to true for new VBS deployments. 12. Click Next. 13. In the Virtual Business Service configuration summary panel, validate the dependencies and click Finish. 14. In the Results panel, click OK. Start/Stop of a VBS with shared service groups

25 In this lab, we will validate start and stop behavior when one VBS has a shared service group configured. 1. Stop HR_VBS from the VOM UI. Note the warning about shared service groups. Do not select Offline shared service groups. Only the non-shared service groups will be stopped now. 2. Stop PR_VBS. Select Offline shared service groups (If you prefer CLI operations, you can use vbssvc stop force PR_VBS.) 3. Start HR_VBS and PR_VBS again and confirm that they online correctly. 4. Induce a failure on ca_oracle1 service group (remove the /testfiles/oracle1 lock file as instructed earlier). Note that only the ca_websphere2 service group is restarted. Lab Configuration Information Hostname IP Cluster Name ca-dbsol ca-dbsolclus1 ca-dbsol ca-dbsolclus1 ca-applinux ca-appclus1 ca-webwin CA-WEBWIN1 ny-dbsol ny-applinux ny-webwin ny-dbsolclus2 ny-appclus2 NY-WEBWIN2 vomserver N/A Service Group IP Cluster

26 ClusterService ClusterService ca-dbsolclus1 ny-dbsolclus2 Credentials: vomserver: root/symc4now ca-dbsol1/ca-dbsol2 root/symc4now ca-applinux1/ root/symc4now ca-webwin1 administrator/symc4now ny-dbsol3 ny-applinux2 ny-webwin2 root/symc4now root/symc4now administrator/symc4now