Cisco TEO Resiliency Strategy

Transcription

1 Cisco TEO Resiliency Strategy The purpose of the Cisco TEO Service Resiliency Strategy is to document the basic approach to support both Operational Continuity (OC) and Disaster Recovery (DR) requirements for a service. This high level document contains a required and an optional section to capture information on a Service's production and recovery locations, and the operational continuity (ART, ADL) and disaster recovery objectives (RTO, RPO) defined in the service impact analysis during requirements definition. Disaster recovery elements may not be required for those services which are classified as a local service, or for those which are globally distributed. A successful strategy is based on its ability to cover both operational continuity and disaster recovery requirements. As such, it is imperative that the requirements are properly identified prior to initiating strategy development. This is accomplished by completing either a Business Impact Analysis (BIA) for Business Services, or a Service Impact Analysis (SIA) for IT Foundational and Client Services. The following list shows the standard recovery objectives needed for services. Acceptable Recovery Time - ART Acceptable Data Loss - ADL Recovery Time Objective - RTO Recovery Point Objective - RPO This strategy document is divided into two sections to support Operational Continuity and Disaster Recovery as independent capabilities. The Operational Continuity section captures the information necessary to understand operational continuity requirements. Completion of the following sections are required for all services: Typical Operational Issue Management Handling of Running Processes across Restarts and Failures Handling of Scheduled Triggers across Restarts and Failures Handling of Events across Restarts and Failures The rest of the Operational Continuity section is optional, providing additional areas for a Service team to capture important recovery characteristics that will be helpful when creating the Operational Continuity section of the more detailed Service Resiliency Plan (SRP). The Disaster Recovery section collects the minimum data deemed necessary, in conjunction with the information in the Operational Continuity section, to create a Disaster Recovery Strategy. Completion of Additional Disaster Recovery Information in Disaster Recovery is required only for those services which require Disaster Recovery. The rest of Disaster Recovery is optional, providing additional areas for a Service team to capture important disaster recovery characteristics for the Disaster Recovery section of the SRP (when required). Operational Continuity The purpose of this section is to document the high level operational continuity strategy of this service. Americas Headquarters: Cisco Systems, Inc., 170 West Tasman Drive, San Jose, CA USA 2011 Cisco Systems, Inc. All rights reserved.

2 Typical Operational Issue Management TEO provides full visibility into processes running on the system. This includes visibility into the status of all running processes, which activities have executed, succeeded, failed, etc. These results are available in the Operations workspace. The Operations workspace is used to monitor the processes that are scheduled to execute, view processes that are currently running, and verify that processes have successfully completed. Also, users can start processes or interact with human steps in processes called Tasks (e.g. approvals). Figure 1-1 Operations Workspace The Auditing/System view provides a list of internal TEO events such as notifications of warnings and errors, and information events about the general functioning of the system. TEO provides IT process records such as alerts and incidents. TEO not only uses these records to supply incidents and events for external tools it may be monitoring or managing, but also uses them to provide management of TEO itself. For example, the TEO Core automation pack performs some self-monitoring of TEO, and may raise an alert or incident if there is something which requires the TEO administrator's attention. These IT process records include the alerts, incidents, and change requests types of assigned tasks. These records represent the core records within IT. These are core ITIL (IT Infrastructure Library) records. While TEO does provide a list of these records so that one can, for example, look at a list of alerts which automation created and manage their lifecycle, more often these records are used to synchronize the records out of TEO to a separate tool. For example, alerts typically need to be pushed to an enterprise event manager so that the alerts can be combined with others to present a view of IT system health. Incidents and change requests are typically the domain of service desks. Rather than managing these records in TEO, many customers prefer to manage these records elsewhere. 2

3 Figure 1-2 Synchronization of Alerts, Incidents, and Change Requests to IT Tools This construct allows processes to be independent of the event manager or service desk used. A process such as one in a network best practice automation pack simply creates an alert, incident, or change request. Separately, a Remedy automation pack may include a process which triggers when a new incident assigned task is created, and creates the corresponding entry in the Remedy system. The Remedy automation pack can synchronize data between the systems and when the incident reaches a terminating state in the Remedy system, the incident in TEO is then automatically kept in sync. When the incident in TEO is closed, the process in the network best practice automation pack can continue. With this abstraction, the network best practice automation pack need not be dependent on the specifics of which service desk is involved, and no dependency even exists between the automation packs. Handling of Running Processes across Restarts and Failures By default, TEO persists all data about running processes so that state is preserved across service restarts, be they planned or unexpected. Process state is persisted to the TEO database. While the best approach is a planned shutdown where the TEO server can wait for running activities to complete and avoid problematic situations, persistence preserves execution should an unexpected failure occur. Generally, processes will pick up and start running again after a restart or failure. Many activities can be restarted without side effect. For example, if an activity is passive such a configuration query, TEO can simply run the query again without any side effect. If the TEO server is running one of these during a failure, the process and activity can simply be restarted after a failure and can proceed as if nothing happened. Other activities may have side effects which cannot be re-run without consequence. For example, if a CLI adds some entry for a device to a file, running the CLI again would cause two entries for the device, which may cause a problem. In these cases, activities running at the time of the failure cannot be restarted. TEO shows these processes as failed after the restart. An operator would need to examine the failures and rerun failed processes. TEO offers process Start Points so that process authors can give operators restart points should a process need to be restarted. If a volatile activity is run in TEO and has not returned a status and TEO shuts down, when TEO restarts, this activity will not resume and the results or what they activity was suppose to do will be lost. Adapters determine whether to publish activities as restartable or not. Also, the TEO server allows activities to save state as they run to allow long-running activities to be restarted. For example, a Create Approval activity might save a state that the approval request task has been created but the activity is blocked waiting on an approval. This mechanism generally allows restart of long-running activities. When the TEO server shuts down in a friendly manner, it allows in-flight activities time to complete, but blocks new activities from running. This typically allows activities which may not be marked as restartable to complete so that processes do not fail during friendly shutdowns. An exception to state being written to the database exists for reporting. To improve performance, reporting data is held in queues and occasionally written to the database in bulk rather than an item at a time. During a friendly shutdown, the queue is flushed so that the reporting data is fully synchronized to the reporting database. In the event of a failure, the data in the queue will be present on restart, and it will be sent whenever the TEO Server starts up. No data will be lost. However, should an unexpected failure 3

4 occur for which there is a need to reinstall the TEO server on a different computer or restore the computer hosting the TEO Server from a backup, reporting data from these queues are lost. Again, this should not be the case with a friendly shutdown or even typical unexpected failures. TEO's writing of state is transactional. TEO will not proceed to the next step until the state change is committed to the database. There should be no time gap where process state may be lost. This persistence has a cost. With every step in every process, TEO writes to the database to update states such as when the activity starts, stops, etc. Where this performance hit is inappropriate, TEO provides a setting on a process to turn persistence off. Many processes need not restart after a shutdown or failure. It may often be acceptable to rerun the process at the next scheduled interval or on the next instance of the trigger. For example, many SAP diagnostic processes fall into this category. In the case of SAP diagnostics, TEO's SAP automation pack process definitions are set to turn off persistence on analysis processes. If the server fails, the state of the SAP server with respect to alerts may not be the same after restart anyway. It is often best to not restart these processes but to instead learn anew the state of the SAP system. Also, by turning off persistence, the performance impact of maintaining state is greatly reduced. Handling of Scheduled Triggers across Restarts and Failures Instances of scheduled triggers that would have fired while the server is down will be missed. For example, if I have a process that runs every minute and I leave my TEO server down for one hour, I will have 60 missed instances when I bring it back up. TEO will not attempt to run the process 60 times all at once to "catch up." Note that "repeats" will still work just fine even if the initial firing of the schedule is missed. If something is supposed to happen at noon and then repeat once per hour, just because the server is down at noon doesn't have any effect on the other instances. If the server comes back up at 12:59 pm, the 1 pm "repeat" will still happen. Handling of Events across Restarts and Failures TEO adapters will resume sending events where they left off wherever possible. While implementation in this area is left to the adapter, existing TEO adapter implementations are consistent with this guaranteed delivery design wherever possible. For example, if when the SAP adapter reads CCMS alerts or the Remedy adapter polls for Incident state, they will resume with the next entry when the TEO Server resumes. Clearly for non-guaranteed delivery, network-initiated event technologies like SNMP traps, the TEO Server cannot know about events that occurred while it was down. There are two types of event systems in the TEO Server. The first is triggers. Because of the adapter implementations described in the previous paragraph, event-based triggers will not be lost across server restarts or failures. Like other types of state, the state of which triggers have been correctly submitted to the server is transactional. When TEO adapters send a trigger to the TEO Server, processes depending on that trigger are initiated as a part of the submission, and these process instances are persisted to the database so that after a restart, the triggered processes are running. With the exception of transient triggers, such as SNMP traps or performance thresholds, there should be no time gap where triggers should be lost. The second event system is correlation. These are manifested in the product via "Correlate..." activities such as "Correlate Windows Event." They allow events to be correlated with other data in the workflow. This is achieved by caching data in which the workflows may be interested so that questions the workflow may ask can be answered when the workflow gets to the "Correlate..." activity. The cache of events for correlation is not retained across server restarts. This may cause issues with some "Correlate..." activities. If an event comes in after a TEO Server failure, it would be submitted when the TEO Server resumes and may match the "Correlate..." activity (excepting transient events like SNMP) supposing its time criteria is satisfied. There is a high probability in this case that the time criteria for the correlation may not be satisfied when the server restarts. If the event came in before the TEO Server failure and the "Correlate..." activity is reached after the failure, the event would not be seen since the event cache is cleared on restart. In practice this has been found to not be an issue. First, correlation is a fairly advanced feature in the product and is not used in a majority of scenarios. Second, best practice also dictates that a correlation time window is configured to be fairly small minimizing the issue. Third, these correlations are typically used in diagnostic situations, and often the events repeat if the problem recurs. Where a problem is still occurring, TEO will typically pick up the condition the next time the diagnostic workflow fires. Finally, TEO automation packs tend to set these processes as non-persistent anyway so they are not restarted. If the TEO Server is down, it is often best to get a fresh view of the health of the component being monitored rather than depend on the status before the failure. In many cases the failure is resolved and it just creates noise to record the old diagnosis. 4

5 Technical Infrastructure Drawings Figure 1-3 High Level Architecture Figure 1-4 A Simplified TEO Environment 5

6 Figure 1-5 A Distributed TEO Installation Dependencies The following tables identify critical application data interfaces and indicates whether the dependent service is required in order to be fully operational. Hard Interface is required by the dependent Service to be fully operational Soft Interface is not required by the dependent Service to be fully operational Upstream dependencies IT services this Service relies upon dependencies IT services that rely upon this Service This table indicates which services rely on this interface. Table 1 IT Business Service (Application) Interfaces IT Business Service Name Hard or Soft Upstream / TEO's Internal databases Hard Upstream Active Directory Hard for most enterprise customers. Some international customers without Windows domains leverage local user accounts in the Windows machine hosting the TEO Server. Upstream Comments TEO leverages Windows and AD for security. Users connecting into the server are authenticated against AD or, in the case of local user accounts, against the user accounts on the Windows server hosting the TEO server. Only authenticated users are allowed to access the service. TEO's AD adapter allows TEO to perform actions against AD. SQL Reporting Services Soft Upstream Used for TEO reporting. Reporting is an optional TEO element, and the customer can choose between SQL Reporting Services and Business Objects Reporting. For customers who want reporting functionality, one of these two reporting technologies will be a hard dependency. 6

7 Table 1 IT Business Service (Application) Interfaces IT Business Service Name Business Objects reporting Microsoft Internet Information Services Hard or Soft Upstream / Comments Soft Upstream Used for TEO reporting. Reporting is an optional TEO element, and the customer can choose between SQL Reporting Services and Business Objects Reporting. For customers who want reporting functionality, one of these two reporting technologies will be a hard dependency. Soft Upstream IIS is the web server which supports the TEO Web Console. While the TEO web console is technically an optional component, we expect it to be deployed in most customer scenarios. Table 2 IT Client and Foundation Services IT Service Name Hard or Soft Upstream / Comments Active Directory Hard Upstream TEO leverages Windows and AD for security. Users connecting into the server are authenticated against AD or, in the case of local user accounts, against the user accounts on the Windows server hosting the TEO server. Only authenticated users are allowed to access the service. TEO's AD adapter allows TEO to perform actions against AD. newscale service catalog Soft Upstream In the TCO solution, The newscale service catalog may invoke TEO through its web service interface. Additionally, atomic processes leveraging the web service adapter may call back into newscale to update status and atomic processes leveraging the database adapter may query the newscale database. The interface can primarily be thought of as a downstream Dependency. Tidal Server Provisioning Soft Upstream In the TCO solution, TEO may invoke TSP. Web Services from additional external services Soft Upstream Can be used in a broad variety of integrations. When used, this introduces new upstream dependencies. UNIX / Linux systems Soft Upstream Primarily TEO runs command lines or scripts on these systems. Command lines are also generally useful for a broad set of IT tool integrations. When used for this purpose, this introduces new upstream dependencies. 7

8 Table 2 IT Client and Foundation Services IT Service Name Windows systems which TEO manages Soft Upstream / Soft Upstream / SNMP Soft Upsream / Devices via SSH/Telnet Soft Upstream Cisco UCS Manager Soft Upsream / VMware ESX, ESXi, vcenter Hard or Soft Upstream / Soft Upsream / Remedy Service Desk Soft Upstream / Oracle Database Soft Upstream SQL Server Database Soft Upstream Comments The heaviest actions against these systems are command line or script invocation, but TEO additionally supports other methods such as registry access, performance counters, and service queries. Command lines are also generally useful for a broad set of IT tool integrations. When used for this purpose, this introduces new downstream dependencies. TEO primarily invokes actions Windows system as an upstream element. However, Windows events can trigger TEO processes, which can be thought of as an downstream dependency. TEO queries for these triggers, so the windows system is not strictly dependent on the TEO service. TEO uses to send notifications from within processes. Incoming s can trigger TEO processes, which can be thought of as an downstream dependency. TEO watches a mailbox for these triggers, so the system is not strictly dependent on the TEO service. Additionally, is generally useful for a broad set of IT tool integrations. When used for this purpose, this introduces new dependencies. SNMP is also generally useful for a broad set of IT tool integrations. When used for this purpose, this introduces new downstream dependencies. Incoming SNMP traps can trigger TEO processes, which can be thought of as an downstream dependency. However, the system sending the SNMP traps is not strictly dependent on the TEO service. TEO primarily invokes actions on the Cisco UCS Manager as a downstream element. However, UCS Faults can trigger TEO processes, which can be thought of as a downstream dependency. TEO queries for these triggers, so the UCS Manager is not strictly dependent on the TEO service. TEO primarily invokes actions against the VMware systems. However, Virtual machine power events and host or virtual machine performance events can trigger TEO processes, which can be thought of as an downstream dependency. TEO queries for these triggers, so VMware is not strictly dependent on the TEO service. TEO primarily invokes actions against Remedy. However, Remedy incidents which are created or updated can trigger TEO processes, which can be thought of as an downstream dependency. TEO queries for these triggers, so the Remedy system is not strictly dependent on the TEO service. 8

9 Table 2 IT Client and Foundation Services IT Service Name DB2 Database Soft Upstream Generic Database Soft Upstream SAP Soft Upstream / Microsoft System Center Operations Manager Hard or Soft Upstream / Soft Upstream / Comments TEO primarily invokes actions against the SAP systems. However, SAP events can trigger TEO processes, which can be thought of as a downdtream dependency. TEO queries for these triggers, so the SAP system is not strictly dependent on the TEO service. The TEO SCOM adapter primarily triggers TEO processes from incoming SCOM alerts, which can be thought of as an downstream activity. TEO queries for these triggers, so the SCOM system is not strictly dependent on the TEO service. However, The SCOM adapter can also update SCOM alerts. Table 3 External Interfaces External Dependency Tidal Server Provisioner (LinMin) Upstream / Vendor Comments Upstream LinMin The TSO and TCO solutions integrate TEO with this OEM product. newscale Upstream newscale The TCO solution integrates TEO with this OEM product. In the TCO solution, The newscale service catalog may invoke TEO through its web service interface. Additionally, atomic processes leveraging the web service adapter may call back into newscale to update status and atomic processes leveraging the database adapter may query the newscale database. The interface can primarily be thought of as a downstream Dependency. Database Strategy Virtually all TEO state information is stored in the databases. We recommend that customers follow standard database backup procedures for data recovery and follow the best practice database resiliency strategies of their database vendors. These database resiliency strategies should be provided in the documentation of your chosen database platform. Critical platforms include: Microsoft SQL Server Oracle Data / Information Recovery Strategy Virtually all TEO state information is stored in the databases. We recommend that customers follow standard database backup procedures for data recovery and follow the best practice database resiliency strategies of their database vendors. These database resiliency strategies should be provided in the documentation of your chosen database platform. Some information is stored on the TEO server. Most of this relates to the credentials used for TEO. A simple backup of the TEO server will generally suffice to facilitate information recovery. See What Information is held in the TEO Server System, page 10 and Saving Off TEO Server Data in Preparation for a Disaster, page 11 for additional information. 9

10 Personnel Strategy We recommend that a team be assigned for TEO administration and operations. While TEO may be useful across a broad span of IT users, it needs a dedicated administrator/operator. This operator would use the Operations workspace to monitor the health of jobs and would receive operational alerts. They would typically serve as the primary contact for customer support issues. Disaster Recovery High-Level Strategy The purpose of this section is to document the high level strategy of this service. Achieving high availability with TEO consists of two elements. The first is the database, for which the recommendation is that customers follow high availability standard procedures of Microsoft SQL Server or, in 2.2, Oracle. The second element is the TEO server. Our recommended approach for high availability of the TEO server is virtualization. Cisco recommends that the TEO Server be installed on a virtual machine backed by networked storage. With this installation, if a host fails the VM can be migrated by tools like vcenter to move to a new host. Many customers find this method of fault tolerance preferable since the host can change to another site in disaster recovery situations. TEO was designed with this emerging pattern of High Availability in mind. Figure 1-6 High Availability Within VMware, one would configure TEO as a VMware HA Cluster. See for more on VMware's support of High Availability. With the virtual machine-based approach, if there is a need to fall back to a backup the process is quick and simple leveraging snapshots. It is very easy to bring back a known good instance of a server from a snapshot. The approach minimizes downtime and can reduce the risks and recovery times associated with patching virtual machines by allowing fast rollbacks to a pre-patch stage. VMware also offers tools for minimizing downtime when applying patches. VM farms tend to be sets of machines connected to the same storage. They may be within the same data center. You may want to replicate your snapshots to a secondary data centers. The TEO team is in development of a hot standby solution for more extreme disaster recovery scenarios. Additional Disaster Recovery Information 1. Contacting Cisco Tidal Enterprise Orchestrator Support The first step of any disaster recovery should be to contact Cisco TEO support. They can ensure that the latest procedures are followed and minimize downtime through their expertise. To minimize downtime, they should be contacted as early as possible and involved throughout the process. Do not wait to call them until the disaster recovery process goes astray. 2. What Information is held in the TEO Server System 10

11 While the bulk of the TEO Server's state is held in the database, there are a few important pieces of data held in the TEO Sever system. They include: a. The TEO Server installation itself. Nothing special here -- you could install the TEO Sever from the CD and get this back. b. The key used for encryption of security credentials is stored in the Windows security subsystem of the Windows server hosting the TEO server. Via the runtime user construct, TEO stores user names, passwords, and other credentials so that it can connect with those credentials when running an activity against some target. Credentials are encrypted with a key specific to the Windows server hosting the TEO server before they are stored in the database. The server-specific key, stored within the Windows security system, can be exported as a file, retained for disaster recovery purposes, and moved to a standby/backup server for recovery. c. The server configuration file. This file configures certain aspects of the TEO Server. For example, it tells the TEO server what ports to open for talking to the client. It tells the server where the database is and which credentials to use to access it. d. Persistent queue files for handling events. These files store events so that they are not lost across server restarts. See section "Handling of Scheduled Triggers across Restarts and Failures" and section "Handling of Events across Restarts and Failures" above. 3. Saving Off TEO Server Data in Preparation for a Disaster The most important piece of information that you want to save is the security key that the TEO server uses to encrypt sensitive data before storing them to the database. The key, stored within the Windows security system, can be exported as a file, retained for disaster recovery purposes, and moved to a standby/backup server for recovery. To export the key to a file, run the following command: aspnet_regiis px Tidal Intelligent Automation Server Keys.xml -pri Note *Aspnet_regiis.exe is a Microsoft.NET framework utility. You may need to specify the full path to it if you don t have the path in the Path environment variable. Another piece of data we want to save off is the TEO server configuration file. This file configures certain aspects of the TEO server. For example, it tells the TEO server what ports to open for talking to the client. It tells the server where the database is and which credentials to use to access it. The file can be found in the installation folder. %INSTALLDIR%\Tidal.Automation.Server.exe.config 4. TEO Server Relocation Procedures The procedure to relocate a TEO server to another machine is exactly the same as the procedures to create and activate a standby TEO server. 5. Hot TEO Server Standby Strategy There are reasons to maintain a backup TEO Server for disaster recovery scenarios. This is especially true where customers chose to run the TEO Server on physical servers. If you have a critical server failure where a restore does not work, or if you have a datacenter failure, you may also want to have a hot standby TEO server (for example in a second data center). To create a standby TEO server, perform the following steps: a. Install the TEO server on a new machine(point to the existing database). Note If the TEO service is able to restart, shut it down. There is still work to be done. b. Import the TEO security key to the new server machine. aspnet_regiis -pi "Tidal Intelligent Automation Server" Keys.xml c. Replace the server configuration file on the new machine with the file copied from the existing TEO server machine. Note If the file contents contains any path (such as the log file path), make sure that the path is still valid in the new server machine. d. Install a new hot standby Web Console on a new machine. This can be tested against the old TEO server. 11

12 e. Update the Web Console configuration file for the new hot standby Web Console to point to the new TEO server. f. Replicate all the TEO local user groups and their members from the existing TEO server machine to the new machine. 6. Activating the Standby TEO Server When an actual disaster occurs, perform the following steps. The syntax might be different if the TEO Process database is on an Oracle server. a. Verify that both TEO server on both machines are shut down. b. Copy all the data files to the new machine. These files store events so that they are not lost across server restarts. They can be found in the following directory. %SystemRoot%\System32\config\systemprofile\AppData\Local\Cisco\Tidal Enterprise Orchestrator\Data c. Run the following query to update the TEO Server machine name stored in the database. The TEO Process database contains the name of the active TEO server machine. This machine name needs to be updated to the new TEO server machine name. USE TEOProcess GO UPDATE [dbo].[version] SET AppServer = <New TEO Server NetBios Name> GO Note If the file contents contains any path (such as the log file path), make sure that the path is still valid in the new server machine. d. Run the following query to update the default automation Windows target to point to the new TEO server. USE TEOProcess GO UPDATE [dbo].[configuration] SET ConfigurationData.modify('replace value of (/TargetConfiguration/NetBiosName/text())[1] with <New TEO Server NetBios Name> ') WHERE ConfigurationId = 'F4DB990A-0C EE-A ' GO Note The syntax might be different if TEO Process database is on an Oracle server. e. Review and update the automation summary configuration. We advise customers to store TEO automation summaries on a share or web site, however if customers choose to store them locally on the TEO server machine, they may want to replicate the TEO automation summaries to the new TEO server machine. f. Restart the TEO server on the standby machine. It is now an active TEO server. 12

13 Database Strategy Virtually all TEO state information is stored in the databases. We recommend that customers follow standard database backup procedures for data recovery and follow the best practice database resiliency strategies of their database vendors. These database resiliency strategies should be provided in the documentation of your chosen database platform. Critical platforms include: Microsoft SQL Server Oracle Hardware Strategy - Disaster Recovery Server Footprint Provide any additional information on how server capacity and performance requirements will be provisioned at time of disaster (ATOD). Is a disaster recovery environment already provisioned and available? Do any servers need to be re-built or re-purposed on declaration? Are Provision on Demand (PoD) servers identified? Do PoD servers have pre-allocated rack space in the recovery site? The hardware strategy for database disaster recovery should follow the best practice of that database. TEO recommends that customer use virtual machines to address disaster recovery. This allows an abstraction to allow TEO to be independent of physical server failures. VM farms tend to be sets of machines connected to the same storage. They may be within the same data center. You may want to replicate your snapshots to a secondary data centers. With that said, there are some reasons to maintain a backup TEO Server for disaster recovery scenarios. This is especially true where customers chose to run on physical servers. In the event that you have a critical server failure where a restore does not work, or in the event that you have a datacenter failure, you may want to have a hot standby TEO server (for example in a second data center. See Hot TEO Server Standby Strategy, page 11. Data / Information Recovery Strategy The Data / Information strategy for database disaster recovery should follow the best practice of that database. Since virtually all state for TEO is retained in the TEO database, the data / information recovery strategy is, for the most part, a database recovery issue. Backups or VM snapshots are the principal means of restoring information on a TEO Server. See What Information is held in the TEO Server System, page 10 and Saving Off TEO Server Data in Preparation for a Disaster, page 11 for information that may be lost on a TEO Server crash. Personnel Strategy Recovery and Operations This strategy ensures recovery teams are in place at the recovery site. Recovery The Cisco TEO Support team should be contacted The team assigned for TEO administration and operations. See Personnel Strategy, page 10. If the disaster is with the TEO database, the DBA team responsible for the database. The manager for the VM farm on which the TEO Server runs. Personnel who can run the restore software to restore the TEO Server. This may be the same as the VM farm operator. Operations The Cisco TEO Support team should be contacted The team assigned for TEO administration and operations. See Personnel Strategy, page 10. Risks to Recovery This section identifies any risks to the recovery strategy in meeting recovery requirements. Failure to contact Cisco TEO support immediately after the disaster Lack of a clear TEO administrator and clear ownership Failure to perform proactive steps beforehand to prepare for a disaster as outlined above. 13

14 Glossary For additional glossary definitions, see IT Service Continuity Resiliency Glossary. Acceptable Data Loss (ADL): The acceptable amount of data that can be lost without invoking DR, expressed as a duration in time. Acceptable Recovery Time (ART): The maximum time the services can manage without the invocation of DR, measured from the moment the service down until the service is restored. Actual Objectives: Achievable recovery objectives (ART, ADL, RTO, RPO, Criticality Level) based on current capabilities. Business Impact Analysis (BIA): A process designed to prioritize business functions by assessing the potential quantitative (financial) and qualitative (non-financial) impact that might result if an organization was to experience a business continuity event. The BIA also defines the Operational Continuity objectives (ART, ADL) and Disaster Recovery objectives (RTO, RPO). Business Objectives: Derived or calculated recovery objectives (ART, ADL, RTO, RPO, Criticality Level) based on client requirements and expectations. Business Service: A GBP-owned application provided to support one or more Business Processes. A Business Service is made up from a combination of people, processes, software and technology and should be defined in a Service Level Agreement. Disaster Recovery: Catastrophic failure of production, leading to recovery at a remote site. Disaster declaration occurs, invoking Disaster Recovery Plans in support of the RTO and RPO objectives. Global Business Process (GBP): An event-driven, end-to-end processing path that starts with an internal or external request and ends with a result for the customer. Business processes are often cross-departmental and can be enterprise wide. Globally Distributed Service: An IT Service (Foundational, Client, or Business) which runs concurrently in multiple production locations (Active/Active), or with a duplicate environment ready to assume production at any time with no data loss (Active/Standby). Each location can serve as the production location, and when one location becomes unavailable, production can continue or resume at the other location. IT Service: An identifiable technology related solution with well defined functional and operational characteristics. An IT Service either directly enables Business Capabilities (IT Client Services) or indirectly enables them by providing necessary underlying functionality (IT Foundational Services). Local Service: An IT Service (Foundational, Client, or Business) deployed in a specific location(s) in support of clients within that location only. Deployments are considered autonomous, so that an outage in one location does not affect deployments in other locations. Operational Continuity: Production outage which initiates the operational continuity plan(s) to support the ART and ADL objectives. This resumption does not declare a disaster and is performed solely by IT Operations. Recovery Point Objective (RPO): The maximum amount of data that may be lost when Service is Restored after an interruption. Recovery Point Objective is expressed as a length of time before the Failure. For example a Recovery Point Objective of one day may be supported by daily Backups, and up to 24 hours of data may be lost. Recovery Point Objectives for each IT Service should be negotiated, agreed and documented, and used as Requirements for Service Design and IT Service Continuity Plans. Recovery Time Objective (RTO): The maximum time allowed for recovery of an IT Service following an interruption. The Service Level to be provided may be less than normal Service Level Targets. Recovery Time Objectives for each IT Service should be negotiated, agreed and documented. Service Impact Analysis (SIA): Activity in IT Service Continuity Management that identifies Vital Business Functions and other IT Services that are dependent upon this IT Service. SIA defines the recovery requirements for the IT Service. These requirements include Recovery Time Objectives, Recovery Point Objectives, Recovery Capability Objectives and minimum Service Level Targets for each IT Service. Service Resiliency Plan (SRP): The management approved document that defines the resources, actions, tasks and data required to manage the technology recovery effort. This is a component of the Business Continuity Management Program. Service Resiliency Strategy (SRS): The basic approach to support both operational and recovery requirements. The Service Resiliency Strategy is a high level document which contains both required and optional sections to capture information on a Service's production and recovery locations, and capture Operational Continuity objectives (ART, ADL) and Disaster Recovery objectives (RTO, RPO) defined in the Service Impact Analysis (SIA). Printed in the USA on recycled paper containing 10% postconsumer waste. 14