EMC SourceOne Version 7.1

Transcription

1 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide REV 01 EMC Corporation Corporate Headquarters: Hopkinton, MA

2 Copyright 2013 EMC Corporation. All rights reserved. Published September, 2013 EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All other trademarks used herein are the property of their respective owners. 2 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

3 Contents Preface Chapter 1 Chapter 2 Solution Overview Executive summary About replication technology...12 Supported products...13 Key components...13 EMC SourceOne architecture Product overview Data sources...15 EMC SourceOne services...16 Native Archive host computer(s)...17 Native Archive storage...18 SQL Server and databases...20 Applications...20 Solution Configuration Details Hot/cold disaster recovery architecture example Hot/warm disaster recovery architecture example Considerations...24 Component disaster recovery configuration Console application...26 Messaging environment...27 Microsoft SharePoint environment...29 File system environment...31 Master computer...32 Worker computer...33 Web server...35 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide 3

4 Contents SQL Server and databases Native Archive host computer Native Archive - Message Center storage Native Archive - Index storage Native Archive - Container storage File server host or NAS device configuration EMC Centera Chapter 3 Failover to the Disaster Recovery site Monitoring the system Monitoring Symmetrix SRDF status - Normal status Monitoring Symmetrix SRDF status - Failure status Monitoring MirrorView status Monitoring RecoverPoint status Failover procedures Step 1: Stop EMC SourceOne processing at disaster recovery site Step 2: Check that replicated data is accessible Step 3: Verify disk health of file servers Step 4: Perform Active Directory changes for SQL and file servers Step 5: Ensure dependent systems are online and accessible.. 57 Step 6: Recover the DiskXtender configuration Step 7: Power on EMC SourceOne computers (hot/cold configuration only) Step 8: Ensure the EMC SourceOne computers can access key resources Step 9: Run the EMC SourceOne Archive Integrity Tool Step 10: Start EMC SourceOne services Step 11: Perform reprocessing Step 12: Rerun the EMC SourceOne Archive Integrity Tool 66 Step 13: Verify EMC SourceOne operations Step 14: (Optional) Run the SharePoint EBS integrity tool EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

5 Contents Chapter 4 Appendix A Failback to the Primary Site Failback procedures Step 1: Stop EMC SourceOne processing at the disaster recovery site...71 Step 2: Ensure the exported DiskXtender registry keys are available...71 Step 3: Copy data back to the primary site...71 Step 4: Re-establish disaster recovery configuration for EMC SourceOne computers...72 Step 5: Create a "gold copy" of replicated data (recommended)...72 Step 6: Enable storage devices at the primary site...73 Step 7: Perform Active Directory changes for SQL and file servers...73 Step 8: Ensure dependent systems are online and accessible Step 9: Recover the DiskXender configuration...74 Step 10: Start EMC SourceOne services...79 Step 11: Verify primary site is replicating to the disaster recovery site...79 Step 12: Verify EMC SourceOne operations...79 EMC SourceOne Archive Integrity Tool Operations Overview Command line description Syntax Required parameters Optional parameters Examples Permissions Sequence of operations Parse command line options Add volumes to the candidate volume set Add objects found to the candidate object set Add full-text indexes found to the candidate index set Remove objects from SQL Verifying operations Query and display results Query and retrieve results Log file viewing EMC SourceOne Version 7.1 Disaster Recovery Solution Guide 5

6 Contents Appendix B EMC SourceOne EBS Integrity Tool Operations Overview Recovery scenarios BLOB records missing from EBS database, BLOBs exist in Native Archive BLOBs missing from Native Archive, BLOB records exist in EBS database BLOBs missing from Native Archive and EBS database Using the EBS Integrity Tool Launching the tool Reviewing scan results Recovering BLOBs Logging Glossary 6 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

7 Preface As part of an effort to improve its product lines, EMC periodically releases revisions of its software and hardware. Therefore, some functions described in this document might not be supported by all versions of the software or hardware currently in use. The product release notes provide the most up-to-date information on product features. Contact your EMC technical support professional if a product does not function properly or does not function as described in this document. Note: This document was accurate at publication time. Go to EMC Online Support ( to ensure that you are using the latest version of this document. Purpose Audience Related documentation This document describes how to implement a disaster recovery configuration for supported EMC SourceOne products. This manual is intended for the person implementing the disaster recovery solution and executing the recovery procedures. The person must have good knowledge of EMC SourceOne products, EMC DiskXtender (if used), Microsoft SQL Server, storage and replication technology and practices. Refer to the EMC SourceOne documentation posted on EMC Online Support ( EMC SourceOne Version 7.1 Disaster Recovery Solution Guide 7

8 Preface Conventions used in this document EMC uses the following conventions for special notices: Note: A note presents information that is important, but not hazard-related.! CAUTION A caution notice contains information essential to avoid data loss or damage to the system or equipment.! IMPORTANT An important notice contains information essential to software or hardware operation. Typographical conventions EMC uses the following type style conventions in this document. Normal Used in running (nonprocedural) text for: Names of interface elements (such as names of windows, dialog boxes, buttons, fields, and menus) Names of resources, attributes, pools, Boolean expressions, buttons, DQL statements, keywords, clauses, environment variables, functions, utilities URLs, pathnames, filenames, directory names, computer names, filenames, links, groups, service keys, file systems, notifications Bold Used in running (nonprocedural) text for: Names of commands, daemons, options, programs, processes, services, applications, utilities, kernels, notifications, system calls, man pages Used in procedures for: Names of interface elements (such as names of windows, dialog boxes, buttons, fields, and menus) What user specifically selects, clicks, presses, or types Italic Used in all text (including procedures) for: Full titles of publications referenced in text Emphasis (for example a new term) Variables Courier Used for: System output, such as an error message or script URLs, complete paths, filenames, prompts, and syntax when shown outside of running text 8 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

9 Preface Courier bold Used for: Specific user input (such as commands) Courier italic Used in procedures for: Variables on command line User input variables < > Angle brackets enclose parameter or variable values supplied by the user [ ] Square brackets enclose optional values Vertical bar indicates alternate selections - the bar means or { } Braces indicate content that you must specify (that is, x or y or z)... Ellipses indicate nonessential information omitted from the example Where to get help Your comments EMC support, product, and licensing information can be obtained as follows. Product information For documentation, release notes, software updates, or information about EMC products, go to EMC Online Support at: Technical support Go to EMC Online Support and click Service Center. You will see several options for contacting EMC Technical Support. Note that to open a service request, you must have a valid support agreement. Contact your EMC sales representative for details about obtaining a valid support agreement or with questions about your account. Online communities Visit EMC Community Network at for peer contacts, conversations, and content on product support and solutions. Interactively engage online with customers, partners, and certified professionals for all EMC products. Your suggestions will help us continue to improve the accuracy, organization, and overall quality of the user publications. Send your opinions of this document to: BRSdocumentation@emc.com EMC SourceOne Version 7.1 Disaster Recovery Solution Guide 9

10 Preface 10 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

11 1 Solution Overview This document describes a reference implementation for a disaster recovery solution that will support EMC SourceOne products that have been qualified with this solution. This chapter includes: Executive summary EMC SourceOne architecture Product overview Solution Overview 11

12 Solution Overview Executive summary This document contains a detailed configuration for an EMC SourceOne deployment that is configured to support disaster recovery. This information is intended to provide a proven configuration for deployment of a disaster recovery environment. This information in this document is framed in a timeline-based structure, starting with procedures to plan and implement the hardware, software and configuration settings to support disaster recovery, the actions taken to fail over to a secondary site following a disaster, and finally, the actions taken to return processing to the primary site once the disaster situation is resolved. This document provides best practices for setting up a manageable disaster recovery solution, without specific regard to the replication technology. The assumption is that a consistent replication scenario is established and, with the assistance of utilities and best practices provided by EMC, the customer is able to recover from a failure in a reasonable timeframe with minimal data loss. About replication technology The solution documented in this guide is based on testing conducted using an array-based replication technology (specifically EMC Mirrorview using the EMC CLARiiON storage system). The use of an array-based solution as described in this document does not preclude the use of other replication technologies from EMC or other vendors, (or even NAS, host-based, software, or backup solutions) to make the key data available at a secondary site. The important information to extrapolate from this document are the procedures and practices for configuring, failing over to a secondary site, and failing back to the original site. Therefore, although this information is framed in an array-based scenario, the focus of this document is not on how the data important to EMC SourceOne is replicated and presented at the other site, but on how the data is used once a disaster scenario occurs. It is up to the implementation team to determine the specific means of making the data available at the disaster recovery site. 12 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

13 Solution Overview Supported products This guide provides a disaster recovery solution for the following EMC SourceOne products: EMC SourceOne Management version 6.5 and above (this product provides the core infrastructure required for all EMC SourceOne products). EMC SourceOne for Microsoft SharePoint version 6.6 EMC SourceOne for File Systems version 6.6 Key components The key components considered in this solution include: EMC SourceOne computers (master, worker, and Native Archive servers) EMC SourceOne data storage locations (file servers and storage devices) Microsoft SQL Server and databases EMC DiskXtender configuration (if used) EMC Centera configuration (if used) Data source configuration: Mail server configuration (EMC SourceOne Management) SharePoint configuration (EMC SourceOne for Microsoft SharePoint) File system configuration (EMC SourceOne for File Systems) The information in this document is intended to supplement other documentation and training material such as EMC SourceOne product installation guides and other documents providing guidelines for various reference configurations. This document discusses the hardware configuration, the software configuration, the infrastructure and the tools and methodologies used to obtain the results. Where applicable, certain proven practices and other configuration settings are noted which affected the results. Executive summary 13

14 Solution Overview EMC SourceOne architecture EMC SourceOne is a distributed set of applications which can run on a single physical computer, or across many computers, as shown in the following figure. Figure 1 EMC SourceOne architecture 14 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

15 Solution Overview Product overview EMC SourceOne products provide an enterprise solution you can use to reduce the space required for your , Microsoft SharePoint servers, and file servers, and to aid in complying with legal requirements for archiving content and maintain efficient storage. A high-level overview of components comprising the EMC SourceOne system include: Data sources - Can include , Microsoft SharePoint and file systems EMC SourceOne services - Includes master, worker, Native Archive and Web Services computers. SQL Server and databases - Includes Microsoft SQL Server and EMC SourceOne databases. Content archives and storage - Includes the Native Archive and associated storage locations used by EMC SourceOne. Applications - Includes the console application used to administer EMC SourceOne. Data sources EMC SourceOne archives, retains and organizes content from Microsoft Exchange, IBM Lotus Domino, SMTP mail servers, Microsoft SharePoint, and file servers. Archived content is stored in the EMC SourceOne Native Archive, and can be searched by administrators and end users with appropriate permissions. Management environments EMC SourceOne Management supports the archiving of messaging content to enable companies to meet storage management, compliance, and legal discovery requirements. In Microsoft Exchange environments, EMC SourceOne Management supports real-time archiving of messages from Exchange mail servers using native journaling functionality provided in Exchange. You can also perform historical archiving and shortcutting of messages from Exchange mailboxes and historical archiving of Outlook PST files. Product overview 15

16 Solution Overview In IBM Lotus Domino environments, EMC SourceOne Management supports real-time archiving and shortcutting of messages from Domino mail servers by installing EMC SourceOne Domino Extensions on Domino mail servers. You can also perform historical archiving and shortcutting of messages from Domino mailboxes and Notes NSF files. EMC SourceOne for Microsoft SharePoint environments EMC SourceOne for Microsoft SharePoint addresses long-term or specialized content archiving strategies while leveraging the EMC SourceOne architecture to provide access to archived content using search user interfaces. EMC SourceOne for File Systems environments EMC SourceOne for File Systems enables your organization to gain control over unmanaged information residing on file share systems. By archiving content from file servers across the organization, administrators and end users can leverage content indexing and search capabilities provided by EMC SourceOne to support information governance, compliance discovery and data protection. EMC SourceOne services The EMC SourceOne services software includes Master Services, Worker Services, Web Services, and Mobile Services software running on one or more computers. The EMC SourceOne services architecture is scalable, supporting the installation of all components on a single physical computer or allowing you to distribute them across multiple physical computers depending on the requirements of your environment. Master computers The EMC SourceOne master computer schedules and distributes jobs that are processed by workers. 16 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

17 Solution Overview Worker computers EMC SourceOne worker computers perform archiving activities from various data sources. Each worker computer can be configured to process only specific activities. EMC SourceOne Web Services - This software can be installed on one or more Worker Services computers to support underlying Web functions such as fielding search requests and performing shortcut resolution. EMC SourceOne Mobile Services - This software provides Universal URL support for EMC SourceOne Management users, specifically: Mobile users Outlook Web Access users Outlook users who do not have the EMC SourceOne Offline Access installed. For EMC SourceOne for File Systems users, the Universal URL also provides support for resolution of links to files in the Native Archive which were archived from a file server. Native Archive host computer(s) The EMC SourceOne Native Archive software uses an architecture which can be configured to perform the following roles: Archive Index Search Retrieve The Native Archive software can be installed on a single host computer or multiple host computers, enabling you to dedicate hardware to specific roles (archive, index, search, and retrieve) to match your organization's archiving policies. Product overview 17

18 Solution Overview Native Archive storage Message Center storage Index Area storage Container Storage Storage locations include: Message Center storage Index Area storage Container file storage Work (unpack area) storage A location where content is stored and built into volumes before they are written to a container file. A location where full-text indexes are stored. Location for long-term storage of containers. Storage scenarios include: Network-accessible storage on page 18 Direct storage on EMC Centera on page 18 Storage on EMC Centera using EMC DiskXtender on page 19 Additional storage options on page 19 Network-accessible storage You can store containers directly on network-accessible storage, such as shareable drives or network-attached storage devices (NAS devices). Direct storage on EMC Centera Your EMC SourceOne configuration may require the long-term storage of EMC SourceOne container files on an EMC Centera content-addressed storage (CAS) system. When using EMC SourceOne Management version 6.5 SP2 and above, you can store containers on EMC Centera devices directly accessed by EMC SourceOne Management (without the use of DiskXtender). 18 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

19 Solution Overview Storage on EMC Centera using EMC DiskXtender EMC SourceOne Management versions prior to version 6.5 SP2 required the use of EMC DiskXtender as a means to connect to EMC Centera. EMC DiskXtender for Windows (along with the EMC Centera media service) facilitates the communication between DiskXtender and Centera. If you have existing archive folders configured to store containers on EMC Centera using DiskXtender, you must continue to use this solution for those folders. After upgrading to EMC SourceOne Management to version 6.5 SP2, you can configure any new archive folders that you create to utilize direct storage on EMC Centera. Note: The scenario of using EMC DiskXtender connected to other long-term storage such as optical devices or tape, while valid, is not considered for the purposes of this document. DiskXtender is only described as a means of connecting EMC SourceOne to Centera. Additional storage options EMC SourceOne Management versions 6.6 SP1 and above support direct storage on the following NAS devices: EMC Celerra EMC Data Domain NetApp For details, refer to the Configuring Archives chapter of the EMC SourceOne Management Administration Guide. Work (unpack area) storage A temporary location in which content ready for indexing is stored. Product overview 19

20 Solution Overview SharePoint External BLOB storage EMC SourceOne for Microsoft SharePoint also provides External binary large object (BLOB) storage, effectively offloading the storage of binary large objects from the SharePoint SQL Server and storing them using EMC SourceOne. EBS is configured at the farm level. This feature includes an EBS Provider database which is installed on the SharePoint SQL Server to maintain identification information for BLOBs stored externally in the Native Archive by EMC SourceOne for Microsoft SharePoint. EBS simply utilizes the Native Archive to access storage EBS content is not archived. Therefore, EBS content is not subject to indexing, retention or available for search using EMC SourceOne Search. However, it is available for search using SharePoint-side search interfaces. SQL Server and databases A database server running Microsoft SQL Server supports the following EMC SourceOne databases: Activity database - This database maintains data associated with EMC SourceOne system processing. Search database - This database maintains data associated with the EMC SourceOne Search application. Native Archive database - This database manages data associated with the EMC SourceOne Native Archive. Applications The EMC SourceOne console enables administrators to configure and manage the product. User applications include EMC SourceOne Search and EMC SourceOne Offline Access. 20 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

21 2 Solution Configuration Details The components which comprise the EMC SourceOne environment are described in the previous section. This chapter expands on this configuration to include the addition of a second site which provides a supportable disaster recovery configuration. This section includes: Hot/cold disaster recovery architecture example Hot/warm disaster recovery architecture example Component disaster recovery configuration Solution Configuration Details 21

22 Solution Configuration Details Hot/cold disaster recovery architecture example The following figure depicts an Management scenario which uses array-based replication for EMC SourceOne data and SQL databases, vendor-recommended replication for the mail environment, and EMC Centera replication. In this configuration, a subset of systems at the secondary site are not active. Note: Although the mail server in Figure 2 is shown as active at both sites, its actual status depends on the specific disaster recovery configuration used. Figure 2 Disaster recovery configuration - hot/cold 22 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

23 Solution Configuration Details Hot/warm disaster recovery architecture example The hot/warm configuration is similar to the hot/cold configuration in terms of replication. However, EMC SourceOne Native Archive computers and Workers are active at the secondary site to provide additional processing capabilities in support of the primary site over the WAN. This is termed a "hot/warm" configuration because some failover steps must be performed to make the secondary site fully operational. Differences in the configuration, failover and failback steps for hot/warm and hot/cold architectures are elaborated throughout this document. Figure 3 Disaster recovery configuration - hot/warm Hot/warm disaster recovery architecture example 23

24 Solution Configuration Details Considerations Specific best practices for using workers and Native Archive computers at the disaster recovery site in a hot/warm configuration include: During regular operation, you can limit the roles on the secondary Native Archive and worker computers to ones which would generate the least WAN traffic. Roles are configured using the EMC SourceOne console application. Deciding whether to limit certain processing on computers at the disaster recovery site depends on the speed and technology used for the WAN. A larger WAN pipe may accommodate the large amount of traffic generated by certain EMC SourceOne activities while a WAN connection with limited bandwidth cannot. Examples of activities that you may want to limit from processing across the WAN include indexing, journaling, and large-scale historical archiving activities. Examples of activities which can be processed across the WAN include search and limited historical archiving activities. Note: Determining the optimum configuration for a hot/warm configuration is up to the implementation team, as dependencies and considerations exist which must be evaluated on a site-specific basis. Disaster recovery of your mail system environment itself is beyond the scope of this document. Although the mail server in Figure 3 is shown as active at both sites, its actual status depends on the specific disaster recovery configuration used. Refer to Messaging environment on page 27 for more details. 24 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

25 Solution Configuration Details Disaster recovery of your SharePoint or file systems is beyond the scope of this document. Though not depicted in the preceding diagrams, EMC SourceOne for Microsoft SharePoint and EMC SourceOne for File Systems installations use the same underlying structure (e.g., master computer, worker computers, Native Archive computers, SQL Server and databases, and data storage). Refer to the following sections for more details: Microsoft SharePoint environment on page 29 File system environment on page 31 Hot/warm disaster recovery architecture example 25

26 Solution Configuration Details Component disaster recovery configuration The following sections provide details about the disaster recovery configuration for each component in the EMC SourceOne environment: Console application Messaging environment Microsoft SharePoint environment File system environment Master computer Worker computer Web server SQL Server and databases Native Archive host computer Native Archive - Message Center storage Native Archive - Index storage Native Archive - Container storage File server host or NAS device configuration EMC Centera Console application The EMC SourceOne console enables administrators to configure and manage the product. Disaster recovery configuration Install the console application on at least one computer with access to other computers at the disaster recovery site. If you use the Microsoft SharePoint or File Systems products, you must also install the corresponding Business Component Extensions (BCE) software: EMC SourceOne Business Component Extensions (BCE) for Microsoft SharePoint EMC SourceOne Business Component Extensions (BCE) for File Systems 26 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

27 Solution Configuration Details Messaging environment The messaging systems supported by EMC SourceOne Management consist of one or more Microsoft Exchange or IBM Lotus Domino servers. Disaster recovery configuration This document assumes that the site has employed a solution recommended by the mail system vendor for replicating the mail system and data. If a failover to the disaster recovery site is necessary, it is assumed that the mail system can be brought back online and be made accessible to the EMC SourceOne system at the disaster recovery site so that processing can resume. The benefit of replicating the messaging environment is the ability to run additional archiving tasks at the disaster recovery site after a failover. This will improve the recovery point objective (RPO) for the EMC SourceOne environment following a failover. Note the following: Exchange Install EMC SourceOne Extensions for OWA software on Exchange servers at the disaster recovery site. Maintain this system in concert with the primary Exchange servers (e.g., performing software upgrades and patches of EMC SourceOne software). Domino Install EMC SourceOne Domino Extensions software on Domino servers at the disaster recovery site. Maintain this system in concert with the primary Domino servers. Ensure that the EMC SourceOne system is connected to journaling databases in both the primary and disaster recovery sites. If a journaling database in a Domino cluster switches to the disaster recovery site during normal operation, this will ensure that the messages are still journaled. Component disaster recovery configuration 27

28 Solution Configuration Details SMTP mail considerations EMC SourceOne archives SMTP mail from a drop directory. If you configure EMC SourceOne to archive messages directly from this location, and the computer hosting this location is one of the components involved in a disaster, those messages are subject to being permanently lost. To support a disaster recovery configuration, you should configure a mechanism that periodically copies the SMTP messages from one location to the actual location against which the archiving task is directed. Although this may result in some duplicate message processing, it provides a backup of the SMTP message data that can be recovered in a disaster. Deleted retention/soft delete recommendations Configure deleted item retention times in Microsoft Exchange, or soft delete retention times in Lotus Domino, for a period that at least doubles the estimated RPO for failover and recovery for journal mailboxes or databases. For example, if the expected RPO is 1 day, configure deleted retention (Exchange) or soft deletes (Domino) to occur for 2 days. This configuration ensures that the messages on the mail server can be "undeleted and replayed" to more accurately represent the mail server's state at the time of failover. This also furthers the goal of achieving a RPO with as close to zero data loss as possible. Shortcutting policy recommendations Your shortcutting policy configuration should also be configured to support disaster recovery. Configure shortcutting to occur at an interval to allow a short period of time for archived messages to be replicated before a shortcut is created. This period of time should represent a number of days that is slightly greater than the amount of time you would expect for the RPO. For example, if you configure the system to archive messages older than 30 days and shortcut messages older than 32 days, this would allow at least 2 days for archived messages to be replicated before a shortcut is created. Note: By default, the volume idle time before a volume closes to an archive container is 2 days. Generally, using a difference of 2 days or slightly more between archive and shortcut policies can help ensure that shortcuts are always pointing to closed volumes. 28 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

29 Solution Configuration Details Consistency groups You can replicate the messaging system components along with the other EMC SourceOne components using a consistency group (or similar functionality supported by your replication solution). Replicating all EMC SourceOne components as a group can normalize the recovery time objective (RTO), minimize data lag, and simplify replication management. Replication timing If the use of a consistency group is not possible (e.g., in cases where the messaging system replication is occurring on an already established or different replication technology), configure the replication timing to be the same or as close as possible as the replication timing configured for other EMC SourceOne components. This furthers the goal of achieving a RPO with as close to zero data loss as possible. Microsoft SharePoint environment The systems supported by EMC SourceOne for Microsoft SharePoint consist of one or more Microsoft SharePoint servers. Disaster recovery configuration This document assumes that the site has employed a solution recommended by Microsoft for replicating the SharePoint system, databases, and data. If a failover to the disaster recovery site is necessary, it is assumed that SharePoint environment can be brought back online and be made accessible to the EMC SourceOne system at the disaster recovery site so that processing can resume. Component disaster recovery configuration 29

30 Solution Configuration Details Best practice recommendations To minimize the chance of data loss, administrators configuring SharePoint archiving activities should observe the best practices described in this section. When archiving content from a SharePoint system with the intent of removing the content from SharePoint, you should always configure two activities: Configure the first activity to copy the content from SharePoint to the archive using the Copy items to archive Archive Action. After archiving, the content will exist in both the archive and in SharePoint. Configure a second activity to move the content from SharePoint to the archive using the Copy items to archive and Delete them from SharePoint Archive Action. After archiving, the content will exist only in the archive and will be deleted from SharePoint. Configure this second activity to occur after a period of time equal to the expected Recovery Point Objective (RPO). Following this practice ensures that content in SharePoint is not deleted until the archived content has been successfully stored and replicated. If a disaster scenario occurs before this process is completed, the SharePoint activities can be replayed against the SharePoint systems. External BLOB Storage integrity utility If your organization is using the optional External BLOB Storage (EBS) feature, an EBS integrity utility (installed with the EBS provider software) is used to compare data in the EBS database on the SharePoint SQL Server with the BLOBs stored in the Native Archive. SharePoint SQL Server The use of this utility assumes that the EBS database is protected from a disaster along with other databases on the SharePoint SQL Server, and is available for query after a disaster occurs. 30 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

31 Solution Configuration Details Native Archive storage and EBS content EBS simply utilizes the Native Archive to access storage EBS content is not archived. Therefore, EBS content is not subject to indexing, retention or available for search using EMC SourceOne Search. However, it is available for search using SharePoint-side search interfaces. The utility requires that the Native Archive and storage is available for comparison with the EBS database after a disaster occurs. Details about using this utility are provided in EMC SourceOne EBS Integrity Tool Operations on page 95. File system environment The systems supported by EMC SourceOne for File Systems consist of one or more supported file systems. Disaster recovery configuration There is no specific disaster recovery configuration required for the file systems from which EMC SourceOne for File Systems archives content. Best practice recommendations To minimize the chance of data loss, administrators configuring file system activities should observe the best practices described in this section. When archiving file content from a file system with the intent of removing the files from the file system (or shortcutting which replaces each file with a link to the archived file), you should always configure two activities. The first activity should be configured to copy the files from the file system to the archive. You can then configure a second activity to either: Move the files from the file system to the archive (the files are removed from the file system) Move the files from the file system to the archive and replace them with a link to each archived file Component disaster recovery configuration 31

32 Solution Configuration Details In either case, the second activity must be configured to occur after a period of time equal to the expected Recovery Point Objective (RPO). Following this practice ensures that content on file systems is not deleted until the archived content has been successfully stored and replicated. If a disaster scenario occurs before this process is completed, the file archiving activities can be replayed against the file systems. Master computer The EMC SourceOne master schedules and distributes jobs that are processed by workers. Disaster recovery configuration The master host computer is stateless and does not require replication. There only can be one active master computer in a EMC SourceOne configuration. You can cluster the master computer at the disaster recovery site for high availability during a failover using the Microsoft Cluster Service. Install and configure an identical standby master computer at the disaster recovery site. Use the following guidelines when installing the EMC SourceOne Master Services software on the disaster recovery master computer: Before installing the Master Services software on the disaster recovery computer, temporarily stop the EMC SourceOne services (ExJobScheduler and ExAddressCacheService) on the primary master computer. Install the software on the disaster recovery computer and restart the system. If clustering the master computer at the disaster recovery site, follow the procedure in the EMC SourceOne Installation Guide to configure the cluster. Disable the EMC SourceOne services on the disaster recovery computer. The services must remain disabled during normal operation. At no time should more than one master server be running. 32 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

33 Solution Configuration Details Restart the EMC SourceOne services on the primary master computer. This process will register the second master computer in the EMC SourceOne Activity database so that it is recognized in a failover scenario, but remains inactive until it is needed during a failover. Maintain this system in concert with the primary master computer (e.g., performing software upgrades and hot fixes). Worker computer EMC SourceOne worker computers perform message archiving activities from various messaging data sources. Each worker computer can be configured to process only specific activities. Disaster recovery configuration Worker host computers are stateless and do not require replication. Note: There is a work directory for the worker in which temporary data is stored. However, this temporary data is not needed in a failover scenario. Install and configure at least one worker computer at the disaster recovery site. If you use the Microsoft SharePoint or File Systems products, you must also install the corresponding Business Component Extensions (BCE) software to enable the workers to process these types of activities: EMC SourceOne Business Component Extensions (BCE) for Microsoft SharePoint EMC SourceOne Business Component Extensions (BCE) for File Systems Component disaster recovery configuration 33

34 Solution Configuration Details Ideally the number of workers at the disaster recovery site should match the number of workers at the primary site to maintain a consistent level of service during a failover. If there are less workers at the disaster recovery site, consider the following: Until the failback occurs and both sites return to regular operation, the level of service may be limited to the number of workers at the secondary site. If dealing with an extended failover, you can consider adding workers at the secondary site to maintain a high level of service. During a failover, you can adjust the roles of the workers to assume the primary site's processing tasks. In a hot/warm configuration, workers can be operational at the disaster recovery site and configured to work over the WAN with devices at the primary site. Review the information described in Considerations on page 24 which describes the types of activities that can be processed across a WAN connection. In a hot/cold configuration, workers at the disaster recovery site are either: Powered down Running with EMC SourceOne services set to disabled Note: Do not pause or stop the worker using the EMC SourceOne administration console as a substitute for powering down or disabling services. If the worker is restarted, the worker does not remain in this state (and is listed as available in the console). Ensure that EMC SourceOne applications such as Web Services and Mobile Services are installed and configured on workers at the disaster recovery site. Maintain this system in concert with the primary worker computers (e.g., performing software upgrades and hot fixes). 34 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

35 Solution Configuration Details Web server The EMC SourceOne Search application provides a Web client interface to end users and passes query and results data to and from the EMC SourceOne Web Services software installed on one or more workers. Typical configuration scenarios include: Scenario 1: You can install the EMC SourceOne Search application on a single EMC SourceOne Web Services computer. Scenario 2: You can install the EMC SourceOne Search application on two or more EMC SourceOne Web Services computers and configure a network load balancing using an appliance or software solution such as Microsoft Network Load Balancing Service (NLBS). Scenario 3: You can install the EMC SourceOne Search application in an IIS Web server farm accessible to end users. EMC SourceOne Search then communicates through the load balancer address to the Web Services application on the Worker Services computer. You can optionally use a Secure Sockets Layer (SSL) connection between the EMC SourceOne Search application and the Web Services application. The disaster recovery configuration information in this section applies only to Scenario 3. If you are using Scenario 1 or Scenario 2, the Search application is installed on a worker computer and the disaster recovery configuration described in Worker computer on page 33 applies. Disaster recovery configuration Web servers are stateless and do not require replication. Install and configure at least one Web server (including the EMC SourceOne Search application) at the disaster recovery site. As with worker computers, the number of Web servers at the disaster recovery site is based on SLA requirements. Maintain this system in concert with the primary Web servers (e.g., performing software upgrades and hot fixes). Component disaster recovery configuration 35

36 Solution Configuration Details SQL Server and databases A database server running Microsoft SQL Server supports the following EMC SourceOne databases: Activity database - This database maintains data associated with EMC SourceOne system processing. Search database - This database maintains data associated with the EMC SourceOne Search application. Native Archive database - This database manages data associated with the EMC SourceOne Native Archive, a role-based architecture that manages the archiving, indexing, searching, and retrieving of content. Disaster recovery configuration Detailing a specific disaster recovery solution for Microsoft SQL Server is beyond the scope of this document. However, ensure that you review the following information: EMC SourceOne requirements General considerations SQL Server host configuration EMC SourceOne requirements To recover from a disaster scenario, EMC SourceOne requires the following: An available copy of the EMC SourceOne databases, SQL system and temporary databases, and logs at the disaster recovery site. A SQL Server host at the disaster recovery site on which the databases can be mounted. An alias configuration in DNS to support re-directing EMC SourceOne servers to the database server at the disaster recovery site during failover. See SQL Server host configuration on page 38 for more details. 36 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

37 Solution Configuration Details General considerations This solution requires the use of existing Microsoft best practices which provide frequent replication of: All EMC SourceOne databases (Activity, Native Archive, Search) SQL system databases. See the following MSDN information for a list of SQL system databases: L.90%29.aspx EMC provides additional guidance on Microsoft SQL Server replication using EMC storage solutions. A representative example includes the solutions for EMC CLARiiON storage arrays described in the following whitepaper: EMC CLARiiON Data Replication Options for Microsoft SQL Server Deployments. You can find this whitepaper on EMC Online Support ( Alternatively, you can replicate SQL Server and the databases along with the other EMC SourceOne components using a consistency group (or similar functionality as supported by your replication solution). Replicating all EMC SourceOne components as a group can normalize the RTO time, minimize data lag, and simplify replication management. If the use of a consistency group is not possible (e.g., in cases where the messaging system replication is occurring on an already established or different replication technology), configure the replication timing to be the same or as close as possible as the replication timing configured for other EMC SourceOne components. Component disaster recovery configuration 37

38 Solution Configuration Details SQL Server host configuration Use a CNAME alias in DNS which, during regular operations, points to the SQL Server host computer (or virtual host if clustering) at the primary site. When using a CNAME alias, follow standard best practices provided by Microsoft. In particular, note the following: Register the Kerberos service principal names (SPNs), the host name, and the fully-qualified domain name (FQDN) for all the new DNS alias (CNAME) records. If you do not do this, a Kerberos ticket request for a DNS alias (CNAME) record may fail and return the following error code: KDC_ERR_S_SPRINCIPAL_UNKNOWN You must be a domain administrator to run this command. For more information, refer to the following Microsoft KB article: Connection issues can occur when using a CNAME to address a server by another name other than the one it recognizes as itself. The server is not "listening" on the alias, and therefore is not accepting connections to that name. To resolve this issue, add a registry value (DisableStrictNameChecking) as described in the following Microsoft KB article: Maintain the SQL Server host in concert with the primary SQL Server host (e.g., performing software upgrades and hot fixes). 38 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

39 Solution Configuration Details Native Archive host computer The EMC SourceOne Native Archive software uses an architecture which can be configured to perform the following roles: Archive Index Search Retrieve The Native Archive software can be installed on a single host computer or multiple host computers, enabling you to dedicate hardware to specific roles to match your organization's policies and requirements. This section describes the high availability and disaster recovery strategy for the Native Archive host computer(s). The storage locations associated with the Native Archive are hosted on hardware recommended using an approved reference configuration and are covered in Native Archive - Message Center storage on page 40. Disaster recovery configuration Native Archive computers are stateless and do not require replication. Install and configure at least one Native Archive computer at the disaster recovery site. Ideally the number of Native Archive computers at the disaster recovery site should match the number of computers at the primary site to maintain a consistent level of service during a failover. If there are less computers at the disaster recovery site, consider the following: Until the failback occurs and both sites return to regular operation, the level of service may be limited to the number of Native Archive computers at the secondary site. If dealing with an extended failover, you can consider adding Native Archive computers at the secondary site to maintain a high level of service. During a failover, you can adjust the roles of the Native Archive computers to assume the primary site's processing tasks. Component disaster recovery configuration 39

40 Solution Configuration Details In a hot/warm configuration, this computer is operational at the disaster recovery site and configured to work over the WAN with devices at the primary site. Review the information described in Considerations on page 24 which describes the types of activities that can be processed across a WAN connection. In a hot/cold configuration, Native Archive computers at the disaster recovery site are either: Powered down Running with EMC SourceOne services set to disabled Maintain the Native Archive server at the disaster recovery site in concert with the primary Native Archive server(s) (e.g., performing software upgrades and hot fixes). Native Archive - Message Center storage When you define one or more archive folders within a Native Archive to contain content, you configure a Message Center storage location in which content is stored and built into volumes before it is written to a container file. Disaster recovery configuration This storage location is defined in the EMC SourceOne console as a UNC path (\\computer_name\shared_folder). Data can be stored on a storage array connected using a Storage Area Network (SAN). Alternatively, the data can be stored directly on the NAS device, including the specific devices described in Additional storage options on page 19. Replicate this storage location. If the storage solution supports the concept of consistency groups, replicate this data consistently with other EMC SourceOne data. File server host or NAS device configuration on page 47 describes the specific disaster recovery configuration for this host or device. 40 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

41 Solution Configuration Details Native Archive - Index storage When you define one or more archive folders within a Native Archive to contain content, you also configure an index storage location in which full-text indexes are stored. Disaster recovery configuration This storage location is defined in the EMC SourceOne console as a UNC path (\\computer_name\shared_folder). Data can be stored on a storage array connected using a Storage Area Network (SAN). Alternatively, the data can be stored directly on the NAS device, including the specific devices described in Additional storage options on page 19. Replicate this storage location. If the storage solution supports the concept of consistency groups, replicate this data consistently with other EMC SourceOne data. File server host or NAS device configuration on page 47 describes the specific disaster recovery configuration for this host or device. Native Archive - Container storage Depending on your requirements and the version of EMC SourceOne you are using, you have the following options for storing containers: Network-accessible storage Direct storage on EMC Centera Storage on EMC Centera using EMC DiskXtender Network-accessible storage You can store containers directly on network-accessible storage, such as shareable drives or network-attached storage devices (NAS devices). Disaster recovery configuration This storage location is defined in the EMC SourceOne console as a UNC path (\\computer_name\shared_folder). Data can be stored on a storage array connected using a Storage Area Network (SAN). Alternatively, the data can be stored directly on the NAS device, including the specific devices described in Additional storage options on page 19. Component disaster recovery configuration 41

42 Solution Configuration Details Replicate this storage location. If the storage solution supports the concept of consistency groups, replicate this data consistently with other EMC SourceOne data. File server host or NAS device configuration on page 47 describes the specific disaster recovery configuration for this host or device. Direct storage on EMC Centera Your EMC SourceOne configuration may require the long-term storage of EMC SourceOne container files on an EMC Centera content-addressed storage (CAS) system. Existing archive folders If you have existing archive folders configured to store containers on EMC Centera using DiskXtender, you must continue to use this solution for those folders. New archive folders When using EMC SourceOne Management version 6.5 SP2 and above, you can store containers on EMC Centera devices directly accessed by EMC SourceOne Management (without the use of DiskXtender). Pool address configuration You configure EMC Centera pool addresses using the EMC SourceOne console. To support the redirection to alternative EMC Centera access nodes in a disaster recovery scenario, configure your EMC Centera pool addresses as symbolic access node addresses (node1,node2,node3, etc.) in the connection string instead of IP addresses. You can then perform the redirection by changing DNS entries. For information on configuring this storage option, refer to the EMC SourceOne Administration Guide. Note: Although the release note information refers only to IP addresses, node names are also valid entries in the Pool Address field. Disaster recovery configuration Refer to EMC Centera on page 48 for configuration details. Note: If using EMC DiskXtender and EMC Centera, see Storage on EMC Centera using EMC DiskXtender on page EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

43 Solution Configuration Details Storage on EMC Centera using EMC DiskXtender EMC SourceOne Management versions prior to version 6.5 SP2 required the use of EMC DiskXtender as a means to connect to EMC Centera. EMC DiskXtender for Windows (along with the EMC Centera media service) facilitates the communication between DiskXtender and Centera. Existing archive folders If you have existing archive folders configured to store containers on EMC Centera using DiskXtender, you must continue to use this solution for those folders. New archive folders After upgrading to EMC SourceOne Management to version 6.5 SP2, you can configure any new archive folders that you create to utilize direct storage on EMC Centera. See Direct storage on EMC Centera on page 42 for details. DiskXtender and other storage media The scenario of using EMC DiskXtender connected to other long-term storage such as optical devices or tape, while valid, is not considered for the purposes of this document. DiskXtender is only described as a means of connecting EMC SourceOne to Centera. Base configuration You install DiskXtender on a Microsoft Windows file server and provide a Windows NT File System (NTFS) volume (called an extended drive) to which EMC SourceOne container files are written. This location is then specified when configuring an archive folder using the SourceOne console. The container files are systematically moved to EMC Centera using a policy-based configuration. The containers are purged from the extended drive and replaced with pointers to the content stored in EMC Centera. Disaster recovery configuration This solution requires the configuration for disaster recovery described in the EMC DiskXtender Microsoft Windows Version Installation Guide for the supported version of DiskXtender. Current information on the version of EMC DiskXtender supported with EMC SourceOne is provided in the EMC SourceOne Products Compatibility Guide. Component disaster recovery configuration 43

44 Solution Configuration Details General considerations Replicate the extended drive storage location. If the storage solution supports the concept of consistency groups, replicate this data consistently with other EMC SourceOne data. DiskXtender does not support a hot/hot configuration. Disaster recovery is provided by replicating the extended drive data to the disaster recovery site so that it is available to a corresponding passive DiskXtender host and extended drive at the disaster recovery site. Host computer configuration This solution requires that you set up DiskXtender replication using the configuration information provided in the EMC DiskXtender Microsoft Windows Version Installation Guide. Key points include: DiskXtender is installed and actively managing files on the server at the primary site. DiskXtender is also installed on the server at the disaster recovery site. However, the DiskXtender service is stopped (set to manual or disabled). In other words, DiskXtender is passive at the disaster recovery site. Ensure that the DiskXtender servers at the primary and disaster recovery site have the same host name and extended drive letter designation (for example, H:). Also ensure that the extended drive at the disaster recovery site has the same Microsoft volume serial number as the extended drive on the primary site. This can be set using the volumeid.exe utility from Microsoft. Maintain the DiskXtender host computer at the disaster recovery site in concert with the DiskXtender host at the primary site (e.g., performing software upgrades and hot fixes). Refer to the DiskXtender documentation set for information on maintaining systems in this configuration. DiskXtender can be installed in a high-availability configuration at each site using Microsoft Windows cluster configuration. For details on implementing this configuration at the primary and disaster recovery site, refer to the EMC DiskXtender Microsoft Windows Version Installation Guide. Refer to the EMC DiskXtender Microsoft Windows Version Installation Guide for more information host computer configuration in a replicated environment. 44 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

45 Solution Configuration Details Exporting and replicating registry keys In addition to replicating the DiskXtender extended drive itself, you must also configure a script to frequently export the DiskXtender registry keys to one of the replicated storage locations. This configuration enables you, during a failover, to import the replicated registry keys to the DiskXtender server at the disaster recovery site. Export and replicate the following registry keys: HKEY_LOCAL_MACHINE\SOFTWARE\EMC\DiskXtender HKEY_LOCAL_MACHINE\SOFTWARE\EMC\XMS The script should look similar to the following: regedit /E \\STORAGE\SOURCEONE_Recovery\DX.reg HKEY_LOCAL_MACHINE\SOFTWARE\EMC\DiskXtender regedit /E G:\Xtender_Recovery\XMS.reg HKEY_LOCAL_MACHINE\SOFTWARE\EMC\XMS Note the following: If you are configuring a standalone DiskXtender server, the script can be run using a Windows scheduled task, which you should configure to run frequently (for example, every two minutes). If you are configuring a clustered DiskXtender server, configure the script to be run as a Generic Application. Ensure the script is written to loop frequently (for example, every two minutes). Replicate the registry keys to a location other than the DiskXtender extended drive. For example, in a clustered DiskXtender configuration, use another replicated disk in the cluster group. DiskXtender media service configuration options for Centera Best practices for configuring the media service options for EMC DiskXtender for Windows and EMC Centera are beyond the scope of this document. For specific guidance on this configuration, including considerations for an environment which includes EMC Centera replication, refer to the following white paper: Connecting EMC DiskXtender for Windows to EMC Centera: Best Practices Planning This white paper can be found on EMC Online Support ( in the EMC DiskXtender for Windows white paper location. Component disaster recovery configuration 45

46 Solution Configuration Details Additional resources: For detailed information and best practices for configuring a media service for EMC Centera, refer to information on media management for configuring EMC Centera in the EMC DiskXtender Microsoft Windows Version Best Practices Guide. In addition, consult the EMC DiskXtender Microsoft Windows Version Installation Guide and EMC DiskXtender Microsoft Windows Version Administration Guide for detailed configuration steps and other information. Purge configuration options Optionally, you can delay the purging of volumes until they have been replicated by Centera. This can prevent a situation in which a stub exists on the extended drive, but the associated clip does not exist on the disaster recovery Centera. Because of current software limitations, you must logically determine when volumes have likely been replicated by Centera and then configure a corresponding purge rule in DiskXtender. For example, if Centera replication generally takes less than a few hours, then you should configure the purge delay of volumes from the extended drive to the lowest granularity (1 day). To support a disaster recovery configuration in which you want to support Direct Read for all files, yet allow data to be replicated by Centera before purging files from the extended drive, consider the following when configuring move rule settings: In the EMC SourceOne console (which allows you to configure some DiskXtender options in the Storage Options page of the New Archive Folder wizard), uncheck the Purge after move and Direct read options. Use the Force Direct Read for all files option on the Options tab of the Extended Drive Properties dialog box. This will automatically perform Direct Read for any purged file, but will not change the attribute. Configure a default purge rule to purge based on high and low drive size watermarks. Note: Refer to the Media Management chapter of the EMC DiskXtender Microsoft Windows Version Best Practices Guide for best practices and other considerations for using EMC Centera. 46 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

47 Solution Configuration Details File server host or NAS device configuration File servers or NAS devices associated with Native Archive storage (as described in Native Archive - Message Center storage on page 40) require a disaster recovery configuration. Disaster recovery configuration Use a CNAME alias in DNS which, during regular operations, points to the host computer (or virtual host if clustering) or NAS device at the primary site. When using a CNAME alias, follow standard best practices provided by Microsoft. In particular, note the following: You must register the Kerberos service principal names (SPNs), the host name, and the fully-qualified domain name (FQDN) for all the new DNS alias (CNAME) records. If you do not do this, a Kerberos ticket request for a DNS alias (CNAME) record may fail and return the following error code: KDC_ERR_S_SPRINCIPAL_UNKNOWN You must be a domain administrator to run this command. For more information, refer to the following Microsoft KB article: Connection issues can occur when using a CNAME to address a server or device by another name other than the one it recognizes as itself. The server or device is not "listening" on the alias, and therefore is not accepting connections to that name. To resolve this issue, add a registry value (DisableStrictNameChecking) as described in the following Microsoft KB article: Ensure the host computer is not running at the disaster recovery site. Maintain the host computer in concert with the primary host (e.g., performing software upgrades and hot fixes). Component disaster recovery configuration 47

48 Solution Configuration Details EMC Centera EMC Centera provides for long-term storage for EMC SourceOne container files. Disaster recovery configuration For information on configuring Centera replication, refer to the EMC Centera documentation set. If you are using EMC SourceOne direct storage with EMC Centera, see Direct storage on EMC Centera on page 42. If you are using DiskXtender with EMC Centera within the context of the EMC SourceOne disaster recovery solution, see the following topics: DiskXtender media service configuration options for Centera on page 45 Purge configuration options on page EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

49 3 Failover to the Disaster Recovery site This section provides a detailed procedure for performing a failover from the primary site to the disaster recovery site. The information in this chapter assumes that the configuration detailed in Solution Configuration Details on page 21 has been implemented. This chapter includes the following topics: Monitoring the system Failover procedures Failover to the Disaster Recovery site 49

50 Failover to the Disaster Recovery site Monitoring the system This section provides information on monitoring system status using the following EMC array-based replication solutions: Monitoring Symmetrix SRDF status - Normal status on page 50 Monitoring MirrorView status on page 52 Monitoring RecoverPoint status on page 52 This information is intended only as an example for environments that may be using an EMC array-based replication technology. For complete details about monitoring one of these systems, refer to the EMC documentation that accompanies the product. If you are using another solution (including another vendor s array-based solution, or a non-array-based replication), refer to the product documentation accompanying that solution. Monitoring Symmetrix SRDF status - Normal status If the EMC SourceOne system is running at the primary site and protected by SRDF, the SRDF status will show information similar to the following: Table 1 Normal status monitoring From primary site From secondary site R1 RW a RW R2 WD b WD Link RW RW SRDF status a.read-write b.write-disabled Synchronized (if using SRDF/S) or Consistent (if using SRDF/A) Synchronized (if using SRDF/S) or Consistent (if using SRDF/A) 50 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

51 Failover to the Disaster Recovery site Monitoring Symmetrix SRDF status - Failure status In a failure state, the EMC SourceOne secondary system is stopped, since the R2 devices are not available for reads or writes to the Symmetrix host. When the primary site becomes unavailable (including the Symmetrix), you can observe the outage at the disaster recovery site using the SRDF status. If a loss of communication between the primary and secondary Symmetrix hosts occurs, the SRDF status will show information similar to the following: From primary site From secondary site R1 RW NA R2 NA a WD Link NR NR b SRDF status Partitioned Partitioned a.not accessible b.not ready This state may not indicate a failure which requires failover to the disaster recovery site. In some cases, such as a link failure, the EMC SourceOne system may still be operative at the primary site (since SRDF does not automatically write disable the R1 devices). Resuming the SRDF links in this scenario should be a manual process - once the links are available again, it is recommended that you create a Business Continuance Volume (BCV) from the Symmetrix at the disaster recovery site to create a "gold copy" before resuming SRDF replication. If a failure status occurs and is coupled with the loss of the EMC SourceOne computers or other components at the primary site, the determination should then be made by the IT staff to perform a failover to the disaster recovery site as described in this chapter. Monitoring the system 51

52 Failover to the Disaster Recovery site Monitoring MirrorView status The Navisphere Management Suite provides an interface to monitor replication status. Specifically, the Navisphere Manager application's Remote Mirror Property dialog box provides a status for the mirror. If a primary failure occurs, Navisphere reports the failure. If the primary failure is catastrophic, the original client that is managing the storage system may be unusable and thus unable to report the failure. For such a failure, the administrator at the secondary site must use a client to manage the storage system. Monitoring RecoverPoint status The RecoverPoint Management Console provides the administrator with a graphical user interface with which to manage the RecoverPoint system. The System Status pane provides an overview of overall system health at a glance. 52 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

53 Failover to the Disaster Recovery site Failover procedures This procedure describes the manual steps required when an outage occurs and communication is lost at the primary site, necessitating a failover to the environment set up at the disaster recovery site. This section includes the following procedures: Step 1: Stop EMC SourceOne processing at disaster recovery site on page 54 Step 2: Check that replicated data is accessible on page 54 Step 3: Verify disk health of file servers on page 56 Step 4: Perform Active Directory changes for SQL and file servers on page 56 Step 5: Ensure dependent systems are online and accessible on page 57 Step 6: Recover the DiskXtender configuration on page 57 Step 7: Power on EMC SourceOne computers (hot/cold configuration only) on page 62 Step 8: Ensure the EMC SourceOne computers can access key resources on page 62 Step 9: Run the EMC SourceOne Archive Integrity Tool on page 63 Step 10: Start EMC SourceOne services on page 63 Step 11: Perform reprocessing on page 64 Step 12: Rerun the EMC SourceOne Archive Integrity Tool on page 66 Step 13: Verify EMC SourceOne operations on page 67 Step 14: (Optional) Run the SharePoint EBS integrity tool on page 67 Failover procedures 53

54 Failover to the Disaster Recovery site Step 1: Stop EMC SourceOne processing at disaster recovery site In order to perform the failover procedures, the EMC SourceOne system must remain in a quiesced state. Ensure that all computers in the EMC SourceOne configuration (master, workers, Native Archive servers) are stopped and no data is moving through the system. In a hot/warm configuration in which workers and Native Archive computers are already operational at the disaster recovery site, stop all of the EMC SourceOne services on each of these computers. In a hot/cold configuration - Ensure that the EMC SourceOne computers remain powered down or the EMC SourceOne services are disabled at the disaster recovery site. Step 2: Check that replicated data is accessible Ensure that the latest good copy of replicated data is available at the disaster recovery site. Data includes: Message Center data Indexes Container files Note: This can also include SQL and mail system databases, logs and data if replicated using the same solution as EMC SourceOne data. However, the expectation is that you will make these systems and data available through existing practices used at your site. SAN storage If using array-based storage on a SAN (and depending on the replication technology used) enable the target storage array. The following examples provide the basic context for EMC Symmetrix SRDF, MirrorView or Recoverpoint solutions. For other storage solutions, refer to the appropriate documentation. 54 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

55 Failover to the Disaster Recovery site Examples include: SRDF - From a control host at the disaster recovery site, read/write enable the secondary (R2) devices. MirrorView - Promote the secondary image. To promote a secondary image, the storage system hosting the secondary image must be currently managed by the application and the state of the secondary image to be promoted must be either Consistent or Synchronized. If the source image and target image can communicate with each other and the secondary image is promoted, the former primary image is demoted to a secondary image. RecoverPoint - Using the RecoverPoint CLI or GUI, enable secondary site host access by choosing the appropriate image. You can choose to recover to a bookmarked image created either manually, through the Group Set feature or through the apply_parallel_bookmark feature. The secondary site image access mode in RecoverPoint is suited to verify that the system is consistent and operational. If you want to run the host applications in the disaster recovery environment, pause the data transfer between the sites and set the mode to Both Sides Active. With this option, RecoverPoint begins marking mode and marks the data changes occurring in both sites on to the local repository volume. Once the test is complete (or if the primary site is operational after a disaster) you can choose the direction of the data transfer and select the production environment. Refer to the EMC RecoverPoint Administrator's Guide for detailed procedures. Other storage If using a copy of the data stored on NAS or other media, ensure that data is accessible. Refer to the appropriate documentation for your storage solution. Failover procedures 55

56 Failover to the Disaster Recovery site Step 3: Verify disk health of file servers Check the EMC SourceOne file server systems hosting the Message Center, index and container shares for errors by running chkdsk. Proactively running this command allows you to review and repair file system errors. You should first run chkdsk in read-only mode (default). This will display a status report of the findings. If you have corruptions, backup that disk before you correct it. To fix the errors, run chkdsk -f. For more information about chkdsk, including usage, command syntax and parameters, refer to the Microsoft Windows Server TechCenter and search for chkdsk or access the following URL: 20utility&ac=1 Step 4: Perform Active Directory changes for SQL and file servers Redirect the CNAME aliases to SQL and file server host computers (or virtual host) names at the disaster recovery site. Do the following: 1. From Active Directory, repoint the CNAME to SQL and file servers to the servers or virtual hosts (if clustered) at the disaster recovery site. 2. Flush the DNS Resolver Cache: ipconfig /flushdns 3. Ping each server/virtual host by host name and confirm that the ping resolves to the correct IP address for each computer at the disaster recovery site. 56 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

57 Failover to the Disaster Recovery site Step 5: Ensure dependent systems are online and accessible Ensure that disaster recovery has been performed so that the following systems on which EMC SourceOne depends are online and accessible: Data sources (Mail servers, Microsoft SharePoint servers, and file systems) SQL Server, databases (SQL system and EMC SourceOne) and logs File servers hosting EMC SourceOne storage shares Follow your organization s procedures for restoring these systems. Step 6: Recover the DiskXtender configuration If you are not using DiskXtender in your environment, continue to Step 7: Power on EMC SourceOne computers (hot/cold configuration only) on page 62. If you are using DiskXtender in your environment, follow the sub-procedures in this step (in the order presented) to recover the DiskXtender configuration at the disaster recovery site: Step A: Verify DiskXtender connectivity to Centera on page 58 Step B: Import the DiskXtender registry keys to the DiskXtender host computer on page 58 Step C: Configure export of DiskXtender registry keys back to primary site on page 59 Step D: Clustered DiskXtender only. Add the Reset Disk Resources registry key on page 59 Step E: Restart the DiskXtender host computer or cluster nodes on page 60 Step F: Verify data consistency between DiskXtender and Centera on page 60 Step G: Start the EMC DiskXtender service on page 61 Step H: Inpect DiskXtender on page 61 Failover procedures 57

58 Failover to the Disaster Recovery site Step A: Verify DiskXtender connectivity to Centera Verify EMC DiskXtender connectivity to EMC Centera using best practices described in the following documents: Connecting EMC DiskXtender for Windows to EMC Centera: Best Practices Planning This white paper can be found on EMC Online Support ( EMC DiskXtender Microsoft Windows Version Best Practices Guide. EMC DiskXtender Microsoft Windows Version Installation Guide EMC DiskXtender Microsoft Windows Version Administration Guide Step B: Import the DiskXtender registry keys to the DiskXtender host computer As part of the disaster recovery solution configuration, the DiskXtender registry key is continuously exported to a designated storage location at the disaster recovery site (as described in Exporting and replicating registry keys on page 45). This enables you to update the DiskXtender host computer at the disaster recovery site with the last known configuration from the primary site. Import the DiskXtender registry key to the DiskXtender host computer at the disaster recovery site by: Deleting the existing DiskXtender key Merging the replicated key to the DiskXtender server at the disaster recovery site Follow this procedure to replace the registry keys at the disaster recovery site with the replicated keys from the primary site: 1. If using clustererd DiskXtender, bring the cluster group online using Cluster Administrator. 2. Go the replicated location that contains the replicated DiskXtender registry keys. 3. Delete keys to remove old data during the merge. For example: regedit /S G:\del.reg REM Example of the del.reg file REM [-HKEY_LOCAL_MACHINE\SOFTWARE\EMC\DiskXtender] REM [-HKEY_LOCAL_MACHINE\SOFTWARE\EMC\XMS] 58 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

59 Failover to the Disaster Recovery site 4. Merge the key that was replicated from the primary site. For example: regedit /S G:\SOURCEONE_Recovery\DX.reg regedit /S G:\SOURCEONE_Recovery\XMS.reg 5. If using clustererd DiskXtender, take the cluster group offline using Cluster Administrator. Step C: Configure export of DiskXtender registry keys back to primary site Using the configuration described Exporting and replicating registry keys on page 45, ensure the DiskXtender registry keys are configured for export to one of the replicated drives and ensure that the keys are replicated back to the primary site for later use during the failback process. Step D: Clustered DiskXtender only. Add the Reset Disk Resources registry key This step may not be necessary in all cases, but if DiskXtender disk resources do not appear in the Extended Drive Resource Groups in Cluster Administrator, a registry key can be configured to recreate it. Note: This procedure is described in the EMC Knowledgebase article esg To recreate the DiskXtender Disk resource, follow these steps: 1. Stop the EMC DiskXtender Service resource in Cluster Administrator. 2. Create a new STRING value in the following location: KEY NAME: HKEY_LOCAL_MACHINE\Software\Emc\DiskXtender\Setup VALUE NAME: ResetDiskResource VALUE: YES 3. Start the EMC DiskXtender Service resource in Cluster Administrator. Note the following: The DiskXtender Disk resource will only be recreated for extended drives that have already been configured in the DiskXtender Administrator. After performing this procedure, the DiskXtender Disk resource will appear in the Cluster Administrator. The registry key will delete itself. Failover procedures 59

60 Failover to the Disaster Recovery site Step E: Restart the DiskXtender host computer or cluster nodes Restart the DiskXtender host computer or cluster nodes. Step F: Verify data consistency between DiskXtender and Centera Assuming you followed the recommendation described in Purge configuration options on page 46, you must verify that the data in the DiskXtender extended drive is consistent with data on the Centera and, if necessary, rewrite data that was not replicated to the Centera at the disaster recovery site. Obtain the DXCenVerify utility. Download the appropriate version for the release of DiskXtender you are using. Registered EMC customers can download software from EMC Online Support ( Run the DXCenVerify utility to determine which volumes are on the extended drive but did not replicate to the disaster recovery (now local) Centera. The DXCenVerify utility outputs a text file indicating the missing volumes (MisFileRep_DateTime.log). After you run the utility and determine that there is a missing volumes issue, you have two options to ensure data consistency. Option 1 - Manually remove DiskXtender attributes from volumes Once you identify the missing volumes from the output log generated by DXCenVerify, you can remove DiskXtender attributes from files to enable DiskXtender to recognize the files that need to be moved to the Centera. 1. Copy all files indicated as missing to a temporary directory other than an extended drive (for example, c:\dx_temp). Copying the files will remove any DiskXtender attributes. 2. On the extended drive on which the copied files reside, perform a privileged delete. Note: You will need to start the EMC DiskXtender service to perform this step. 3. With the EMC DiskXtender service now running, copy the files from the temporary directory back to the extended drive location. 4. Because the "Fetched" attribute has been removed, DiskXtender will move these files to the Centera based on existing move rules. 60 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

61 Failover to the Disaster Recovery site 5. Run the DXCenVerify utility again to validate the files have been moved to the remote (now local) Centera. 6. Force a DiskXtender background scan. Background scans enable DiskXtender to verify that all files on the extended drive that need to be managed are being managed. Refer to the EMC DiskXtender Microsoft Windows Version Administration Guide for additional information on background scans. Option 2 - Remove DiskXtender attributes from volumes using the DxDmChk utility Alternatively, you can remove the DiskXtender attributes using the DiskXtender Data Manager Check (DxDmChk) utility. This utility can be used to remove the extended drive attributes from a file so that it can be moved to the Centera using the existing move rules. Note the following: Download the appropriate version for the release of DiskXtender you are using. Registered EMC customers can download software from EMC Online Support ( You can run the utility to generate a report to determine if there are issues. However, to actually remove extended drive attributes you must obtain an activation key from EMC DiskXtender Support and receive assistance with using the utility. Step G: Start the EMC DiskXtender service Start the EMC DiskXtender service (if not already started when performing the previous step). Step H: Inpect DiskXtender Use the DiskXtender File System Manager interface to review your DiskXtender configuration and verify that media is present and online. It may also be useful to run a DiskXtender file report to view the status of EMC SourceOne containers on the extended drive: 1. Right-click the EMC SourceOne container folder on the extended drive and select DiskXtender > File Report. 2. Accept the report defaults. Failover procedures 61

62 Failover to the Disaster Recovery site 3. Verify that all containers have a status of fetched or purged. Tips: Search for the word NOT to quickly find volumes that are not fetched or purged. Search for the word offline to find media which isn t online. Step 7: Power on EMC SourceOne computers (hot/cold configuration only) Assuming the correct steps were taken during the disaster recovery configuration, when you start the computers at the disaster recovery site, all EMC SourceOne services will be in a Manual or Disabled state. Note: Do not start the EMC SourceOne services yet. Step 8: Ensure the EMC SourceOne computers can access key resources Ensure that EMC SourceOne computers can access SQL Server, the data sources (mail servers, SharePoint servers, file systems) and shares. Perform another check to ensure that all computers and resources are accessible. For example: On each SourceOne computer (master, workers, Native Archive), flush the DNS Resolver Cache: ipconfig /flushdns Ping each server/virtual host by host name and confirm that the ping resolves to the correct IP address for each machine at the disaster recovery site. Use ODBC tools to check database connectivity. 62 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

63 Failover to the Disaster Recovery site Step 9: Run the EMC SourceOne Archive Integrity Tool Ensure that you run the version of the EMC SourceOne Archive Integrity Tool for the version of EMC SourceOne Management you are running. Log in to a Native Archive server at the disaster recovery site using the EMC SourceOne primary service account. Run the EMC SourceOne Archive Integrity Tool and ensure that it successfully completes its operations. Follow this general procedure: 1. (Optional) Run the tool in simulation mode (default) to assess the situation. 2. Run the tool using the -nosim option. After completing this process, the tool will identify and correct consistency issues and generate a log file which is used later in the failover process. The remaining steps of the failover procedure include starting the EMC SourceOne services at the disaster recovery site, performing some reprocessing activities, and rerunning the tool. For details on usage, command syntax, parameters, and the sequence of operations, see EMC SourceOne Archive Integrity Tool Operations on page 81. Step 10: Start EMC SourceOne services Start the EMC SourceOne services on each computer in the configuration at the disaster recovery site in the following order: 1. Master 2. Native Archive server(s) 3. Worker(s) Failover procedures 63

64 Failover to the Disaster Recovery site! IMPORTANT For environments in which the system was in the midst of a backup (and is therefore in a suspended state which was likely persisted through replication), run the resume scripts provided with the EMC SourceOne backup utilities. For details, refer to the backup information provided in the EMC SourceOne Administration Guide. Step 11: Perform reprocessing Before returning EMC SourceOne to full operation at the disaster recovery site, you must perform the following reprocessing steps: Step A: Start the EMC SourceOne management console on page 64 Step B: Adjust Worker and Native Archive computer roles on page 64 Step C: Reprocess deleted retention (Exchange) or soft deleted (Domino) messages on page 65 Step D: Rerun historical archiving activities on page 65 Step A: Start the EMC SourceOne management console Log in to a computer on which the EMC SourceOne management console is installed using a valid EMC SourceOne administrator account. Start the console. Step B: Adjust Worker and Native Archive computer roles To maintain expected service levels while operating the EMC SourceOne system from the disaster recovery site, adjust the roles for the workers and Native Archive servers to pick up the primary site's processing tasks. In environments in which the number of EMC SourceOne worker and Native Archive computers is less than the number of computers at the disaster recovery site, you should define a processing strategy applicable for the period of time you expect to be operating EMC SourceOne from the disaster recovery site. For example, you may want to dedicate the computers to journaling, indexing and searching activities while limiting historical archiving activities until the primary site is re-established. 64 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

65 Failover to the Disaster Recovery site Step C: Reprocess deleted retention (Exchange) or soft deleted (Domino) messages In EMC SourceOne Management environments, perform manual procedure to reprocess deleted retention (Exchange) or soft deleted (Domino) items. Exchange 1. Log into a worker computer using an account which has rights to all journaling mailboxes (e.g., the EMC SourceOne primary service account). 2. Mount each journal mailbox to this account so you can see all of them. 3. For each journal mailbox, click on the Inbox and select Tools > Recover Deleted Items. All deleted s are restored to the journal mailbox and available for archive again. Domino 1. Log into a worker computer using an account which has rights to all journaling mailboxes (e.g., the EMC SourceOne primary service account). 2. Open each EMC SourceOne journal database (e.g., ExJournal.nsf ExJournal1.nsf, ExJournal2.nsf, etc.) on the Domino server. 3. Click Soft Deletions. 4. Click Restore All at the top of the view. All soft deleted s are restored to the journal database and available for archive again. Step D: Rerun historical archiving activities Rerun any historical archiving activities that were within the RPO window. Refer to the EMC SourceOne Administration Guide for specific procedures. Step E: Rerun File Systems activities Rerun any EMC SourceOne for File Systems activities that were within the RPO window. Refer to the EMC SourceOne Administration Guide for information about configuring activities. Failover procedures 65

66 Failover to the Disaster Recovery site Step F: Rerun SharePoint activities Rerun any EMC SourceOne for Microsoft SharePoint activities that were within the RPO window. Refer to the EMC SourceOne Administration Guide for information about configuring activities. If the activity is recursive, you must delete the previous record from the activity list which is located in the EMC SourceOne sub-site of the SharePoint Central Administration site and then run the activity again. Step 12: Rerun the EMC SourceOne Archive Integrity Tool! IMPORTANT Ensure all historical archiving (performed in Step D: Rerun historical archiving activities on page 65) and indexing activities (started automatically as a result of running the tool earlier) have successfully completed before performing this step. Ensure that you run the version of the EMC SourceOne Archive Integrity Tool for the version of EMC SourceOne Management you are running. Log in to a Native Archive server at the disaster recovery site using the EMC SourceOne primary service account. Run the EMC SourceOne Archive Integrity Tool as follows: 1. Rerun the tool using the -FullMsgVerify option (using the log file generated when you ran the tool in Step 9: Run the EMC SourceOne Archive Integrity Tool on page 63). 2. Observe the results to ensure that all possible inconsistencies have been identified and corrected. A log is output indicating any remaining inconsistencies which were not possible to be corrected. For details on usage, command syntax, parameters, and the sequence of operations, see EMC SourceOne Archive Integrity Tool Operations on page EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

67 Failover to the Disaster Recovery site Step 13: Verify EMC SourceOne operations At this point, the EMC SourceOne system should be recovered and operational at the disaster recovery site. Run EMC SourceOne applications and ensure normal operation. Suggested operations include: Performing a full-text index search of archived content between the RPO and current time. Testing that shortcuts can be resolved through the Offline Access client and that and file shortcuts can be resolved using the universal URL. Step 14: (Optional) Run the SharePoint EBS integrity tool If your organization is using the optional EMC SourceOne for Microsoft SharePoint External BLOB Storage (EBS) feature, an EBS integrity tool (installed with the EBS provider software) is used to compare data in the EBS database on the SharePoint SQL Server with the BLOBs stored in the Native Archive. For details on using the tool, refer to EMC SourceOne EBS Integrity Tool Operations on page 95. Failover procedures 67

68 Failover to the Disaster Recovery site 68 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

69 4 Failback to the Primary Site This section provides a detailed procedure for performing a controlled failback of an operational disaster recovery site to the primary site. This chapter includes the following topics: Failback procedures Failback to the Primary Site 69

70 Failback to the Primary Site Failback procedures This procedure describes the manual steps required to re-establish operations at the primary site. Step 1: Stop EMC SourceOne processing at the disaster recovery site on page 71 Step 2: Ensure the exported DiskXtender registry keys are available on page 71 Step 3: Copy data back to the primary site on page 71 Step 4: Re-establish disaster recovery configuration for EMC SourceOne computers on page 72 Step 5: Create a "gold copy" of replicated data (recommended) on page 72 Step 6: Enable storage devices at the primary site on page 73 Step 7: Perform Active Directory changes for SQL and file servers on page 73 Step 8: Ensure dependent systems are online and accessible on page 74 Step 9: Recover the DiskXender configuration on page 74 Step 10: Start EMC SourceOne services on page 79 Step 11: Verify primary site is replicating to the disaster recovery site on page 79 Step 12: Verify EMC SourceOne operations on page EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

71 Failback to the Primary Site Step 1: Stop EMC SourceOne processing at the disaster recovery site Stop all EMC SourceOne processing at the disaster recovery site. Refer to the procedure described in the EMC SourceOne Release Notes for quiescing the system prior to an upgrade.! IMPORTANT Ensure the master computer at the disaster recovery site is disabled. There can only be one active master computer in a EMC SourceOne configuration. Step 2: Ensure the exported DiskXtender registry keys are available If you are not using DiskXtender in your environment, continue to Step 3: Copy data back to the primary site on page 71. Using the configuration described Exporting and replicating registry keys on page 45, ensure the DiskXtender registry keys are configured for export to one of the replicated drives and ensure that the keys are replicated back to the primary site. Step 3: Copy data back to the primary site Start the process to copy data from the disaster recovery site back to the primary site: Examples of copy back options for array-based storage include: Incremental copy back - If the original storage array hardware is back online at the primary site, perform an incremental copy back. Full copy back - If the original Symmetrix or CLARiiON hardware is no longer physically available at the site and is being replaced by another Symmetrix or CLARiiON, a full copy back is required. If using another storage solution (such as NAS), ensure the data is available at the primary site. Failback procedures 71

72 Failback to the Primary Site Step 4: Re-establish disaster recovery configuration for EMC SourceOne computers At the disaster recovery site, return the configuration to the hot/cold or hot/warm configuration that was in place before the disaster occurred: In a hot/warm configuration, worker and Native Archive computers are operational at the disaster recovery site and configured to work over the WAN with devices at the primary site. Review the information described in Considerations on page 24 which describes the types of activities that can be processed across a WAN connection. In a hot/cold configuration, workers and Native Archive computers at the disaster recovery site are either: Powered down Running with EMC SourceOne services set to disabled Step 5: Create a "gold copy" of replicated data (recommended) To prevent any issues from being carried during replication from the primary site to the disaster recovery site, effectively overwriting the good copy at the disaster recovery site, you should create a "gold copy" of the replicated data that can be used in the event that issues occur when you bring the primary site back online. The following examples provide the basic context for EMC Symmetrix SRDF, MirrorView or RecoverPoint array-based solutions. For other solutions, refer to the appropriate documentation. Examples include: SRDF - Create a Business Continuance Volume (BCV) from the Symmetrix at the disaster recovery site. MirrorView - Create a clone of the target MirrorView LUN from the CLARiiON at the disaster recovery site. RecoverPoint - Creating a "gold copy" of the data is not required with RecoverPoint since a good copy of the recovery data is always available from the RecoverPoint journal. 72 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

73 Failback to the Primary Site Step 6: Enable storage devices at the primary site Once all EMC SourceOne processing is stopped and the data between the primary and disaster recovery sites is synchronized, suspend replication and enable the storage array at the primary site. The following examples provide the basic context for EMC Symmetrix SRDF, MirrorView or Recoverpoint solutions. For other solutions, refer to the appropriate documentation. Examples include: SRDF - Read-write enable the devices in the primary site Symmetrix. MirrorView - Promote the primary image. RecoverPoint - Using the RecoverPoint CLI or GUI, enable target site host access by choosing the appropriate image. You can choose to recover to a bookmarked image created either manually, through the Group Set command or through the apply_parallel_bookmark command. For other storage solutions, ensure the storage is available at the primary site. Step 7: Perform Active Directory changes for SQL and file servers Redirect the CNAME aliases to SQL and file server host computers (or virtual host) names at the primary site. Do the following: 1. From Active Directory, repoint the CNAME to SQL and file servers to the servers or virtual hosts (if clustered) at the primary site. 2. Flush the DNS Resolver Cache: ipconfig /flushdns 3. Ping each server/virtual host by host name and confirm that the ping resolves to the correct IP address for each machine at the primary site. Failback procedures 73

74 Failback to the Primary Site Step 8: Ensure dependent systems are online and accessible Ensure that disaster recovery has been performed so that the following systems on which EMC SourceOne depends are online and accessible: Data sources (Mail servers, Microsoft SharePoint servers, and file systems) SQL Server, databases (SQL system and EMC SourceOne) and logs File servers hosting EMC SourceOne storage shares Follow your organization s procedures for restoring these systems. Step 9: Recover the DiskXender configuration If you are not using DiskXtender in your environment, continue to Step 10: Start EMC SourceOne services on page 79. If you are using DiskXtender in your environment, follow the sub-procedures in this step (in the order presented) to recover the DiskXtender configuration at the primary site: Step A: Verify DiskXtender connectivity to Centera on page 75 Step B: Import the DiskXtender registry keys to the DiskXtender host computer on page 75 Step C: Configure export of DiskXtender registry keys back to disaster recovery site on page 76 Step D: Clustered DiskXtender only. Add the Reset Disk Resources registry key on page 76 Step E: Restart the DiskXtender host computer or cluster nodes on page 77 Step F: Verify data consistency between DiskXtender and Centera on page 77 Step G: Start the EMC DiskXtender service on page 78 Step H: Inspect DiskXtender on page EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

75 Failback to the Primary Site Step A: Verify DiskXtender connectivity to Centera Verify EMC DiskXtender connectivity to EMC Centera using best practices described in the following documents: Connecting EMC DiskXtender for Windows to EMC Centera: Best Practices Planning This white paper can be found on EMC Online Support ( EMC DiskXtender Microsoft Windows Version Best Practices Guide EMC DiskXtender Microsoft Windows Version Installation Guide EMC DiskXtender Microsoft Windows Version Administration Guide Step B: Import the DiskXtender registry keys to the DiskXtender host computer As part of the disaster recovery solution configuration, the DiskXtender registry key is continuously exported to a designated storage location at the primary site (as described in Exporting and replicating registry keys on page 45). This enables you to update the DiskXtender host computer at the primary site with the last known configuration from the disaster recovery site. Import the DiskXtender registry key to the DiskXtender host computer at the primary site by: Deleting the existing DiskXtender key Merging the replicated key to the DiskXtender server at the primary site Follow this procedure to replace the registry keys at the primary site with the replicated keys from the disaster recovery site: 1. If using clustered DiskXtender, bring the cluster group online using Cluster Administrator. 2. Go the replicated location that contains the replicated DiskXtender registry keys. 3. Delete keys to remove old data during the merge. For example: regedit /S G:\del.reg REM Example of the del.reg file REM [-HKEY_LOCAL_MACHINE\SOFTWARE\EMC\DiskXtender] REM [-HKEY_LOCAL_MACHINE\SOFTWARE\EMC\XMS] Failback procedures 75

76 Failback to the Primary Site 4. Merge the key that was replicated from the disaster recovery site. For example: regedit /S G:\SOURCEONE_Recovery\DX.reg regedit /S G:\SOURCEONE_Recovery\XMS.reg 5. If using clustered DiskXtender, take the cluster group offline using Cluster Administrator. Step C: Configure export of DiskXtender registry keys back to disaster recovery site Using the configuration described Exporting and replicating registry keys on page 45, ensure the DiskXtender registry keys are configured for export to one of the replicated drives and ensure that the keys are replicated back to the disaster recovery site. Step D: Clustered DiskXtender only. Add the Reset Disk Resources registry key This step may not be necessary in all cases, but if DiskXtender disk resources do not appear in the Extended Drive Resource Groups in Cluster Administrator, a registry key can be configured to recreate it. Note: This procedure is described in the EMC Knowledgebase article esg To recreate the DiskXtender Disk resource, follow these steps: 1. Stop the EMC DiskXtender Service resource in Cluster Administrator. 2. Create a new STRING value in the following location: KEY NAME: HKEY_LOCAL_MACHINE\Software\Emc\DiskXtender\Setup VALUE NAME: ResetDiskResource VALUE: YES 3. Start the EMC DiskXtender Service resource in Cluster Administrator. Note the following: The DiskXtender Disk resource will only be recreated for extended drives that have already been configured in the DiskXtender Administrator. After performing this procedure, the DiskXtender Disk resource will appear in the Cluster Administrator. The registry key will delete itself. 76 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

77 Failback to the Primary Site Step E: Restart the DiskXtender host computer or cluster nodes Restart the DiskXtender host computer or cluster nodes. Step F: Verify data consistency between DiskXtender and Centera Assuming you followed the recommendation described in Purge configuration options on page 46, you must verify that the data in the DiskXtender extended drive is consistent with data on the Centera and, if necessary, rewrite data that was not replicated to the Centera at the primary site. Obtain the DXCenVerify utility. Download the appropriate version for the release of DiskXtender you are using. Registered EMC customers can download software from EMC Online Support ( Run the DXCenVerify utility to determine which volumes are on the extended drive but did not replicate to the primary (now local) Centera. The DXCenVerify utility outputs a text file indicating the missing volumes (MisFileRep_DateTime.log). After you run the utility and determine that there is a missing volumes issue, you have two options to ensure data consistency. Option 1 - Manually remove DiskXtender attributes from volumes Once you identify the missing volumes from the output log generated by DXCenVerify, you can remove DiskXtender attributes from files to enable DiskXtender to recognize the files that need to be moved to the Centera. 1. Copy all files indicated as missing to a temporary directory other than an extended drive (for example, c:\dx_temp). Copying the files will remove any DiskXtender attributes. 2. On the extended drive on which the copied files reside, perform a privileged delete. Note: You will need to start the EMC DiskXtender service to perform this step. Failback procedures 77

78 Failback to the Primary Site 3. With the EMC DiskXtender service now running, copy the files from the temporary directory back to the extended drive location. 4. Because the "Fetched" attribute has been removed, DiskXtender will move these files to the Centera based on existing move rules. 5. Run the DXCenVerify utility again to validate the files have been moved to the remote (now local) Centera. Option 2 - Remove DiskXtender attributes from volumes using the DxDmChk utility Alternatively, you can remove the DiskXtender attributes using the DiskXtender Data Manager Check (DxDmChk) utility. This utility can be used to remove the extended drive attributes from a file so that it can be moved to the Centera using the existing move rules. Note the following: Download the appropriate version for the release of DiskXtender you are using. Registered EMC customers can download software from EMC Online Support ( You can run the utility to generate a report to determine if there are issues. However, to actually remove extended drive attributes you must obtain an activation key from EMC DiskXtender Support and receive assistance with using the utility. Step G: Start the EMC DiskXtender service Start the EMC DiskXtender service (if not already started when performing the previous step). Step H: Inspect DiskXtender Use the DiskXtender File System Manager interface to review your DiskXtender configuration and verify that media is present and online. It may also be useful to run a DiskXtender file report to view the status of EMC SourceOne containers on the extended drive. 1. Right-click the EMC SourceOne container folder on the extended drive and select DiskXtender > File Report. 2. Accept the report defaults. 3. Verify that all containers have a status of fetched or purged. 78 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

79 Failback to the Primary Site Tips: Search for the word not to quickly find volumes that are not fetched or purged. Search for the word offline to find media which isn t online. Step 10: Start EMC SourceOne services Start the EMC SourceOne services on each computer in the configuration at the primary site in the following order: 1. Master 2. Native Archive server(s) 3. Worker(s) Step 11: Verify primary site is replicating to the disaster recovery site The data at the primary site should now be available for validation. Once you have validated the data at the primary site, you can begin replication from the primary site back to the disaster recovery site. Step 12: Verify EMC SourceOne operations At this point, the EMC SourceOne system should be recovered and operational at the primary site. Run EMC SourceOne applications and ensure normal operation. Suggested operations include: Performing a full-text index search of archived content between the RPO and current time. Testing that shortcuts can be resolved through the Offline Access client and that and file shortcuts can be resolved using the universal URL. Failback procedures 79

80 Failback to the Primary Site 80 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

81 A EMC SourceOne Archive Integrity Tool Operations This appendix describes how to use the EMC SourceOne Archive Integrity Tool and includes the following sections: Overview Command line description Permissions Sequence of operations Verifying operations Log file viewing EMC SourceOne Archive Integrity Tool Operations 81

82 EMC SourceOne Archive Integrity Tool Operations Overview The EMC SourceOne Archive Integrity Tool, version 6.6 supports the following environments: EMC SourceOne Management version 6.6 and higher EMC SourceOne for Microsoft SharePoint version 6.6 EMC SourceOne for File Systems version 6.6 The EMC SourceOne Archive Integrity Tool performs an extensive number of operations to ensure that a consistent environment exists at the disaster recovery site, with as close to zero data loss as possible, before normal EMC SourceOne operations are resumed. You run the utility (twice) in conjunction with the failover procedure described in Failover to the Disaster Recovery site on page 49 for details. Once a consistent environment is up and running at the disaster recovery site, you can perform a proactive failover to the primary site and continue normal processing. When performing a controlled failback, you do not need to run the EMC SourceOne Archive Integrity Tool again at the primary site, as the failback is planned and executed with the system quiesced. 82 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

83 EMC SourceOne Archive Integrity Tool Operations Command line description This section describes the command line syntax and optional parameters for the EMC SourceOne Archive Integrity Tool. Syntax ES1IntegrityTool -d Log File Path -t Temp Directory [options] The following table describes the command options. Table 2 EMC SourceOne Archive Integrity Tool command syntax Option Description -s Process all content greater than or equal to this date -d Destination path to store log file -t Temporary disk directory for utility. Contents of this directory are deleted each time the tool is run, so take care in defining this location. -e Optional end date to process all objects less than or equal to this date -DBServer -ArchiveDB -nosim -FullMsgVerify -MissMsgVerify Required. Name of the EMC SourceOne SQL Server computer hosting the Archive database Optional. Name of the EMC SourceOne Archive database (default is ES1Archive if not specified) Turn off simulation mode After SourceOne reprocessing occurs, perform full verification of all objects After SourceOne reprocessing occurs, perform missing object verification Command line description 83

84 EMC SourceOne Archive Integrity Tool Operations Required parameters! The EMC SourceOne Archive Integrity Tool requires at least three parameters to operate in a disaster recovery scenario. IMPORTANT By default the utility operates in simulation mode, meaning that it operates as read-only and generates a log file. Simulation mode allows you to review the log before actually performing the disaster recovery operation. After running in simulation mode, you can add the optional nosim parameter (see Optional parameters on page 85) to perform the actual disaster recovery operation. -d parameter -t parameter! Specifies the directory where a log file is generated. Each execution of the EMC SourceOne Archive Integrity Tool produces an output log file detailing its findings. This log file name includes a date and time stamp, as shown in the following example: ES1IntegrityTool_ T txt Defines a temporary disk directory that is used as a scratch location for the utility. IMPORTANT All contents of this directory are removed by this utility so take care in defining it. Do not use this directory for anything else except a scratch area for running the tool (for example, log file storage). If the path does not exist, it is created. -s parameter Denotes the starting date of objects to be verified, also referred to as the RPO. This parameter uses the following form in Greenwich Mean Time (GMT): yyyy-mm-ddthh:mm:ss For example, August 8 th, 2009 at 2:00 P.M. would be expressed as follows: T14:00:00 84 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

85 EMC SourceOne Archive Integrity Tool Operations Because EMC SourceOne writes real-time data to a number of different areas (which may or may not be on different devices and may or may not be on different replication schedules), the RPO date is the last known date where it can be assured that replication has completed successfully for all replication schedules. In most cases this date will not exceed a few days. The closer the RPO date is to the current date, the quicker the RTO is for EMC SourceOne to be operational at the disaster recovery site. Note: When identifying candidate objects, the tool looks at the file date on disk (last accessed). -DBServer Required parameter specifying the name of the of the computer hosting the EMC SourceOne Archive database. Optional parameters -e parameter -ArchiveDB -nosim parameter This section describes the optional parameters, or parameters that can be used in other scenarios. Used to verify objects between certain date ranges and should not be used during disaster recovery. Optional. Used only if the name of the EMC SourceOne Archive database is not the default name (ES1Archive) Instructs the EMC SourceOne Archive Integrity Tool to actually perform its functions. Unless this parameter is specified, the operation of the EMC SourceOne Archive Integrity Tool is read-only (except for generating a log file). Simulation mode allows you to review the log before actually performing the disaster recovery operation. Command line description 85

86 EMC SourceOne Archive Integrity Tool Operations -FullMsgVerify parameter Used after reprocessing has occurred during the failover process. This option is followed by the log file path that was generated during the initial run of the EMC SourceOne Archive Integrity Tool. All objects processed after the RPO date are queried against EMC SourceOne. This allows the operator to discover possible lost objects during the recovery process. For more details on using the verification options, see Verifying operations on page 92. -MissMsgVerify parameter Used after reprocessing has occurred during the failover process. This option is followed by the log file path that was generated during the initial run of the EMC SourceOne Archive Integrity Tool. Only the objects that were flagged as possibly lost are queried. This is a faster verification than FullMsgVerify. For more details on using the verification options, see Verifying operations on page EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

87 EMC SourceOne Archive Integrity Tool Operations Examples The following table provides usage examples for the EMC SourceOne Archive Integrity Tool. Table 3 The EMC SourceOne Archive Integrity Tool utility syntax examples Example Syntax -s T17:00:00 -d C:Utils\ES1IntegrityTool\Logs t C:Utils\ES1IntegrityTool\Temp -s T17:00:00 -d C:Utils\ES1IntegrityTool\Logs t C:Utils\ES1IntegrityTool\Temp -nosim -fullmsgverify C:Utils\ES1IntegrityTool\Logs\ES1IntegrityTool_ T txt -d C:Utils\ES1IntegrityTool\Logs Operation Provides a simulated recovery from 9/20/2009 at 5:00 PM to the current date. Provides an actual recovery from 9/20/2009 at 5:00 PM to the current date. Verify content after a recovery. This command is run after EMC SourceOne services are started and the system has fully recovered. Permissions You run the EMC SourceOne Archive Integrity Tool on a Native Archive server. Ensure that you log in to the server using the EMC SourceOne primary service account. Permissions 87

88 EMC SourceOne Archive Integrity Tool Operations Sequence of operations This section describes in detail the sequential operations that the EMC SourceOne Archive Integrity Tool performs when recovering from a failover. Each step includes details about the operation being performed as well as prescriptive information that you can use to troubleshoot issues you may encounter when running the utility. Note: For details on subsequently running the utility to verify the environment after EMC SourceOne is operational, see Verifying operations on page Parse command line options The utility parses the command line options entered by the user to determine the scope of operations. Refer to Command line description on page 83 for a detailed description of the command line options available when you run the utility. 2. Add volumes to the candidate volume set Add volumes as candidates for reprocessing by identifying volumes that have a date (from which the volume name is comprised in the format YYYYMMDDHHMMSS) greater than the RPO. The types of volumes include: SQL volumes A table named Volume in the EMC SourceOne SQL database contains all of the volumes archived by EMC SourceOne. This table is scanned for all volume names greater than the RPO date. Each matching entry is added to the candidate volume set. 88 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

89 EMC SourceOne Archive Integrity Tool Operations Container drive volumes During the recording process, volumes are written as container files to one of the following locations, depending on your configuration: Network-accessible storage The DiskXtender extended drive (if used). This drive is scanned for all container files created on or after the RPO data and added to the candidate volume set. The EMC Centera (if storing containers directly on Centera without DiskXtender). The Centera is scanned for all container files created on or after the RPO data and added to the candidate volume set. At this point the candidate set contains all volumes found in EMC SourceOne that were created after the RPO date. The utility has captured information on which volumes are in SQL or closed container files. Ideally, every volume will exist in both places. Note: Volume information can also exist in a full-text index. This information is identified during the full-text index scan, which is described later in this document. 3. Add objects found to the candidate object set Add objects as candidates for reprocessing. The utility accomplishes this by identifying objects that have a receive date greater than the RPO date as well as objects that are orphaned, or objects that are in containers but not in the SQL database. Candidate objects are added from the following sources: SQL objects A table named Messages in the EMC SourceOne SQL database is scanned for all objects with a receive date greater than the RPO date. All matching objects are added to the candidate object set. Objects in Candidate Volumes If a historical archive task was run after the RPO date, objects could exist in which the receive date is earlier than the RPO date, but were actually archived after the RPO date. To ensure this situation is accounted for, all candidate volumes are scanned, and every object contained in each candidate volume is included in the candidate object set. Sequence of operations 89

90 EMC SourceOne Archive Integrity Tool Operations Message Center - During the archive process, EMC SourceOne writes objects to a Message Center directory. These objects remain in the Message Center until a volume reaches a certain size or a certain amount of time passes. The utility scans the Message Center and places any objects it finds into the candidate object set. In the case of messages, each message is validated for integrity. If the message has large attachment (in the large content, or LC, sub-directory) it is loaded into the message object. The message ID is then calculated and compared to the ID used for the file name. If the ID's don't match, the message is considered corrupt. Archive location - The archive location holds closed container files. The share is enumerated and each container created after the RPO time is opened and validated. Each object is added to the candidate object set. At this point EMC SourceOne Archive Integrity Tool has a comprehensive set of objects that it identifies as processed after the RPO date. It also contains any possible orphaned objects found on the disk. The candidate object set is stored in a SQL table called MsgMap which resides in the EMC SourceOne Archive SQL database. Every time EMC SourceOne Archive Integrity Tool is run, it removes the previous MsgMap table from SQL and recreates a new one. This means that after successful operation the MsgMap table is left in the EMC SourceOne database. This allows interrogation by the user of possible offending objects. This table can be removed after the disaster recovery site is operational. 90 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

91 EMC SourceOne Archive Integrity Tool Operations 4. Add full-text indexes found to the candidate index set Add full-text indexes as candidates for reprocessing by identifying indexes whose last modified date is greater than the RPO date. To identify indexes that may have been in a pending state (not written to an index yet) when the outage occurred, the utility also scans all indexes for candidate volumes and includes them in the candidate index set. This is done because it is possible for objects to be in an index pending state during disaster recovery, and the actual index may not have been written to yet. The utility also automatically schedules full-text indexes for re-indexing. Note: Although there may be an RTO impact to performing the re-indexing of all candidate indexes, it is the best method to guarantee the consistency and integrity of the EMC SourceOne environment. 5. Remove objects from SQL The candidate object set is traversed and all objects that exist in SQL without having been assigned to a volume are removed from SQL. This will prevent duplication detection during the reprocessing of objects. Sequence of operations 91

92 EMC SourceOne Archive Integrity Tool Operations Verifying operations After successful operation of the EMC SourceOne Archive Integrity Tool, a log file is generated which lists the entire candidate object set and a possible missing object set (which is always smaller than the candidate object set). The tool can be run using optional parameters to replay the log file against one of the object sets. Query and display results You can use the following command line options when running the tool and specifying the log file created from a previous run: FullMsgVerify This command line option selects the entire candidate object set -MissMsgVerify this command line option selects only the possible missing object set During verification, each object in the selected set is queried for from EMC SourceOne and the results are displayed. These results can be cross-referenced with the SQL MsgMap table to help find any possible lost objects. Examples of the EMC SourceOne Archive Integrity Tool command using each of these options include: -MissMsgVerify D:\ES1IntegrityTool_ T txt -d D:\Logs -FullMsgVerify D:\ES1IntegrityTool_ T txt -d D:\Logs 92 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

93 EMC SourceOne Archive Integrity Tool Operations Query and retrieve results You can also specify an option called GetMsgDuringVerify which, when used with either the FullMsgVerify or MissMsgVerify option, retrieves the object from EMC SourceOne and verifies its integrity. Examples of the ES1IntegrityTool command using this option include: -MissMsgVerify D:\ES1IntegrityTool_ T txt -d D:\Logs -GetMsgDuringVerify -FullMsgVerify D:\ES1IntegrityTool_ T txt -d D:\Logs -GetMsgDuringVerify Verifying operations 93

94 EMC SourceOne Archive Integrity Tool Operations Log file viewing Log files written by the utility are in binary unicode format, and therefore will not display well using a basic text viewer such as Notepad. Use an appropriate viewer (such as Microsoft Internet Explorer). 94 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

95 B EMC SourceOne EBS Integrity Tool Operations This appendix describes how to use the EMC SourceOne for Microsoft SharePoint External BLOB Storage (EBS) Integrity Tool and includes the following sections: Overview Recovery scenarios Using the EBS Integrity Tool EMC SourceOne EBS Integrity Tool Operations 95

96 EMC SourceOne EBS Integrity Tool Operations Overview EMC SourceOne for Microsoft SharePoint also provides External binary large object (BLOB) storage, effectively offloading the storage of binary large objects from the SharePoint SQL Server and storing them using EMC SourceOne. EBS is configured at the farm level. This feature includes an EBS Provider database which is installed on the SharePoint SQL Server to maintain identification information for BLOBs stored externally in the Native Archive by EMC SourceOne for Microsoft SharePoint. EBS simply utilizes the Native Archive to access storage EBS content is not archived. Therefore, EBS content is not subject to indexing, retention or available for search using EMC SourceOne Search. However, it is available for search using SharePoint-side search interfaces. If your organization is using the optional EMC SourceOne for Microsoft SharePoint External BLOB Storage (EBS) feature, the EBS Integrity Tool (installed with the EBS provider software) is used to compare data in the EBS database on the SharePoint SQL Server with the BLOBs stored in the Native Archive. 96 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

97 EMC SourceOne EBS Integrity Tool Operations Recovery scenarios The EBS Integrity Tool can accommodate several recovery scenarios. BLOB records missing from EBS database, BLOBs exist in Native Archive In this scenario, one or more BLOB records are missing from EBS database, but the BLOBs exist in the Native Archive. Details: The missing BLOBs are in the same archive folder that is currently configured for EBS. Some missing BLOBs are located in different archive folders than the currently configured folder because of the configuration change. In this case, the tool will search for the BLOBs in all valid archive folders on the Native Archive server and restore the correct database records. BLOBs missing from Native Archive, BLOB records exist in EBS database In this scenario, one or more BLOBs are missing from Native Archive but BLOB records still exist in the EBS database. Details: The missing BLOBs are located in the cache folder that was currently configured for EBS. Some missing BLOBs are located in some other folders that were previously used as a cache folder. In this case, the tool will search for BLOBs in the cache folder. You can optionally instruct the tool to add additional folders to search for missing BLOBs. The tool provides an option to manually disable the local cache garbage collection function so that files in the cache will not be automatically deleted. Note: If you run out of disk space, the garbage collection will be run regardless of the setting you configure. Recovery scenarios 97

98 EMC SourceOne EBS Integrity Tool Operations BLOBs missing from Native Archive and EBS database In this scenario, one or more BLOBs are missing from both the Native Archive and EBS database, and cannot be found in the local cache or additional folders. It is not possible to recover them in this case. Using the EBS Integrity Tool This section describes how to use the EBS Integrity Tool. Launching the tool The EBS Integrity Tool is installed with the EBS Provider software and can be launched from the following location: %Program Files%\EMC SourceOne\EBS Provider\Emc.SourceOne.ES1EBSIntegrityChk.exe When you launch the tool, a dialog displays which enables you to: Automatically scan the entire farm. Skip the automatic scan and manually scan individual site collections. Depending on your environment, do one of the following: For a small farm with fewer BLOBs and site collections, you should use the automatic scan option to scan the entire farm. For larger farm, you can perform a manual scan of individual site collections to better control the duration of the scan. 98 EMC SourceOne Version 7.1 Disaster Recovery Solution Guide

99 EMC SourceOne EBS Integrity Tool Operations Reviewing scan results Once an automatic or manual scan is performed, the EMC SourceOne EBS Integrity Check Results page displays. Figure 4 Scan results If site collections are scanned, they will be sorted by the status of the scan results. There are three possible icons to show the status: Critical The tool failed to verify some of all BLOBs because of unexpected errors. Warning There are some BLOBs missing. OK The site collection is intact with no BLOBs missing. Using the EBS Integrity Tool 99