WHITE PAPER. Recovery of a Single Microsoft Exchange 2000 Database USING VERITAS EDITION FOR MICROSOFT EXCHANGE 2000

Transcription

1 WHITE PAPER Recovery of a Single Microsoft Exchange 2000 Database USING VERITAS EDITION FOR MICROSOFT EXCHANGE 2000 June,

2 TABLE OF CONTENTS Overview...3 Background...3 Traditional Backup Processes...4 Traditional Restore processes...4 How Does It Work?...5 Overview of Hot Snapshots...6 Recovery of an Exchange Database...6 Single Database Recovery...7 Single Database Recovery Procedure...8 Preparing for Another Snap...16 Summary...19 References

3 OVERVIEW This document contains the best practices to perform a recovery of a single Microsoft Exchange 2000 database using VERITAS Edition 1.1 for Microsoft Exchange VERITAS Edition for Microsoft Exchange 2000 is an integrated suite of industry-leading VERITAS storage management technologies engineered specifically to improve manageability, increase availability, and ensure quick recovery of Microsoft Exchange servers. VERITAS Edition for Microsoft Exchange 2000, referred to in this document as VERITAS Edition, provides Exchange administrators with the ability to backup Microsoft Exchange data with minimal disruption to the Exchange services or users, as well as rapidly recover Exchange data in the event of corruption. A key feature of VERITAS Edition is the FlashSnap Utility. The VERITAS Edition FlashSnap Utility has two components. The FlashSnap Backup and Recovery feature provides the ability to create split-mirror snapshots that can be used off-host for operations such as backup. This virtually eliminates the performance impact of backup operations on the production Exchange server, allows the backup server to backup the snapshot at local fibre channel speeds, and reduces the LAN data traffic impact of online backups. The FlashSnap Quick Recovery feature provides the ability to create multiple on-host splitmirror snapshots for quick recovery of Exchange data. These snapshots can be used to restore corrupted Exchange storage groups directly from disk, reducing the time needed to bring Exchange back online after a failure. BACKGROUND systems have become a critical part of many organizations communication infrastructure. Microsoft Exchange is the leading provider of services. Historically, Microsoft Exchange contained only one database and one set of transaction logs. Structuring Exchange in this fashion caused a high impact for any given Exchange failure because all users of the Exchange database were affected. Also this structure for Exchange limited how many users an Exchange server could handle, since the size of the single database was a major limiting factor for the time to recover the database. Microsoft Exchange 2000 introduced the concept of storage groups. There can be up to four storage groups on a single Exchange Server and each of the storage groups can contain up to five databases. All of the databases in the storage group share the same transaction logs. The theory is that administrators can keep the size of the databases down while serving more users. Theoretically, smaller databases are not as prone to corruption and are quicker to restore from tape. Storage groups also make it easier to cluster Exchange in an Active/Active configuration In the real world, administrators are not always able to follow best practices for database sizes. This is due to a number of reasons. Users tend to retain their , causing the databases to grow quickly. Deploying additional Exchange systems to handle this growth takes time and equipment. As a consequence, information stores can get quite large. This presents a recovery nightmare, because the larger the database, the more susceptible it is to spontaneous corruption, and the longer it will take to recover from tape. In the case of some very large databases, it could take a day or more. Having more databases increases the likelihood that recovery of a single database will be required. Fortunately, having multiple databases with Microsoft Exchange 2000 minimizes the impact on other users. To better understand the VERITAS Edition Quick Recovery process, we must look at what occurs during traditional Exchange backups and restores using tapes. VERITAS Edition takes advantage of many of these processes, but facilitates faster restores by maintaining local, full, point-in-time copies of the database(s). Recoveries are much quicker because the databases do not need to be copied from another source. This can reduce restore times from hours to minutes. 3

4 TRADITIONAL BACKUP PROCESSES When running a full backup on an Exchange system, a backup agent interfaces with the Exchange backup API and puts the database into a Hot Backup mode. During this process, the database is checked for basic corruption, transaction logs are committed and truncated (deleted), and the database is backed up to tape. In the event of a complete failure, Exchange can be restored to the point of the most recent backup. If the transaction logs are available, the restore process will replay the logs, and the database can be rolled forward to the time of failure. This is why an administrator can restore an Exchange database after a backup without data loss. To accomplish this, the administrator must have all of the transaction logs since the most recent full backup. There are two types of backups: differential and incremental. These backup types do not copy the database files to backup media but only copy the transaction log files. Since the database files are not part of the incremental or differential backup, no page verification or data checksums are performed on the Exchange database during the backup to ensure database integrity. The incremental backup type copies just the accumulated transaction log file changes since the last full or incremental backup-to-backup media. The committed transaction log files are then truncated (deleted) after they are saved. Restoring from incremental backups requires a full backup of the database plus all the incremental backups made since that full backup was done. Restoring from incremental backups is complicated by the many backup data sets required for a successful restore. The differential backup again only backs up transaction log files. But each time an incremental backup is created, it backs up all the transaction log changes since the last full backup and does not truncate the log files. Restoring from incremental backup thus only requires the last full backup and the last differential backup. A differential backup can take longer and use more media because the transaction logs are not truncated after each backup. However the differential backup recovery is simpler because only two backup data sets are needed for a successful restore. TRADITIONAL RESTORE PROCESSES Before any recovery of Exchange data can take place, an Active Directory Domain Controller and a Global Catalog server must be available. This document presumes that those resources are available. Recovery of these resources is a critical part of any Exchange Server full recovery plan. Restoring a corrupted database can take time. For this reason, administrators often spend time trying to fix the database before performing a recovery. Recovery can take nearly twice as long as it takes to perform a backup. There are many organizations with Service Level Agreements (SLA) requirements for Exchange that technically cannot be met. Many system administrators find that recovery operations on Exchange servers are difficult to plan for, because database corruptions can occur spontaneously. Similarly, an -borne virus can create Exchange storage management issues with little or no warning. Thus, administrators often develop robust backup plans in an effort to cover all possible recovery situations. VERITAS Edition can provide the administrator with a substantial advantage in this planning. 4

5 HOW DOES IT WORK? VERITAS Edition is based on VERITAS Volume Manager for Windows and VERITAS FlashSnap technology. VERITAS Edition includes the VERITAS Edition FlashSnap Utility that communicates directly with the Microsoft Exchange Server to automate and manage the snapshot process. The FlashSnap Utility has two components: the FlashSnap Backup and Recovery feature and the FlashSnap Quick Recovery feature. FlashSnap Backup and Recovery is an automated feature of VERITAS Edition. The vxexcmd command automatically coordinates and executes the backup of the volumes that contain Exchange objects and transaction logs files. FlashSnap Backup and Recovery can make local point-in-time or off-host point-in-time copies of the Microsoft Exchange data stores and transaction logs. The FlashSnap Backup and Recovery feature uses VERITAS FlashSnap technology to create point-in-time copies of Exchange objects. FlashSnap Backup then invokes a specified backup application (for example, NetBackup or Backup Exec) to copy the point-in-time data to tape or other media. During a recovery operation, FlashSnap Backup and Recovery copies the restored data back to the volumes that contain the Exchange database and transaction log files. The FlashSnap Quick Recovery feature uses the VERITAS FlashSnap technology to make a snapshot copy of production Exchange storage group databases and mounts them locally on the Exchange server for quick access. If database corruption in the storage group should occur, the FlashSnap Quick Recovery feature can restore the production volumes from the local snapshot copy. The user can make multiple snapshot copies and has the option to refresh existing copies, that is, resynchronize copies with the current Exchange data on the production volumes. A point-in-time set of Quick Recovery volumes for a single storage group is referred to as an SGIMAGE. In each SGIMAGE there is a snapshot copy of each volume in the storage group. Each SGIMAGE has a unique name generated by the FlashSnap Quick Recovery feature. The unique name is used to manage the SGIMAGE using the FlashSnap Utility. It is recommended that Quick Recovery images be created at the end of each online full backup of Exchange using the post-processing script capabilities of the backup application. This post-processing capability is available in VERITAS NetBackup and VERITAS Backup Exec. See the VERITAS Rapid Recovery for Microsoft Exchange 2000 Solution Guide for more details on creating and updating Quick Recovery images as part of an Exchange backup with NetBackup in particular. VERITAS Edition is designed to work in conjunction with a complete Exchange backup strategy. A full backup of Exchange through the Exchange Backup API is the best way to catch corruption without shutting down the database. The full, online backups also commit the transaction logs and prune them. VERITAS strongly recommends that customers use standard Exchange online, full backups to create backup tapes. However the VERITAS FlashSnap Quick Recovery images excel as a first line of defense for damaged Exchange databases. 5

6 OVERVIEW OF HOT SNAPSHOTS The VERITAS Edition Hot Snapshot feature provides the ability to create a snapshot of an Exchange storage group without dismounting the storage group databases. This feature allows the Exchange production volumes to continue to operate without interruption during backups. When performing a hot snapshot, there are two types of issues to consider during the break-off of the mirror image: database physical corruption and database logical inconsistencies. A physically corrupt database is a database with internal damage resulting in database pages that cannot be read by the Exchange application. Microsoft supplies two tools with Exchange that can be used to analyze a database and determine its physical condition Esefile and Eseutil. Both tools can check for database physical corruption by doing database page comparison and checksum validation. The tools read every database 4K page in sequential order, and the page number and the page checksum is verified. If the database file passes with no errors, then the snapshot of the database is considered physically consistent. The VERITAS Edition Hot Snapshot feature ensures that the database has no physical corruption by utilizing Volume Manager to perform the snapshot, and then using the Esefile or Eseutil programs to inspect the database as the user directs. If physical corruption is detected, the hot snapshot is rejected and another hot snapshot attempt is made -- up to a user-defined number of tries. During normal operations, an Exchange database, by design, is physically consistent, but logically inconsistent, or dirty. (Exchange 2000 Service Pack 2 and later does not report the database logical state as "Consistent" or "Inconsistent", but as "Clean Shutdown" or "Dirty Shutdown." The meaning of "Clean Shutdown" is the same as "Consistent," and the meaning of "Dirty Shutdown" is the same as "Inconsistent.") A logically inconsistent Exchange 2000 database file is a file where all of the outstanding transactions have not yet been written to it. During normal operation, Exchange 2000 database files are always logically inconsistent because there is information in the Exchange application cache that has not yet been physically written to the database file. The only time the Exchange database is in a consistent state is when the database is cleanly dismounted. In breaking off the Exchange database snapshot while the Exchange server is actively using the database, VERITAS Edition always creates logical database inconsistency because there are transactions in the logs that have yet to be committed. This snapshot of logically inconsistent database with all the transaction logs in place is recovered and made logically consistent when the hot snapshot recovery feature initiates a soft recovery when mounting the database. Soft recovery allows the Exchange ESE database engine to replay any transaction log files required to bring the database into a logically consistent, or clean shutdown state. RECOVERY OF AN EXCHANGE DATABASE To recover a database, the volume containing the database is simply resynchronized by the FlashSnap Utility with the snapshot volume. After the volume is resynchronized, all of the databases in the storage group must be briefly taken offline. When the recovered database is remounted, it is made logically consistent and the transaction logs are replayed. Replaying the logs requires the greatest amount of time, which is dependent on the number of logs to be replayed. Next, the remaining databases in the storage group are brought online. Even in the most active environments, this should only take a few minutes. In most cases, the entire recovery procedure could be completed in less than fifteen minutes. 6

7 SINGLE DATABASE RECOVERY Based on the Microsoft Exchange 2000 design of storage groups and databases, there may be circumstances in which only one database of a storage group needs to be recovered. Storage group Quick Recovery images are great tools to recover the database in these circumstances. Quick Recovery images, named SgImage000, SgImage001, etc., are created using the vxsgimg c command. You can recover the entire storage group and all its databases with a Quick Recovery image by using the vxsgimg r command. But for single database recovery you do not need to overwrite all the production volumes with the snapshot volumes, but instead only snapback the one volume that has the corrupted database. To facilitate single database recovery, the vxsgimg command has a special option that can be used to free up an existing image. The m option removes the image information from the Edition's image catalog, and un-mounts the image s volumes from their mount point. Once the image's volumes are released they can be manipulated to recover only a single production volume. The m option prints out the device path of all the volumes in the image making it easier for the user to reference which snapped volumes were released. Single database recovery has the limitation that only one database can exist on a production volume. The VERITAS Edition Storage Configuration Wizard enforces this rule when it creates or encapsulates Exchange storage groups. However, if this limitation is not met, you can use the vxsgimg r command to recover all databases in a storage group. CAUTION: Do not have any open handles to the production Exchange data volumes or the recovery image snapshot volumes before beginning the single database recovery procedure. Close all Windows Explorer screens and consoles that might access the Exchange data. The following procedure describes the recovery of a single database. The example shows how to do the following: Locate the production database volume to recover, Release an existing image's snapshot volumes for the storage group, Perform a manual snapback of the snapshot volume containing the good database files, Mount the Exchange database so that Exchange replays the current transaction logs of the storage group into the recovered database. In this example, ee-it.ini is the name of the initialization file for this storage group. Refer to the VERITAS Edition 1.1 for Microsoft Exchange User's Guide for more information about how to create and populate a FlashSnap Utility initialization file. This guide also provides a complete description of the vxsgimg command and its options. NOTE: The other databases in the storage group will need to be briefly dismounted to recover the failed database. Note that when performing this procedure, you should make sure that your folder option is set to display file extensions. This will be important when you locate the checkpoint file for deletion, so that you do not inadvertently delete the wrong file (for example, E00.chk rather than E00.edb). 7

8 SINGLE DATABASE RECOVERY PROCEDURE 1. Determine which storage group database needs to be recovered using the Microsoft Exchange System Manager or other standard Microsoft Exchange utilities, such as Eseutil.exe. In the following figure, the IT database in the SG1 storage group cannot be successfully mounted and needs to be recovered because it is missing a database file. 2. Determine the Quick Recovery image that can be used to recover the database, based on the storage group name and the time stamp. Open a Windows command line console and use the vxsgimg l command to get a list of available images. See the example below. The time stamp indicates the date and time when the image was created, or the last date and time it was updated with production data. Usually the most recent image is the best candidate for recovery since it requires less time to replay the current transaction logs into the database. 8

9 In this example, the SgImage001 Quick Recovery image is used to recover the IT database of the SG1 storage group. 3. Release the image to be used for the recovery from the FlashSnap Quick Recovery image catalog by opening a Windows command line console and entering the vxsgimg m command. The volumes released from the image are identified as "\Device\HarddiskDmVolumes\DiskGroupName\SnapVolumexx." For example, \Device\HarddiskDmVolumes\DiskGroup001\SnapVolume04 is a snapshot volume released from the SgImage001 image. The SnapVolume04 identifies the Volume Manager name for the snapshot volume. Note: You need to refer back to this information so do not close this console window. 9

10 4. Determine the production volume to recover by using the VERITAS Enterprise Administrator GUI. Start the VEA console and connect to the host running the Exchange Server. Under the host name, expand the Exchange Servers->Exchange Server Name->Storage Group hierarchy to locate the database you want to recover. After you click on the database, the right pane displays information about the database volume, including disk group, disks used, and the path for the data files. In the following figure, the IT database volume is mounted at C:\ExchangeMountPoints\DiskGroup001\Volume002, and the volume resides on Harddisk2 and in DiskGroup

11 Next, under the host name, expand the Disk Group menu. Highlight the disk group that contains the volume and locate any disk that is part of the database volume. Double click on that disk. Under the Volumes tab, you can find the volume with the data files. If multiple volumes are on the disk, explore them to locate the data files. Note the drive letter and volume label for this volume. In the following figure, there is one volume on Harddisk2 of DiskGroup001; that is the IT database volume. 11

12 5. Perform a snapback of the snapshot volume containing the database data files. Using the procedures in Step 4 above, find the production volume you want to recover. Double click on the volume. In the right panel there is now a Snapshot Volumes tab. Click on the Snapshot Volumes tab to see a list of snapshot volumes. The volume names listed under the Snapshot Volume tab are listed as "SnapVolumexx." Locate the correct volume to snapback by matching the "SnapVolumexx" portion of one of the volumes released from the image (see step 3 above ) to the same name string in the SnapVolume tab. Below the SnapVolume06 is associated with the IT database production volume and the released Quick Recovery image, so it is selected. 12

13 6. Right click on the snapshot volume, and select Snap->Snap Back. 7. When prompted, select Resynchronize using the snapshot and click OK. This action reattaches the snapshot volume to the production volume and begins resynchronizing the production data from the snapshot data. The production data is immediately recovered. There is no need to wait for resynchronization to complete. Caution: Resynchronize using the original volume is the default option. If this option is applied, the image will no longer be available for recovery. 13

14 8. Dismount all the databases in the storage group using the Exchange System Manager. 9. Delete the transaction log checkpoint file in the Transaction Log folder. a. First locate the transaction log production data by selecting the storage group in the VEA display. The right panel shows the path. 14

15 b. Use Windows Explorer to locate the transaction log checkpoint file. In this example it is E01.chk. Removing this file allows the current transaction logs of the storage group to replay into the recovered database Use Exchange System Manager to mount the recovered database. The database is made consistent and the transaction logs replayed into it. 11. Finally, mount the remaining databases in the storage group. The Exchange storage group and all its databases are now online. 1 Note that Microsoft does not support the removal of the checkpoint file to force a soft recovery even though this is an accepted industry practice. 15

16 PREPARING FOR ANOTHER SNAP The procedure to restore a single database used one of the volumes of the image. The procedure did not, however, use all of the volumes, so it is necessary to reattach the remaining images volumes to their production volumes at this time. You should manually reattach the image snapshot volumes of all other production databases and transaction log volumes in the storage group EXCEPT the volume that was recovered in the previous section. To reattach the remaining image snapshot volumes, follow these steps: 1. Determine the non-recovered production database volumes. Start the VEA console and connect to the host running the Exchange Server. a. Under the host name, expand the Exchange Servers->Exchange Server Name->Storage Group hierarchy to locate the databases you did not recover. After you click on the database, the right pane displays information about the database volume, including disk group, disks used, and the path for the data files. In this example, the Accts database file is located on the K: volume, on Harddisk0 of DiskGroup

17 b. Next, under the host name, expand the Disk Group menu. Highlight the disk group that contains the volume and locate any disk that makes up the volume. Double click on that disk. Under the Volumes tab, you can find the volume with the data files. If multiple volumes are on the disk, explore them to locate the data files. c. Double click on the volume. In the right panel there is a Snapshot Volumes tab. Click on the Snapshot Volumes tab to see a list of snapshot volumes. The volumes names listed under the Snapshot Volume tab are listed as "SnapVolumexx." Locate the correct volume to snapback by matching the "SnapVolumexx" portion of one of the volumes released from the image with the same name string in the Snapshot Volume tab. In the example, SnapVolume0005 was part of the Quick Recovery image and needs to be reattached to the production volume. 2. Right click on the snapshot volume, and select Snap->Snap Back. 17

18 3. When prompted, select Resynchronize using the original volume and click OK. 4. The transaction log volume must also be reattached. You can follow the procedure outlined in steps 1-3 above with the following exception. In step 1a, locate the transaction log production volume by clicking on the Exchange Servers->Exchange Server Name->Storage Group. The right pane displays the information about the transaction log volume. Proceed with step 1b above. 5. From a console window, verify that the sgimage no longer exists by running vxsgimg l c:\ee\sgx.ini. Now, create another image by running vxsgimg c c:\ee\sgx.ini n 1. 18

19 SUMMARY Exchange system administrators find that traditional tape recovery operations of Exchange Server databases are difficult to plan, take a long time, and disrupt users. VERITAS Edition provides the system administrators a substantial advantage with the FlashSnap Quick Recovery images. These local, snapshot images can be used to quickly recover an entire storage group or only a single database. The time to accomplish the recovery keeps user disruption to a minimum. VERITAS Edition snapshot images provide an excellent first line of defense for Exchange database problems, thus supplementing traditional Exchange backups. REFERENCES VERITAS Edition 1.1 for Microsoft Exchange 2000 User s Guide VERITAS Rapid Recovery for Microsoft Exchange 2000 Solution Guide 19

20 VERITAS Software Corporation Corporate Headquarters 350 Ellis Street Mountain View, CA or For additional information about VERITAS Software, its products, or the location of an office near you, please call our corporate headquarters or visit our Web site at 20