TECHNICAL OVERVIEW File Data with Snap Enterprise Data Replicator (Snap EDR) Contents 1. Abstract...1 2. Introduction to Snap EDR...1 2.1. Product Architecture...2 3. System Setup and Software Installation...3 3.1. Installing the Snap EDR Management Console Software...3 3.2. Installing Snap EDR Agents...3 4. Data Management Tool...3 4.1. Default Functionality...3 5. Configuring an Job...3 5.1. Remote Data Details...4 5.1.1. Files Generated...4 5.2. Host Report File...4 5.3. Host File Contents...4 5.3.1. Host Information Section...5 5.3.2. Volume Scan Information...5 5.3.3. File Scan Information...5 5.4. File Report File...5 6. Best Practices -...6 6.1. Access to System Information...6 6.2. Append Results to E-Mail Notification...6 6.3. File Scan Report vs. Aggregate File Information...6 6.4. Source Directories Omitted...6 6.5. Source Directories Specified...6 6.6. Debugging...6 7. Monitoring Performance...6 7.1. Viewing Job Statistics...6 7.2. Viewing the Job Log Files...6 8. Verifying Performance...7 8.1. Report Types...8 9. Conclusion...8 1. Abstract This white paper provides an overview of the Snap Enterprise Data Replicator (Snap EDR) product, and details the components and configuration of the Snap EDR data management tool. The tool allows users to schedule a job to determine storage utilization. This solution contains a rich set of reporting criteria that users can set to report on various aspects of the file system, such as disk usage for files that have not been accessed for a certain amount of time. This solution can be used to help formulate archival policies which would be implemented by the Remote Archive solution. Snap EDR runs on the following platforms: Snap Servers powered by GuardianOS, Windows clients, and UNIX/Linux clients. Common business needs for taking inventory of file data include, but are not limited to, the following: Determining storage utilization to identify files that are never or rarely used Creating archive policies to remove rarely used files from the system and store them elsewhere Creating additional storage space by removing rarely used files from the system 2. Introduction to Snap EDR Snap EDR includes three data management tools to automate the secure flow of data between systems: The Aggregate tool allows jobs to be scheduled to transfer files from multiple agents to a single target agent The Distribute tool allows jobs to be scheduled to transfer files from one source machine to one or more target machines The Replicate tool allows jobs to be scheduled to transfer files between two systems Snap EDR also includes the Advanced Data Management (ADM) Suite that groups together the functionality of Remote, Remote Archiving, and Remote Backup and Restore. These solutions provide exceptional functionality to create and deploy policies for solving remote data management problems such as variability, dissimilar computing platforms, security needs, and data integrity.
2 The Advanced Data Management Suite performs the following tasks:, to determine storage utilization Archive, to move selected data for permanent archive Backup, to protect data Restore, to recover backup data The following describes a typical use case of the Advanced Data Management Suite: Remote Locations Central Sites Archive Separate white papers provide details on the Aggregate, Distribute and Replicate tools and each tool in the ADM suite, and are available from the following location: http://www.adaptec.com/whitepapers. 2.1. Product Architecture Snap EDR consists of a Management Console that is installed on a Snap Server powered by GuardianOS, and a collection of agents running on the other servers/workstations involved in the transfer. The Management Console coordinates and logs the data transfer activities carried out by the distributed agents. The Snap EDR agents are responsible for the actual transfer of data. Restore Restore Users access the Management Console using a web browser that allows them to configure Snap EDR and to create data management and transfer jobs from any location. The following diagram shows an example of Snap EDR implementing the data management tool: Remote Locations Source Agents SNAP SERVER 4500 SNAP SERVER 18000 Disk-Based Backup Secondary DR Sites Figure 1: ADM Configuration Off-Line Backup Report File Report File SNAP EDR AGENT SNAP EDR AGENT SNAP EDR AGENT Report File 1. The user has a set of locations (with Snap EDR Agents) that are part of the enterprise but connected over the wide area network (WAN). (Note: agents can be anywhere on the network.) 2. The user runs a remote inventory job to report on the remote location file sets. It shows that more than 15 gigabytes of data have not been accessed in over 270 days. The user decides to create an archiving policy based on the information provided by the remote inventory job. 3. The user runs a remote archive job with the policy created in step 2 to archive unused files. This results in more disk space availability in the remote locations, less data to be backed up (the archived files no longer need to be backed up), and less time to run a backup job. 4. The user wants to run backups of the remote locations. Using a simple Web UI, s/he schedules a remote backup job to run on a regular basis. The backed up data from the remote locations is stored in a central location where it can be restored quickly, if required, using the Remote Restore tool. 5. It is assumed that this location backup data would then be backed up by any central backup software. such as BakBone NetVault which comes standard on GuardianOS-based Snap Servers. MANAGEMENT CONSOLE WAN Target Agents SNAP SERVER 18000 EDR MANAGEMENT CONSIOLKE Central Repository Figure 2: Sample Deployment Report File 1. The Administrator logs into the Management Console to create and configure jobs through this interface. 2. The Management Console communicates with the Snap EDR agents from which the inventory job will collect statistics. 3. The Management Console retrieves the information, logs the transfer activity, and allows the user to view these logs and reports.
3 The Snap EDR Agent is deployed on all of the machines involved in the inventory job. Taking inventory of data using Snap EDR is a straightforward process involving the activation of the Snap EDR agents on two or more host machines, configuring the parameters of the inventory job (e.g., source and target directories, date criteria etc.) and scheduling the inventory task. 3. System Setup and Software Installation The Snap EDR system requires users to install the Management Console that coordinates and logs the data transfer activities of the distributed agents. The Management Console should be installed on a Snap Server running GuardianOS. Users can install the Agent software on a GuardianOS-based Snap Server, Windows, or Unix/Linux systems involved in the data transfers. Installing and configuring the Snap EDR software involves the following procedures: Installing the Management Console software on a Snap Server Installing Snap EDR agent software on a Snap Server, Windows, and/or UNIX/Linux system Note that Snap EDR installs only on GuardianOS 3.2.019 or higher. Once the software is installed, users can login to the Management Console and configure and schedule data inventory jobs. 3.1. Installing the Snap EDR Management Console Software Note: Only one Management Console should be configured to handle all of the agents in the Snap EDR replication environment. Install the Management Console through the browser-based Administration Tool for the Snap Server. For detailed procedures on how to install the Management Console software, refer to the product documentation. 3.2. Installing Snap EDR Agents Snap EDR agents are responsible for the actual transfer of data. An agent is installed automatically on the Snap Server that is designated as the Management Console. The Snap EDR agent software must be installed on each remote node in the system that will be part of the Snap EDR data replication environment. Users can install a Snap EDR agent on any Snap Server, Windows, or Unix/Linux system that meets the following requirements: Item Description Operating Snap Server with GuardianOS v3.2.019 or higher System AIX 4.3, 5.1, 5.2 Solaris 5.7, 5.8 HP-UX 11.x, 11i Fedora Linux, RedHat Linux 7.2/7.3, Enterprise Linux 2.1 or 3.0 or SUSE LINUX 8.2 and 9.0 Windows 2000 (with SP2 or higher), Windows 2003, Windows Server Appliance Kit (SAK) 2.01, or Windows XP System Memory 128 MB or more Disk Space Windows systems: 5 MB or more UNIX systems: 5 MB or more Installation directory: 100 MB, 20 MB free in /tmp on all platforms Network 100 Mbps Ethernet or faster OR a WAN connection capable of Connection supporting the throughput required to meet the volume and window of time requirements For detailed procedures on installing the Snap EDR Agent software on a Snap Server, Windows, or UNIX/Linux system, refer to the product documentation. 4. Data Management Tool The Remote Data tool is primarily used to assist in determining file archiving policies. The tool allows the user to specify very flexible criteria to report on file systems. The criteria can be set to show when files were last accessed, last modified, files that meet certain size ranges, or other combinations. 4.1. Default Functionality The following are the features of the Remote Data tool: File system reports allowing very flexible reporting criteria Volume information reporting Host information reporting including CPU and physical memory Centralized control for viewing job status, controlling job execution, and managing hosts involved. Report results stored on a per-host, per-report date basis allowing consolidation of data for analysis Built-in email notification of report results Report on heterogeneous host platforms in a single job 5. Configuring an Job Once you have installed the Snap EDR software, you can login to the Management Console to configure the tool to schedule the type of inventory job you want to run. When you login to the Management Console, the following screen appears, displaying a menu on the left side that allows users to schedule a variety of jobs (Aggregate, Distribute, Backup, Replicate, and so on), install agents, change user preferences, run maintenance tasks, generate reports, and view transfer logs:
4 5.1.1. Files Generated The Remote Data tool will potentially generate two files per source host to store the inventory information. These two files are the main host inventory report and the host file inventory report. These files are generated on the source host where the information is gathered. Once the information gathering is complete, the files are transferred to the target host and stored in a userdefined location. Each time a Remote Data job runs, the results are stored in a sub-directory, based on the current date, beneath the user-defined directory. This permits a history to be kept of all the reports, as illustrated in the following: Figure 3: Management Console Snap EDR provides a straightforward interface to schedule an inventory job by clicking on the Remote Data menu item on the left side of the screen. The configuration and scheduling screen is similar to the following: Figure 5: Sample Reports Directory Users can expand the directories and click on the reports to view them. 5.2. Host Report File Each source host involved in the Remote Data job generates a host inventory report file. This file contains all of the information specified to be included in the report. The inventory report name for a file has the following format: <hostname>stmreport.html The host name portion identifies the report for a particular host. For example, the report file for host "se-src1.senet.com" is "sesrc1senetcomstmreport.html". Figure 4: Remote Data Job Creation Screen When creating a job, users can specify the agents on which the inventory job should run, the type of information on which to report (host name, size, date files were last accessed, etc.), date and size criteria, etc.. For detailed instructions on creating an inventory job, see the product documentation. 5.1. Remote Data Details This section describes details about the Remote Data tool. 5.3. Host File Contents The contents of the host inventory information file vary depending on the options selected to report when the job was created. The file will contain the following sections based on the options selected to report. 1. Host Information Section (this section appears if any of the host information to report options were selected) 2. Volume Scan Information (this section appears if the volume scan information option was selected) 3. File Archiving Report (this section appears if the scan file information option was selected)
5 5.3.1. Host Information Section The host information section of the report contains information such as the host name and operating system, as illustrated in the following diagram: Remote Data Report se-src1.senet.com - Thu Aug 5 16:26:26 2004 HOST INFORMATION - se-src1.senet.com Host Name se-src1 Operating System Microsoft Windows 2000 Server (Version 5.0.2195 Service Pack 4) Processor Physical Memory Figure 6: Remote Data Report Sample 5.3.2. Volume Scan Information The following figure provides an example of the volume scan information from a host inventory report file. In this case, all options were selected to report on, and no exclusions were specified. If the user had specified to report only volumes where free space was less than 20 percent, only volume E, F, G, I, and K would be displayed. VOLUME SCAN INFORMATION - se-src1.senet.com Figure 7: Volume Scan Information Report Sample 5.3.3. File Scan Information The file scan report section is included in the host inventory report if the user specified the Scan Files option as part of the host inventory information to report. The section header is File Archiving Report, since a primary use of this information is to create Remote Data Archiving policies. This section has two formats depending on the file scan criteria selected. All the user-defined search parameters used to generate the file report appear in the file archiving report section. Default Display Format x86 Family 15 Model Stepping 9 Genuine Intel 2666Mhz 523764 Kbytes Volume Name Percent Utilizes Size (Kbytes) Used (Kbytes) Free (Kbytes) C: 65% 10,482,412 6,894,852 3,587,560 E: 19% 90,943,932 18,066,820 72,877,112 F: 20% 96,205,220 20,113,804 76,091,416 G: 10% 1,280,540,672 128,188,416 1,152,352,256 H: 98% 10,008,460 9,867,388 141,072 I: 20% 90,935,900 18,971,244 71,964,656 J: 22% 90,935,900 20,038,480 70,897,420 K: 7% 90,935,900 6,944,876 83,991,024 If the user specifies only default file scan criteria (no date ranges, no size ranges, etc.), the file scan report generates a default report based on files that have not been accessed in the last 30, 60, and 90 days (based on source directories to search). This format will provide the total number of files that match the criteria (last accessed > 30, 60, 90 days) and the disk space used for each set of files. The default file scan report provides valuable information to use as an archiving policy. User-Defined Display Format The second format option for the file scan report section is the user-defined display format. In this display format, the user has determined the file scan criteria to use. As a result, only the number of files and total disk space to archive appear. All of the user-defined search parameters used to generate the file report appear as part of the file archiving report section. File Archiving Statistics Number of Files to Archive Figure 8: File Archiving Statistics Sample Report 5.4. File Report File Each source host involved in the remote data inventory job generates a file scan report file if the user has selected Scan Files AND has selected at least one file information option to display. This file contains all of the file information specified to be included in the report. The file scan report name for a file has the following format: <hostname>filereport.html The host name portion identifies the report for a particular host. For example, the report file for host se-src1.senet.com is sesrc1senetcomfilereport.html. A separate file is generated to contain the specific file information for easier management. The file scan report can be quite large (i.e., several MBs). Keeping a separate file scan report also makes importing the file inventory report into other applications (e.g., Excel spreadsheet) easier because the other report information does not have to be removed first. The following figure provides an example of a file inventory report file: Remote Data Report se-src2.senet.com - Fri Aug 5 12:27:04 2004 HOST INFORMATION - se-src2.senet.com Host Name se-src2 Operating System Microsoft Windows 2000 Server (Version 5.0.2195 Service Pack 4) Processor Physical Memory Total Disk Space to Archive x86 Family 15 Model 2 Stepping 9 Genuine Intel 2666Mhz 523764 Kbytes Disk Space for Files >30 Days Old Disk Space for Files >60 Days Old VOLUME SCAN INFORMATION - se-src2.senet.com Volume Name Percent Utilizes Size (Kbytes) Used (Kbytes) Free (Kbytes) C: 32% 10,482,412 3,424,008 7,058,404 E: 20% 90,935,900 18,971,244 71,964,656 F: 28% 95,418,032 26,858,967 68,559,056 Figure 9: Remote Data Sample Report Disk Space for Files >90 Days Old 526,980 16.00 Gbytes 16.00 Gbytes 16.00 Gbytes 10.00 Gbytes
6 6. Best Practices - This section describes the recommended configuration options and best practices to deploy the remote data inventory solution. 6.1. Access to System Information The remote data inventory solution queries system information from the hosts specified in the job. To ensure proper access to required system information, it is recommended that the source user for the job have administrative rights on the source hosts. This is not mandatory, but may be required to determine all required information. 6.2. Append Results to Email Notification The user has the option to append the results of a remote data inventory job to an email for notification. Care should be taken to avoid generating excessively large emails by reducing the number of source hosts per job. The results of each source host inventory report are appended to the email. Reducing the number of source hosts per job will reduce the size of the email generated. 6.3. File Scan Report vs. Aggregate File Information This section applies only if the scan files option is being used. The main host inventory report contains an aggregation of the file scan results. This data will contain the file scan criteria selected and the results in terms of number of files that meet the criteria and total disk size occupied by these files. A separate file scan inventory file is generated if any of the "file information to report" options are selected. common directories to file scan, e:\documentrepository. Create one job for host 1 and 3, specifying the source directory c:\appdata\userfiles, and another job for host 2, 4, and 5 specifying the source directories e:\documentrepository. 6.6. Debugging Users can set the job log detail level option when scheduling a job, to produce agent and job logs useful for debugging. For more information on how to do this, see the product documentation. 7. Monitoring Performance There are a number of ways a user can monitor the performance of a job, including the following: Viewing Job Statistics Viewing the Job Log Files 7.1. Viewing Job Statistics When a job that has run to completion is selected, the screen displays the job's statistics, including information such as the start and end time of the job and whether or not the job ran successfully. The Scheduled Job View screen also allows access to console functions where other tasks can be performed with the scheduled job. These tasks include manually running a job and viewing past log files. The following is a sample of the Job View screen: If only the aggregated results of a file scan are required and not all of the individual file details, then do not select any of the "file information to report" options. This saves processing time and disk space on the target directory for the inventory reports. 6.4. Source Directories Omitted This section applies only if the Scan Files option is being used. If using the Scan Files option, specify directories to scan using the source directories prompt. If the source directories prompt is omitted, the default files to scan are all root drives. This can be very time consuming and generate very large file scan report files. 6.5. Source Directories Specified This section applies only if the Scan Files option is being used. The source directories to search for files to scan should be consistent across all source hosts of a scheduled remote data inventory job. If the source hosts to run a file scan report have different source directories, the source hosts with the common directories should be grouped into the same job. For example, hosts 1 through 5 must be part of a remote data inventory job with file scan. Hosts 1 and 3 have common directories to file scan, c:\appdata\userfiles. Hosts 2, 4, and 5 have the Figure 10: Job Statistics Screen Exit codes appear in the Exit Code column in the list of job runs. An exit code indicates whether or not a job completed successfully, and if there was an error, the type of error that occurred. A 0 exit code indicates that the job completed successfully. For a description of what each non-zero exit code means, click more info beside the exit code in the Exit Code column of the job screen, or refer to the product documentation. 7.2. Viewing the Job Log Files Whenever users run a scheduled job, the Management Console generates a log file. If the job is clicked on, the job's display area shows log files for the most recent run and up to ten previous job runs. If there are more than ten job runs, the oldest job log is deleted to allow space for the latest job log.
7 Clicking on the word view beside any past job run displays a detailed log summary for that particular job run: Template names can be assigned to reports that are created. The template appears in the Report view. Report templates allow reports to be generated without having to change the initial parameters. Templates are especially useful if for generating the same type of report on a regular basis. Run, Edit, Remove or Schedule a report by clicking on the appropriate word beside the report template. Click Schedule to run reports daily, weekly or monthly at specific times. The following diagram is a sample of a Stats Summary Report: Figure 11: Job Log Screen This summary displays messages about the data transfer generated by the Management Console and the agents while the job is running. These messages vary in severity, from simple information messages (e.g., starting group: RemoteBackup (sequential)), to warnings (e.g., Preserve ownership may fail for target 'snap01' because the system types are different. ) to errors and debug messages. These messages are useful for evaluating the job run. Click the Filter button to change the type of log information displayed or to download the job log file to your local machine. Downloading the file allows you to load it into different programs (for searching, printing and so on). The log file is a plain text file. 8. Verifying Performance The Management Console creates statistical records for each completed job run and stores them in the rules database. Using the Management Console, users can generate a number of different types of reports to evaluate the operation of each job. Figure 12: Sample Stats Summary Report For detailed instructions on how to generate reports, see the product documentation. 9. Conclusion The Snap EDR data management tool is a valuable and easy-to-use tool to determine storage utilization. The tool is a comprehensive solution that can assist users in the in developing file archiving policies. 8.1. Report Types Using the Management Console, the following types of reports can be generated: Report Type Stats Summary Report Detail Report Custom Query Report Description Provides a summary and totals of the data from all jobs in the selected job group (data inventory jobs are in the RemoteData group), based on how the user chooses to group the data (by month, day, etc.). Provides a detailed record of the data from the selected job group (data inventory jobs are in the RemoteData group). Note that the number of records the report displays depends on the number of agents on which the job runs. The report displays a record for every agent on which the job runs. Provides a customized query of detail records from the job name within a selected job group (data inventory jobs are in the RemoteData group). Note that the number of records the report displays depends on the number of agents on which the job runs. The report displays a record for every agent on which the job runs.
8 Adaptec, Inc. 691 South Milpitas Boulevard Milpitas, California 95035 Tel: (408) 945-8600 Fax: (408) 262-2533 Literature Requests: US and Canada: 1 (800) 442-7274 or (408) 957-7274 World Wide Web: http://www.adaptec.com Pre-Sales Support: US and Canada: 1 (800) 442-7274 or (408) 957-7274 Pre-Sales Support: Europe: Tel: (32) 2-352-34-11 or Fax: (32) 2-352-34-00 Copyright 2005 Adaptec Inc. All rights reserved. Adaptec and the Adaptec logo are trademarks of Adaptec, Inc., which may be registered in some jurisdictions. All other trademarks used are owned by their respective owners. Information supplied by Adaptec Inc., is believed to be accurate and reliable at the time of printing, but Adaptec Inc., assumes no responsibility for any errors that may appear in this document. Adaptec, Inc., reserves the right, without notice, to make changes in product design or specifications. Information is subject to change without notice. P/N 666759-011 Printed in USA 02/05 3724_1.3