DXi-Series and Veritas NetBackup 6.5 from Symantec BPG00001 BPG00001
Table of Contents DXi-Series Best Practices for NetBackup... 1 Scope... 1 Installation Recommendations... 2 Virtual Tape Library Presentation: Fibre Channel/iSCSI... 2 Robot/Media Changer Device Serialization and Recommendations... 2 Links to Mapping Files and Supported Device by NetBackup... 2 Supported Hardware Compatibility List... 2 Virtual Tape Drive Device Description and Recommendation... 3 DXi Tape Drive Emulations and Recommend Drivers... 3 Number of Concurrent Tape Drives in Use... 3 Quantity and Capacity of Tapes... 4 Suggested Tape Cartridge Capacity... 4 Tape Drive LUN Mapping... 5 NAS as a Target... 5 Replication Recommendations... 5 Integration Recommendations... 6 VTL Fibre Channel/iSCSI...6 Performance Tuning... 6 Eliminating Interference from the Host System...6 Fragment Size for the Storage Unit...6 Performance Gains with Buffer Settings in NetBackup...7 Network Buffering For Windows Servers Overview...7 NAS as a Target... 10 Network-Attached Storage Settings and Tuning... 10 Testing the Source Data to Isolate Bottlenecks... 10 Utilization Recommendations... 11 De-duplication Limitations...11 Good Candidates for Data De-Duplication... 11 Not So Good Candidates for Data De-Duplication... 11 Space Reclamation... 11 Recommended Handling of Expired Media... 12 Additional Information... 12 The information provided in this document by Quantum is for customer convenience and is not warranted or supported by Quantum. Quantum expects users to customize installation of third-party software for use to fulfill a customer driven requirement. However, Quantum is not responsible for the usability of third-party software after installation. This information is subject to change without notice. Quantum Corporation BPG00001 Page i
DXi-Series Best Practices for NetBackup Quantum Best Practices guide for DXi-Series and Veritas NetBackup by Symantec is provided to assist system administrators and backup administrators to get the most out of their investment. Quantum s global field teams and development engineering teams contributed their expertise in creating these recommendations. The Quantum Best Practices Guides routinely are updated to include the latest enhancements, check with your Quantum sales team for the latest version. Scope The intended audience for this guide is our customers and field sales teams as guidance to enhance the installation, integration, and utilization of the Quantum DXi-Series. The guide provides the users with guidance and best practices for the utilization of the Quantum DXi-Series and Veritas NetBackup. This guide provides recommendations and considerations impacting installation, integration, and utilization of the DXi-Series products specifically for Veritas NetBackup by Symantec environments. Quantum Corporation BPG00001 Page 1
Installation Recommendations Virtual Tape Library Presentation: Fibre Channel/iSCSI Robot/Media Changer Device Serialization and Recommendations Quantum is a leading manufacturer of automation devices and extends this knowledge and experience to the Virtual Tape Library (VTL) disk-based solutions. One of the key components to ensure that SANconnected physical and virtual tape libraries are seen properly by backup servers is serialization. Serialization provides a unique identifier for each device in a physical or virtual tape library to automate device association from multiple backup servers. These identifiers, returned by the VTL devices are separate from the element address that defines the position of devices in the library. The element address used by the library s robot or medium changer to manage the tape drives. Serialization allows the servers running the data protection application (the media servers) to coordinate tape drive configuration by aligning the device serial number with the device s element address. This enables the data protection application to automatically discover the devices in the library during the device discovery and configuration phase (e.g., the NetBackup Device Configuration Wizard ), reducing the potential for improper configuration. This feature enables Symantec NetBackup device discovery to align these two addresses. If the Device Configuration Manager does not serialize the devices listed, do not commit the changes, and check the VTL online state. The DXi VTL partition must be online for this to function properly. The Quantum recommended device identification for each DXi system is the native mode (e.g., use the DXi3500 inquiry response string as the identification on a DXi3500, and use the DXi7500 inquiry response string on a DXi7500). This allows product identification for the service teams at both Symantec and Quantum. When using the native device mode, Windows environments will display the device in the device manager as an unknown media changer. This is normal and not an error or problem to Net- Backup. If the customer environment has requirements for a specific changer device for compatibility, the Quantum DXi products support emulation of many popular devices to meet those requirements. Please note the M2500 emulation (available on all DXi systems) is not a supported configuration for Net- Backup6.5 at this time. Quantum recommends installing NetBackup s latest device mapping file to support properly identification of devices in NetBackup. Links to Mapping Files and Supported Device by NetBackup Windows platforms: http://seer.entsupport.symantec.com/docs/302599.htm UNIX platforms: http://seer.entsupport.symantec.com/docs/302598.htm Supported Hardware Compatibility List NetBackup enterprise server hardware compatibility list: http://www.symantec.com/business/support/compatibility.jsp?language=english&view=comp&pid=15143 Virtual Tape Library operation can be tested with the robtest utility that is included with NetBackup. Details of the robtest utility can be downloaded form the NetBackup support site: http://seer.entsupport.symantec.com/docs/274063.htm Quantum Corporation BPG00001 Page 2
Virtual Tape Drive Device Description and Recommendation Quantum DXi products support multiple tape drive emulations and allow user-definable capacities of tape cartridges to support the drive type. The drive type selection does not internally impose any throttling of the ingest rate. The host system uses tape device drivers to communicate with the virtual tape drives and the NetBackup application. The qualified tape drive emulation published on the hardware compatibility list is the Quantum DLT7000 emulation as this drive has the widest support across platforms. Quantum recommends using the Symantec tape drive drivers for backup hosts running the Windows 32- bits operating systems. For backup hosts running the 64-bit versions of the Windows operating system, the OEM tape device drivers are recommended. The complete tape drive driver recommendations from Symantec can be located in their hardware compatibility list (see above). Quantum recommends against using the Windows update function to get the latest tape drive drivers, as it is possible that Windows Update can find unexpected driver matches for a tape drive. These unexpected matches often are not the same as supplied by the tape drive manufacturer and may introduce unexpected incompatibilities. This can result in a change to the device in the Windows Device Manager. DXi Tape Drive Emulations and Recommend Drivers All drive types are tested by Quantum and any limitations are listed below: DLT7000: Use the Symantec driver or OEM driver SDLT 600 and DLT-S4: Use the Symantec driver or OEM driver HP LTO drives: Use the Symantec driver or OEM driver IBM LTO drives: IBM Driver 6.1.5.4 with DPO function for 64-bit systems Please note: Quantum does not support mixing drive types in a single partition. Number of Concurrent Tape Drives in Use Each DXi model has a maximum number of virtual tape drives that can be configured. Each DXi model has a maximum aggregate throughput rate that will be divided relatively equally between the virtual tape drives in use. That does not prohibit a single tape drive from utilizing all available bandwidth. The media server typically determines individual tape drive performance. It is not a good idea to configure the maximum number of virtual tape drives and to perform I/O through all of them concurrently. Better performance can be achieved by using a subset of those virtual tape drives at the same time. Quantum expects the customer configuration to distribute those virtual tape drives among multiple media servers to simplify initial installation by providing dedicated resources to each media server. Quantum recommends that backups be staggered such that only a subset of drives are in use at one time. During a backup, the data transfer rate is primarily controlled by the media server because the DXi system does not restrict the ingest data rate. This creates the opportunity for one or more media servers to burst data at a higher rate leaving the remaining virtual tape drives less bandwidth. Conversely, it supports the coexistence of fast data streams with slow streams for the maximum utilization of the available bandwidth. Note: Increasing the number of concurrently active virtual tape drives does not increase the aggregate DXi bandwidth and could result in a failed backup job due to a timeout from a bandwidth-starved operation. The recommended maximum number of concurrently active virtual tape drives for the respective system bandwidth is listed in the table below. Quantum Corporation BPG00001 Page 3
DXi Model Maximum Virtual Tape Drives Maximum Aggregate Bandwidth Recommended Maximum Number of Concurrently-Active Virtual Tape Drives DXi3500 32 290GB/Hr or 80MB/sec Up to 8 concurrently-active virtual tape drives * DXi5500 64 800GB/Hr or 222MB/sec Up to 22 concurrently-active virtual tape drives * DXi7500 Single Node 80 4TB/Hr or 1,111MB/sec Up to 55 concurrently-active virtual tape drives ** DXi7500 Dual Node 160 8TB/Hr of 2,222MB/sec Up to 110 concurrently-active virtual tape drives ** * Based on a user wanting no less than 10MB/sec per virtual tape drive. ** Based on a user wanting no less than20 MB/sec per virtual tape drive. Note: iscsi with a lower maximum transfer rate should stay below the SCSI2 limit of eight LUNs maximum and concurrent drives should be set to four based on a GbE interface connection and ½ the 2GB Fibre Channel suggested limits. It is also a good practice to keep the iscsi traffic separate from the production system network to prevent contention with end users for bandwidth. Quantum iscsi devices for Solaris initiator require the Qlogic TOE HBA. Quantity and Capacity of Tapes Quantum suggests not oversubscribing the device capacity when initially creating tape cartridges and recommends that the DXi installation team show the customer how to create additional virtual tape cartridges to be added as needed. This recommendation is to assist the user with managing the available space on the DXi device. The initial number of tapes to create should be based on the sizing exercise performed by the Quantum System Engineer. The DXi system can send alerts when the unit detects the user-selectable capacity point. Quantum recommends this feature be configured and utilized to alert the user before unknowingly arriving at a low space condition. When the backup environment requires additional virtual tapes cartridges and there is sufficient space available, the user can add additional virtual tape cartridges to a partition and run an inventory to update the NetBackup database. Suggested Tape Cartridge Capacity Space on tape cartridges cannot be reused until after all backup data on that cartridge has expired. The greater the capacity of a cartridge, the longer it will typically take for all data on that cartridge to expire. Expired data continues to take up space on the virtual tape cartridge as well as in the DXi. A lesser tape cartridge capacity is more desirable so that tapes are returned the scratch pool for reuse sooner. There is virtually no relationship between the configured capacity of a virtual tape cartridge and the tape drive emulation that has been configured for the partition: Any backup/restore operation will span the number of tapes required, ignoring the configured capacity. Any vaulting/duplicating operation performed by the backup application will ignore the virtual capacity when writing to another cartridge, whether virtual or physical. DXi3500 and DXi5500 allow you to specify a maximum capacity up to the available capacity of the DXi; the minimum is 5GB. Be aware that capacity utilization for these models is tracked in UNCOMPRESSED GB, although the data is stored compressed. That is, 100GB of data that is 2:1 compressible will report that it occupies 100GB of virtual tape cartridge space. If it is new unique data, it will be compressed 2:1 before being written to the de-duplication data pool. If it is not new unique data, then a pointer will be substituted for that data and the pointer will point to the original copy of that data in the de-duplication pool. DXi7500 will limit you to the maximum capacity permitted by the tape drive emulation; the minimum is 5GB. Quantum Corporation BPG00001 Page 4
Be aware that capacity utilization for DXi7500 is tracked in COMPRESSED GB and the data is stored compressed. That is, 100GB of data that is 2:1 compressible will report that it occupies 50GB of virtual tape cartridge space. Quantum s general guidance is to specify a smaller virtual tape cartridge capacity such as 100GB, for the reasons mentioned above. Tape Drive LUN Mapping Quantum recommends mapping the device starting with LUN 0 on each port and not skipping any LUNs. Quantum recommends a best practice of zoning the VTL devices and the NetBackup media servers to prevent other servers from taking control of the VTL resources. Additionally, we recommended using the HBA driver to bind the devices to a specific address. This helps keep devices in the same order after a reboot. Quantum also recommends setting the WWNN = WWPN for DXi3500 and DXi5500 systems. This allows for binding on the HBA to use either WWNN or WWPN. Quantum DXi-Series VTL devices support reserve and release to accommodate the SSO NetBackup option. The option allows devices to be shared between media servers. The advantage is having a pool of drives available to each media server. Other SAN architectures assign drives to each media server and eliminate the shared function. For both conditions, it is a good practice to keep the NetBackup media server separate from the production server to eliminate downtime from maintenance. This requires the media servers to have fast network connection to the source data. NAS as a Target The Quantum Network-Attached Storage (NAS) appliance is intended to act as a target for backup applications. This includes Network-Attached Storage or shares. NetBackup can use the NAS shares as Disk Storage Units (DSUs). Note: The NAS shares created on the DXi system are not recommended for general-purpose storage. Quantum DXi systems support both CiFS (Windows-based) and NFS shares. Each system can support multiple NAS shares with a maximum of 128 shares. Quantum recommends users only creating the required number of shares for each media server. DXi systems can support concurrent NFS and CiFS shares and can support iscsi and Fibre Channel VTLs concurrently with those NFS and CiFS shares. When using NAS shares on the DXi systems, Quantum recommends creating at least one share for each media server to use. Media servers should not share the NAS shares during normal backup operations. Root access to a NFS share is not allowed and the access rights will be changed to nfsnobody as a security precaution. This does not impact the access to the share from the NetBackup application. Replication Recommendations For first-time replication setup, it is important to manually replicate the name space once the target system is configured and is online. This facilitates the first replication following the first backup to that share/partition. The replication is only available to VTL and NAS shares with de-duplication enabled. The DXi supports 128-bit AES encryption for replication. Data is only encrypted while in transit between replication source and replication target. Data is unencrypted upon arrival at the replication target. Encryption may affect replication performance. It should be disabled if the customer s WAN is already secured. Quantum Corporation BPG00001 Page 5
Integration Recommendations VTL Fibre Channel/iSCSI Performance Tuning Combining our in-house testing, partner information sharing and customer feedback, we have found enhanced performance for NetBackup environments by using the following tuning parameters: Eliminating Interference from the Host System According to Microsoft, a conflict in Windows Server 2003 causes a Test Unit Ready (TUR) request issue on SCSI-attached and fiber-attached devices. When this issue occurs, an overflow of TUR requests causes the storage unit not to respond or to respond slowly to SCSI commands. In a SAN environment, any Windows Server 2003-based computer that is zoned to detect the Tape Backup Unit hardware can send TUR requests. The cause and workaround are documented in the Microsoft knowledge base article number 842411. Microsoft support link: (http://support.microsoft.com/kb/842411/en-us). Fragment Size for the Storage Unit Based on customer feedback and performance tuning, changing the fragment size for the storage unit in NetBackup from the default to 2000 will balance backup and restore operations. The fragment size indicates the size for a single fragment before a file mark is inserted. When restoring a file, the operation will have to search the full fragment. Using the smaller size allows the operation to search a limited area of the backup. Figure 1. Fragment Size Change Quantum Corporation BPG00001 Page 6
Performance Gains with Buffer Settings in NetBackup There are three main settings in NetBackup that have been shown to have a major performance impact for backups. These are the size of the data buffer, the number of the data buffers, and the size of the network data buffers. SIZE_DATA_BUFFERS After changing this value, it is important to test both backups and restores, as sometimes data can be written at the modified size, but potentially cannot be read at the modified size. Our lab environment testing and field experience has been positive with the recommended value of 262,144 block size. To change the SIZE_DATA_BUFFERS setting: 1. Create a file called SIZE_DATA_BUFFERS in the <INSTALL_PATH>\\NetBackup\\db\config. Note: The file name is case sensitive and must have no extension. 2. Add the value 262144 in this file. Note: If the file is not present, the default for the SIZE_DATA_BUFFER is 65,536. Windows-specific guidance: To use 256K block size, Windows Server 2003 requires SP2 to be installed; otherwise, the maximum allowed is 64K. After this file is created you do not have to cycle any services. The change will be done on the next backup job started. In combination with the change in block size, the number of buffers used will impact overall system performance. The default setting for the number of buffers used is 16. Increasing the number of buffers allows more data to be buffered before being sent to the device. Our lab testing has yielded better results using 32 for the number of buffers. NUMBER_DATA_BUFFERS To change the NUMBER_DATA_BUFFERS setting: 1. Create the <INSTALL_PATH\NetBackup\db\config\NUMBER_DATA_BUFFERS file. Note: The file name is case sensitive and must have no extension 2. Add the value 32 in this file Note: If the file is not present, the default of 16 will be used. After this file is created, you do not have to cycle any services. The change will be done on the next backup job started. Network Buffering For Windows Servers Overview When a backup is initiated, the client packages data of the amount specified by the Buffer_size value, and transfers the information to the media server, which, in turn, buffers that data in the NET_BUFFER_SZ. When the NET_BUFFER_SZ is full, it transfers data to the array of space created by Quantum Corporation BPG00001 Page 7
a combination of NUMBER_DATA_BUFFERS and SIZE_DATA_BUFFERS. As soon as at least one of the SIZE_DATA_BUFFERS is full, the information is written to the tape drive. NET_BUFFER_SZ Note the NET_BUFFER_SZ value is the size of the buffer on the media server that receives data from the client server. To change the Net_Buffer_SZ: 1. Create the NET_BUFFER_SZ file in the <INSTALL_PATH>\NetBackup directory. Note: The file name is case sensitive and must have no extension. The value in the file needs to be a multiple of 1024. If the file is not present, the default value of 262144 (or 256K) is used. 2. Add the appropriate value to this file. Our lab testing has yielded improved performance by matching the Buffer_Size on the clients to NET_BUFFER_SZ on the media server. Buffer_size Please note the value of the Buffer_size does not show up in the bptm log file, but can be viewed and set from the master server or by modifying a registry value. The default value for the Buffer_size setting is 32K. Changing the client server, the buffer size (Buffer_size) can be modified from the application user interface. Changing the Client Buffer_size (called communications buffer): From the NetBackup Administration console, Host Properties Clients, double-click the client you are configuring. Expand the options under Windows Client select Client Settings. Change the Communications buffer to the desired size in KB. Quantum Corporation BPG00001 Page 8
Figure 2. Communication Buffer Size For additional information on the settings our partner site link: http://support.veritas.com/docs/244652 Quantum Corporation BPG00001 Page 9
NAS as a Target Quantum DXi-Series products support NFS and CIFS protocols to act as a target for backups. In Windows Active Directory environments, the share acts as the target for the NetBackup. The share is not intended as primary storage or drag-and-drop storage. A best practice is to create a new account and workgroup as opposed to joining the domain to limit the access and prevent accidental file deletion by another user. Quantum recommends users not reconfigure or delete NAS shares while data is being written. There is no mechanism to detect the I/O and provide a warning to the user. Network-Attached Storage Settings and Tuning If you experience slow performance on disk storage units and disk staging storage units, implement the following recommendations: 1. Increase the data buffers used by Symantec NetBackup disk manager process on the master and media servers. The Default value is set to 262144 (256K) 2. Create a file <install path>/usr/openv/netbackup/db/config/size_data_buffers_disk Add the value 1048576 to the file content. Note: The file name is case sensitive 3. Create a file <install path>/netbackup/db/config/number_data_buffers_disk Add the value of 16 to the file. This can be increased based on available memory Testing the Source Data to Isolate Bottlenecks The following commands will show if bpbkar can read data from a hard disk. An even better use of this command is for performance troubleshooting, because it is never possible to write to tape any faster than the disk subsystem can produce it.the command will show how fast bpbkar can read data from the hard disk. Note on command syntax: The "<path-to-read>" is the path to the directories to back up and the "> /dev/null" is where the files list will be sent along with the report regarding the speed of reading the data from the disk. On UNIX systems:./netbackup/bin/bpbkar -nocont <absolute-path-to-read> > /dev/null Example:./netbackup/bin/bpbkar -nocont /home > /dev/null On Windows systems:.\netbackup\bin\bpbkar32 -nocont <absolute-path-to-read> > NUL 2>temp.f Example:.\netbackup\bin\bpbkar32 -nocont C:\ > NUL 2>temp.f The starting point for reading data for the path in the Windows example is on C:\ and it will continue to read down all subdirectories.so C:\winnt is also valid, for example. Quantum Corporation BPG00001 Page 10
Utilization Recommendations De-duplication Limitations De-duplication results can be negatively impacted by compression, encryption and, multiplexing. The use of multiplexing was intended for slow source data and a minimum transfer required by physical tape drives. Multiplexing backup streams was intended to provide a more efficient use of a limited number of physical tape drives. Since the virtual tape drives in the DXi systems are not susceptible to performance losses from slow data transfer rates, the number of virtual tape drives can easily be increased in quantity without any time penalty for re-positioning. It is not necessary to use multiplexing with the DXi systems. Additionally, the multiplexing adds additional header information to the data lowering the deduplication ratio. Compression and/or encryption of data before it reaches the DXi may also cause a similar reduction in de-duplication results and can also cause much higher CPU loading on the media server. Quantum recommends that multiplexing, inline compression by the backup application, and inline encryption by either the backup application or an inline appliance be disabled in order to achieve maximum deduplication of the data. Good Candidates for Data De-Duplication VMware, large databases (note exceptions below), PowerPoint presentations, Word documents, Excel spreadsheets, SQL, Oracle (note exceptions below), Exchange database, source code Not So Good Candidates for Data De-Duplication In-line compressed data, SQL with LiteSpeed (in-line compression), inline multiplexed data, Oracle with multi-channel RMAN (in-line multiplex), and in-line encrypted data For long-term archiving, we recommend vaulting the data to a physical tape device. Space Reclamation Space management involves two processes: data reconciliation and data reclamation. Data reconciliation is used to create a list of what can be removed. It runs automatically every twelve hours at noon and midnight, unless data reclamation is running. Data reclamation is the process of deleting the data on the data reconciliation list. It can be scheduled or run manually. There is significant overhead associated with this process and, therefore, it should not be run during periods of high appliance use. Replication, reclamation, and backup stream ingest all consume system resources and, therefore, should not all be done at the same time. Quantum recommends scheduling daily reconciliation and reclamation to manage the available space. The scheduled time should be configured to start the data reclamation process after the daily backups are complete. The default schedule is weekly and the default time for the data reclamation is set to 12:00AM Sunday. These parameters are user configurable and should be configured for the customer backup window. Quantum Corporation BPG00001 Page 11
Recommended Handling of Expired Media When a tape is expired by NetBackup, there is no direct communication of the event to the DXi-series. The result is a tape may be displayed as empty or SCRATCH to the NetBackup graphical Interface, but will show the same tape on the DXi-series GUI as containing data. This indicates the data on the expired tape is still using space on the DX-Series. To reclaim this space, we recommend using the bplabel utility. This will tell NetBackup to write a new label on the tape at the beginning of tape. This new label is a data block written to the virtual tape cartridge. The DXi-Series VTL will act similar to a physical tape and the data after the label becomes no longer accessible. The space reclamation can be initiated by the scheduled time or started manually from the DXi-series GUI. Additional information about the reclamation process can be found in the DXi-Series user manual. Note: Running the bplabel on a tape will make the contents inaccessible. Make sure the tape is expired before labeling the cartridge. Additional Information Quantum web site: http://www.quantum.com Symantec web site: http://www.symantec.com Microsoft web site: http://www.microsoft.com Guardian web site: http://www.quantum.com/serviceandsupport/services/guardianinformation/index.aspx StorageCare Vision web site: http://www.quantum.com/products/software/storagecarevision/index.aspx Quantum Service web site: http://www.quantum.com/serviceandsupport/index.aspx Call Center Americas. To contact our world-class support representatives, please refer to the information below: Telephone: 800-284-5101 (toll-free), 949-725-2100 (local/long distance) Hours of operation (subject to change without notice): 7 days a week, 24 hours a day with valid contract All other contracts can contact Quantum during normal business days from 5AM to 5PM US Pacific Time. View our Service Level Objective: http://www.quantum.com/serviceandsupport/servicelevelagreement/index.aspx Quantum Corporation BPG00001 Page 12