SwiftStack Object Storage

Transcription

1 Integrating NetBackup 8.1.x with SwiftStack Object Storage July 23,

2 Table of Contents Table of Contents 2 Introduction 4 SwiftStack Storage Connected to NetBackup 5 Netbackup 8.1 Support for SwiftStack 6 Installation Steps of Veritas NetBackup 8.1 on Linux 7 Configuration Steps of Veritas Netbackup Enable S3 API Support 9 Netbackup User Account and Bucket Creation 9 Configure Cloud Storage Server 10 Configure Disk Pools 15 Creating Storage Unit from a Disk Pool 19 Storage Unit Group 20 Backup and Restore 22 Create Backup Policy 22 Examples of Manual Backup and Restore 26 Tuning 31 Backup Concurrency 31 Media Servers 31 Master Servers 33 Migrating from existing storage pools to SwiftStack 34 Duplicating Backups 35 Increasing Duplicate Job Size 38 Conclusion 39 References 40 July 23,

3 Introduction For storage administrators looking for a more cost-efficient, reliable, scalable, and easier to use solutions for backup and recovery, object storage is an excellent destination target. Traditionally, backup applications needed disk storage to emulate tape or be locally mounted to the media server. Starting with version 8.0.1, Veritas NetBackup can now natively utilize object storage using cloud storage APIs (Swift and S3). This type of storage offers several unique advantages for as a backup target: Nearly unlimited scalability of capacity without reducing performance. No more running into hard capacity limitations of a storage silo, forcing you to recycle your backup data and/or manage it across many namespaces. Automatic disaster recovery protection by replicating data offsite. Cluster nodes can exist in multiple physical locations, so no need to worry about hauling tape media to the mountain anymore. Cost-efficient hardware infrastructure based on standard servers, Ethernet-based networking, and affordable disk drives rather than on more expensive proprietary solutions. This how-to guide will walk you through the specific steps you need to take to optimally configure Veritas NetBackup to utilize a SwiftStack object storage for backup and recovery. If you have questions when designing, testing, or deploying SwiftStack with NetBackup, please feel free to contact us. July 23,

4 SwiftStack Storage Connected to NetBackup Architecting cloud storage SwiftStack is easy, and reference hardware examples are available. In this case, SwiftStack will be used as a direct target for NetBackup Media servers. SwiftStack storage can be deployed alongside your existing backup targets or replace existing targets such as NAS, DAS, PBBAs, or even tape. SwiftStack clusters start with a minimum of three nodes to ensure data availability and durability. Relative to other storage targets, your overall backup performance can also improve with SwiftStack, as all nodes in the cluster can be directly addressed for both backups and restores. SwiftStack is a scale-out storage system, and all nodes share a common namespace. Having a common namespace eliminates the need for silos that comes with direct attached storage, SANs, or file servers that don t scale. Each node in a SwiftStack cluster is active, and performance scales linearly as more storage servers are added. SwiftStack software included a built-in load balancer to make scaling out easy, without the requirements of an external solution. While not addressed in this guide, SwiftStack clusters also have the ability to span multiple sites or regions. Unlike typical storage systems, nodes at other sites are active, share the same namespace, and handle cross-site replication of data automatically. This makes SwiftStack unlike any other solution on the market with its ability to scale out without boundaries and automatically protect your vital business data from a major disaster. July 23,

5 Netbackup 8.1 Support for SwiftStack Swiftstack Object Storage support is included in Veritas Netbackup version 8.1. The official Hardware Compatibility Lists (HCL) for NetBackup can be found using the following link: (Refer to Page 108 for SwiftStack support on Windows, Linux and other Unix OS). Currently deduplication (CloudCatalyst product) is supported only when using the S3 API connector. Except for deduplication, all other functionalities of NetBackup 8.1 work with the Swift API connector as well. July 23,

6 Installation Steps of Veritas NetBackup 8.1 on Linux Refer to the following document for detailed Installation instructions on Linux and Windows System: Type=pdf Listed below are some of the important pre install software and hardware configuration checklist: 1. Create a Linux VM for Netbackup 8.1, and install Netbackup Master Server, Media Server and Netbackup client on it. 2. Ensure that your VM is configured with the following minimum hardware configurations: a. CPU - 4 Processor Cores b. Memory - 16 GB c. Hard Disk GB (Data Disk) 3. Copy Netbackup tar file to /root directory 4. Install from tar file: tar xvf NetBackup_8.1_LinuxR_x86_64.tar 5. Setup user account nbwebsvc and group nbwebgrp 6. Set ulimit value. Minimally, Netbackup will require the number of file descriptors to be set to 8000 or greater. Set the following values in /etc/security/limits.conf file: * soft nofile * hard nofile Set the following semaphore values required for Netbackup 8.1 installation: NetBackup recommends the following values: SEMMSL SEMMNS SEMOPM SEMMNI Validate and change semaphore values to match with the recommended values above 8. Change directory to Netbackup_8.1_LinuxR_x86_64 and run the command./install Netbackup_8.1 a. Install License Key obtained from Veritas Partner Program b. Install Netbackup Master Server, Media Server and also Client on the same VM 9. Install Netbackup Administration Console on a Windows System to get a Webui interface. 10. From Windows Start Menu, launch Veritas Admin console: a. Start->All Programs->Veritas NetBackup->Netbackup 8.1 Administration Console b. Login from the Admin console to Netbackup Master Server. Login as root and use the same password for login to Master Server which is hosted a Linux VM July 23,

7 July 23,

8 Configuration Steps of Veritas Netbackup 8.1 Enable S3 API Support NetBackup communicates with SwiftStack using both Swift and S3 API. To enable S3 API support, if not already enabled, follow these steps: Step 1: In the SwiftStack Controller, navigate to Manage > Middleware. Step 2: Click on S3 API Support in the list. Step 3: Check the Enabled box and click Submit. Step 4: Click Deploy Config to Cluster to apply the changes. Netbackup User Account and Bucket Creation Create a Netbackup user account on SwiftStack controller - sec01.swiftstack.demo 1. Create a new user: Step 1: In the SwiftStack Controller, navigate to Manage > Users & Accounts. Step 2: In the Manage Swift Users page, click Create New User. July 23,

9 Step 3: Enter a username and password for this new account. a. username - netbackup81 b. password - swiftstack Step 4: The new user will now be displayed under Existing Users. Click on Deploy Changes. Step 5: Click Deploy Config to Cluster to apply the changes. 2. Create Buckets Create at least two buckets my-s3netbackup81, since Netbackup expects buckets to be pre-created prior to tests: a. my-netbackup8-1 bucket for testing with Swift API b. My-s3netbackup81 bucket for testing with S3 API Tested with both Swift and S3 API. However, the document covers the procedure and instructions to test with S3 API only since the steps are exactly the same (except for de-dup functionality which is applicable only for S3 API). (Note: NetBackup uses Bucket terminology which is synonymous with SwiftStack Container) Configure Cloud Storage Server 1. From the NetBackup Admin console, click on the top button NetBackup Master Server, and click on Configure Cloud Storage Server to follow the steps in the screen shots shown below: July 23,

10 July 23,

11 July 23,

12 In the screenshot above, Service host refers to SwiftStack controller. Create a Storage Server name SS-NB8-S3. Enable NetBackup Cloud Catalyst for deduplication, and create a local cache directory /home/nbdedup required for deduplication. Access key ID is the same as Netbackup user account, and Secret access key is S3 API key, which can be found by clicking the Show S3 API Key button. Now Click on Add Cloud Storage, and add the following to General Settings. Service Name and Storage Server host are the same as defined earlier. July 23,

13 In Region Settings, add the following: Note that Region name is Netbackup internal name (us-east) and can be any name. However, Location Constraint name should match with Region Name of SwiftStack Web Console. In this case it is us-east-1. July 23,

14 For Advanced Settings, disable SSL: Now in the main Cloud Storage Server setting, press Next, and Cloud Storage Server will be successfully created. Configure Disk Pools The next Step will be to click on Configure Disk Pool from NetBackup Admin Console under Master Server menu. July 23,

15 Click on Cloud Storage (PureDisk_swiftstack_rawd), and then click next. Select Storage Server SS-NB8-S3 and press next. July 23,

16 Now we should be able to see all the buckets created for Netbackup81 account. July 23,

17 Now click on bucket my-s3netbackup8-1 and we can create a Disk pool (named NBPool03). July 23,

18 Once Disk pool is successfully created, Create Storage Units. Creating Storage Unit from a Disk Pool On NetBackup Admin console, right click on Storage Unit under Netbackup Management button, and create a storage unit (backup-pool07). Choose Disk Type as Cloud Storage(PureDisk_swiftstack_rawd), and select Disk Pool NBPool03. Also, select Maximum Concurrent Job to 10. July 23,

19 Similarly, create another storage unit named backup-pool08. Storage Unit Group Now create a Storage Unit Group which allows pooling of Storage Units in order to increase the amount of storage available to a backup policy. 1. Expand NetBackup Management 2. Expand Storage July 23,

20 3. Right click Storage Unit Group 4. Click New Storage Unit Group In the above storage unit group, select round robin method to choose the least recently selected/used storage unit in the list. July 23,

21 Backup and Restore Create Backup Policy Right Click on Policy under Netbackup Management, and create a new Policy: Once the policy is named, press OK, and then fill in Attributes, Schedule, Clients and Backup Selection. Backup Attributes (Choose Policy Storage unit-group-s3dedup): July 23,

22 Select backup schedule: July 23,

23 We can select from one of the following backup policies: Full Backup, Differential Incremental Backup, Differential Cumulative Backup, User Backup or User Archive. The following is a typical example of a Full backup schedule (in this example, S3Policy name refers to full backup. We can rename it as Full also). The client settings for backup are shown in this screenshot: July 23,

24 Typical Examples of Selection of Backup files/folders are as follows: July 23,

25 Examples of Manual Backup and Restore In addition to scheduled backup policy, to perform a manual backup and restore, right-click on Backup Policy (NB_BU_PolicyS3), and activate manual backup. Look at the Activity Monitor screen, and check the status of the manual backup job recently initiated. Once the backup job is completed, the status will change from Active to Done. You can also review detailed status information and look at de-dup % and cache hit rate. This is a screenshot after a couple of backup operations (for /etc and /tmp directories) July 23,

26 The De-dup % will be initially lower if it is the first back up (in this case it was around ~ 65%) and then improved with subsequent backups. Also, De-dup % will depend on the type of data set and application profile. Once the manual backup is completed, we can then do a manual restore. Click on Backup, Archive and Restore Button and then click on Restore files: July 23,

27 Click on the Restore button, and choose the following restore path: /home/nbrestore8 (as an example shown in the next screenshot). Netbackup restore will automatically create the NBrestore8 directory as specified in restore path. July 23,

28 Now press Start Restore, and watch the Task Progress button in the previous screen. Wait until the last restore status changes to successful: July 23,

29 We can now look at files under etc and tmp directories created under /home/nbrestore8 directory, and they match with the files under /etc and /tmp. NBrestore8]# pwd /home/nbrestore8 NBrestore8]# ls etc tmp July 23,

30 Tuning Performance of backups is governed by the speed of the object storage system, the number and capability of media servers, network speed, and a number of other factors. The following best practices for NetBackup and SwiftStack will optimize performance. Backup Concurrency Object Storage excels with an increase of simultaneous backup streams. Where traditional filers drop dramatically in performance with each connection, object storage can scale linearly with proper equipment and load balancing (which is integrated into SwiftStack). A single backup job may proceed at 20MB/s, since NetBackup uses only a single connection per backup job. A second backup job will also progress at 20MB/s, but the aggregate for the two jobs is 40MB/s. In order to achieve minimal backup windows, many jobs should run concurrently. Fifty simultaneous jobs on a single media server can max a 10Gb network link. Just be sure to back off if primary storage is affecting application performance during backups. In order to allow a media server to run more than one concurrent jobs, please make the configuration changes in the next section. Media Servers In order for media servers to support multiple backup streams, the following changes must be configured; otherwise media servers will restrict throughput to a single backup by default. The following must be completed for each media server that intends to backup to a cloud storage unit. Expand NetBackup Management. Expand Host Properties. Expand Media Servers. Right click on the desired Media Server. Select Properties July 23,

31 2. Select Scalable Storage from the left panel. Increase the Total available bandwidth to the full network bandwidth of the media server. In our case with 10Gb networking, the value was adjusted from (default) to Increase the Maximum concurrent jobs from 1 to 10, the max should be 100. This will allow the media server to take full advantage of the concurrency of object storage. Tune this setting as needed to meet the ability of the media server to handle client backup streams. Click OK when finished. July 23,

32 Master Servers In order for media servers to support multiple backup streams from within a job, the following changes must be configured. This will allow an agent to backup multiple drives within a target machine at the same time thus reducing the time per agent and improving throughput. Expand NetBackup Management Expand Host Properties Click on Master Servers Right Click the master server, choose Properties Select Global Attributes from the left panel Increase Maximum jobs per client. Start with 5 and increase up to a max of 10 July 23,

33 Migrating from existing storage pools to SwiftStack Migrating data from existing existing Storage Units to new Cloud based Storage Units is a simple process. This can be useful in a number of scenarios: Aging data from expensive disk pools to economical Cloud Storage Duplicating data from Cloud Storage to Tape for DR compliance Restoring data from Offsite tape to Cloud Storage All of the above scenarios will follow the same process outlined below: July 23,

34 Duplicating Backups The following assumes that the backup is currently loaded into the NetBackup Catalog. If this is not the case, please follow the article below to import an expired backup back into the catalog. Import Expired Media to Catalog Expand NetBackup Management Click on Catalog 1. Select Action Duplicate Select Disk Type - Cloud Storage (PureDisk_swiftstack) under the current location of the backups under Media Select the Date Range Select Search Now Results will populate below. 2. Highlight the desired backups to duplicate in the above screen Right click and select Duplicate July 23,

35 3. If only one copy of the data is desired, leave Copies at 1. Check the checkbox of Copy 1 if it is to be made the new primary copy. Choose the cloud Storage unit to store the backup in. If desired, change the Retention time in backup policy Preserve multiplexing is not required. It is for tape to tape transfers. July 23,

36 4. Confirm your request to proceed. Click OK 5. The job can be monitored in the Results Tab or the Activity Monitor. Output of the result is as follows: July 23,

37 6. Once the job is complete, the original backup can age out according to the original retention policy or can be expired immediately. To expire immediately: Select the backups to Expire... Right Click the backups. Click on Expire. Select Yes on the next screen. Increasing Duplicate Job Size By default NetBackup limits the size of a duplication job to 100GB. This may be too small for most duplication jobs involving LTO tape. To change this valve perform the following: 1. Expand NetBackup Management Expand Host Properties Click on Master Servers Right Click the master server, choose Properties 2. In the left panel, select SLP Parameters. Change the Maximum size per duplication job from 100GB to the required value for the environment July 23,

38 Conclusion Congratulations! Your Veritas NetBackup environment is now configured to backup data to SwiftStack, allowing you to protect more data with deduplication feature with full compliance of your backup and recovery policies and maintaining high degree of scalability and fully automated disaster recovery functionality. If you have any questions about this solution or would like any assistance with design and implementation, please feel free to contact us. We re here to help. Phone - (415) contact@swiftstack.com Chat - Just go to swiftstack.com and look for the chat pop-up in the bottom right July 23,

39 References 1. NetBackup Installation Instructions: =pdf 2. NetBackup Administration Guide Volume I ex.html 3. NetBackup Administration Guide Volume II ex.html 4. NetBackup Cloud Administration Guide ex.html 5. NetBackup Deduplication Guide ex.html July 23,