Veritas HyperScale 2.0 for Containers Administrator's Guide. Docker, Kubernetes, Red Hat OpenShift

Transcription

1 Veritas HyperScale 2.0 for Containers Administrator's Guide Docker, Kubernetes, Red Hat OpenShift February 2018

2 Veritas HyperScale for Containers Administrator's Guide Last updated: Document version: 2.0 Rev 0 Legal Notice Copyright 2018 Veritas Technologies LLC. All rights reserved. Veritas, the Veritas Logo, Veritas InfoScale, and NetBackup are trademarks or registered trademarks of Veritas Technologies LLC or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. This product may contain third-party software for which Veritas is required to provide attribution to the third party ( Third-Party Programs ). Some of the Third-Party Programs are available under open source or free software licenses. The License Agreement accompanying the Software does not alter any rights or obligations you may have under those open source or free software licenses. Refer to the third-party legal notices document accompanying this Veritas product or available at: The product described in this document is distributed under licenses restricting its use, copying, distribution, and decompilation/reverse engineering. No part of this document may be reproduced in any form by any means without prior written authorization of Veritas Technologies LLC and its licensors, if any. THE DOCUMENTATION IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID. VERITAS TECHNOLOGIES LLC SHALL NOT BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES IN CONNECTION WITH THE FURNISHING, PERFORMANCE, OR USE OF THIS DOCUMENTATION. THE INFORMATION CONTAINED IN THIS DOCUMENTATION IS SUBJECT TO CHANGE WITHOUT NOTICE. The Licensed Software and Documentation are deemed to be commercial computer software as defined in FAR and subject to restricted rights as defined in FAR Section "Commercial Computer Software - Restricted Rights" and DFARS , et seq. "Commercial Computer Software and Commercial Computer Software Documentation," as applicable, and any successor regulations, whether delivered by Veritas as on premises or hosted services. Any use, modification, reproduction release, performance, display or disclosure of the Licensed Software and Documentation by the U.S. Government shall be solely in accordance with the terms of this Agreement. Veritas Technologies LLC

3 500 E Middlefield Road Mountain View, CA Technical Support Technical Support maintains support centers globally. All support services will be delivered in accordance with your support agreement and the then-current enterprise technical support policies. For information about our support offerings and how to contact Technical Support, visit our website: You can manage your Veritas account information at the following URL: If you have questions regarding an existing support agreement, please the support agreement administration team for your region as follows: Worldwide (except Japan) Japan CustomerCare@veritas.com CustomerCare_Japan@veritas.com Documentation Make sure that you have the current version of the documentation. Each document displays the date of the last update on page 2. The document version appears on page 2 of each guide. The latest documentation is available on the Veritas website: Documentation feedback Your feedback is important to us. Suggest improvements or report errors or omissions to the documentation. Include the document title, document version, chapter title, and section title of the text on which you are reporting. Send feedback to: doc.feedback@veritas.com You can also see documentation information or ask a question on the Veritas community site:

4 Contents Chapter 1 Introducing Veritas HyperScale for Containers... 7 About Veritas HyperScale for Containers... 7 Reliable and efficient storage performance... 7 Read cache support... 8 Chapter 2 Displaying information about your environment... 9 About the HyperScale management console... 9 Displaying an overview of your environment Displaying information about the compute plane and data plane Displaying information about a container Displaying information about a volume Chapter 3 Working with volumes Creating a volume Creating a volume from a snapshot Deleting a volume Configuring Kubernetes/OpenShift pods to use HyperScale storage Chapter 4 Working with volume groups About volume groups Creating a volume group Viewing volume group details Deleting a volume group Creating a volume group snapshot Creating a volume group from a volume group snapshot Deleting a volume group snapshot Volume group limitations... 34

5 Contents 5 Chapter 5 Working with data nodes Adding a data node Removing a data node Chapter 6 Compute node and data node resiliency Compute node resiliency Data node resiliency Chapter 7 Ensuring quality of service with policies About storage policies Using storage policies Adding a storage policy Deleting a storage policy Chapter 8 Protecting HyperScale volumes with NetBackup About integrating NetBackup into your HyperScale environment NetBackup Media Server requirements NetBackup integration requirements HyperScale integration requirements Registering NetBackup with HyperScale Configuring a default NetBackup policy for HyperScale Creating storage policies in HyperScale using the configured backup policies Backing up volumes by assigning storage policies Restoring backups of a volume Chapter 9 Managing access to HyperScale using role-based access control About role-based access control Managing domains Adding authentication providers Deleting domains Managing roles Creating roles Editing roles Deleting roles Managing users Adding users Editing users... 69

6 Contents 6 Deleting users Managing groups Creating groups Editing groups Deleting groups Chapter 10 Working with HyperScale using Docker commands Creating a volume Displaying volume details Starting a container Chapter 11 Troubleshooting About HyperScale for Containers logs About HyperScale services Logging levels for HyperScale services Changing the logging level for a HyperScale service Viewing the logging level for a HyperScale service Modifying the systemd journal configuration Troubleshooting general issues Volume does not get deleted even after the container is removed HyperScale UI shows the usable storage as zero or lesser than what is available Appendix A Reference HyperScale alerts and messages Index... 88

7 Chapter 1 Introducing Veritas HyperScale for Containers This chapter includes the following topics: About Veritas HyperScale for Containers Reliable and efficient storage performance Read cache support About Veritas HyperScale for Containers Veritas HyperScale for Containers is a software-defined storage management solution for container-based architectures. HyperScale leverages direct attached storage (DAS) and provides enterprise-strength storage services for robust, production-scale deployments while meeting your required performance and data protection needs. HyperScale's architecture is designed for optimal application performance while providing the required resiliency and data protection necessary for enterprise applications. Reliable and efficient storage performance Although many container workloads do not require persistent storage, certain workloads, especially containerized enterprise database workloads do. Without persistent storage, applications cannot reliably maintain a consistent state across ephemeral containers instances.

8 Introducing Veritas HyperScale for Containers Read cache support 8 Read cache support The deployment of multiple workloads in the same cluster contending for the same resources can lead to noisy neighbor effects, adversely impacting application performance. Further, HyperScale's architecture employs separate planes for compute (where the workloads are deployed) and data (for data protection). This separation insulates the compute plane from the side-effects of resource-intensive data protection tasks. It also helps HyperScale achieve quality of service (QoS) objectives and deliver on the promise of a "zero backup window." Veritas HyperScale for Containers introduces read cache support for improving the read workload performance. Currently Veritas HyperScale for Containers uses the Linux native caching solution called the DM cache for enabling the read cache support for Tier1 storage on each of the compute node. Thus it improves the IO performance of all the containerized applications running on the same compute node. Prerequisites 1. Each compute node should have a dedicated SSD device for read cache (This SSD device is in addition to the SSD device required for HyperScale write-back cache (that is Tier0)). 2. In case a compute node has a single SSD device, and if the SSD vendor supports splitting of a single SSD device into two different virtual controllers, then also it serves the purpose. Refer to the SSD vendor documentation for splitting SSD device. 3. The single SSD device should not be partitioned into two partitions, using utilities like fdisk, parted, and so on. This impacts the performance adversely. Installation Separate steps are not required for enabling the read cache on the compute node. During the HyperScale service startup, the read cache gets enabled by the iomgr service depending on the available storage.

9 Chapter 2 Displaying information about your environment This chapter includes the following topics: About the HyperScale management console Displaying an overview of your environment Displaying information about the compute plane and data plane Displaying information about a container Displaying information about a volume About the HyperScale management console You can administer your HyperScale configuration using the HyperScale user interface, referred to as the web management console. The console provides a centralized consolidated graphical view of your entire HyperScale configuration. After installing and configuring the HyperScale cluster, the first step is to use the management console to view the storage that can be made available to the application containers that are running in your container environment.

10 Displaying information about your environment Displaying an overview of your environment 10 To access the web management console 1 Open a web browser and type the following URL in the address bar: Here, [ComputeNodeIPaddress] is the public IP address of any compute node in the HyperScale configuration. Note: Your browser may display a security certificate validation message. You may have to resolve the error by adding a security exception for the specified URL. 2 On the login page, specify the administrative user name and password and click Sign In. 3 The Overview tab displays the various elements of your configuration. You can use the console for performing administrative tasks such as: monitoring and managing HyperScale nodes in the compute and data plane configuring volumes for containers and monitoring storage usage configuring storage policies to meet quality of service (QoS) objectives Displaying an overview of your environment To display an overview of you environment Navigate to the Overview page.

11 Displaying information about your environment Displaying an overview of your environment 11 This page displays the following information: The status of the nodes in the compute plane, the total compute plane storage, the amount used, and the amount free. The status of the nodes in the data plane, the total data plane storage, the amount used, and the amount free.

12 Displaying information about your environment Displaying an overview of your environment 12 An alerts area that displays recent operations, as well as issues you may need to take action on. To display information on all events, click View All. A cluster summary. This area displays aggregate data in the following categories: CPU usage Memory usage Containers running per host Storage utilization To display more detailed information about the compute plane or data plane, click on the Compute Nodes or Data Nodes tabs. See Displaying information about the compute plane and data plane on page 13.

13 Displaying information about your environment Displaying information about the compute plane and data plane 13 Displaying information about the compute plane and data plane To display information about the compute plane Navigate to the Compute Nodes page and review the information displayed under Compute Nodes. The Compute Nodes tab displays the number of compute nodes that are healthy, at risk, or faulted, the number of volumes that are mounted, and the total number of containers running on the node. You can also click on a compute node name to display the following for each node: The node name, status, and IP address CPU and memory utilization The number solid-state drives (SSDs) and hard disk drives (HDDs) associated with the node, the storage used, and the storage available Performance information on Input and output per second (IOPS), throughput, and latency. The bottom of the page displays the containers associated with the node and the number that are running, paused, or stopped. It also displays the following: The container name and status The image the container is based on The number of volumes How long the container has been running If the list of containers is long, enter the container name or image name in the Search field.

14 Displaying information about your environment Displaying information about the compute plane and data plane 14 Click a container name to display detailed information about it. See Displaying information about a container on page 15. You can use this view to add a compute node. To display information about the data plane Navigate to the Data Nodes page and review the information displayed under Data Nodes. The Data Nodes tab displays the number of data nodes that are healthy, at risk, or faulted. You can also click on a data node name to display the following for each node: Name and IP address CPU and memory utilization Total storage available and in use Number of volumes provisioned You can use this tab to add a data node.

15 Displaying information about your environment Displaying information about a container 15 Displaying information about a container To display information about a container 1 Navigate to the Compute Nodes page and click on a compute node. 2 On the node details page, in bottom section, click a container name. 3 Review the container details page. It includes the following information: The container name, status, and image name.

16 Displaying information about your environment Displaying information about a volume 16 CPU usage Memory usage The policy that is associated with the container Displaying information about a volume To display information about a volume 1 Navigate to the Volumes page. 2 Click on a volume name and then review the volume details on the page. It includes the following information: The volume name and status Storage usage

17 Displaying information about your environment Displaying information about a volume 17 A performance summary that displays information on IOPS, throughput, and latency for read, write, aggregate operations The bottom of the page has tabs that display information on the volume's configuration, performance, associated containers, and snapshots. Tab name Storage Policy Information displayed The policy associated with the volume. A policy defines performance, availability, and data protection parameters for the volume, including: The minimum and maximum IOPS guaranteed (threshold) The minimum and maximum latency (threshold) The resiliency factor The number of snapshots and how often they are taken Performance Snapshots A performance graph. Use the drop-down list to select the metric to graph: IOPS, throughput, or latency. The number of snapshots in progress and the number completed. The tab also displays the following information on each snapshot: Name Size The time the snapshot was created

18 Chapter 3 Working with volumes This chapter includes the following topics: Creating a volume Creating a volume from a snapshot Deleting a volume Configuring Kubernetes/OpenShift pods to use HyperScale storage Creating a volume To create a volume 1 Navigate to the Volumes page and click Add Volume. 2 The Create Volume wizard begins with an overview of how volumes are added to your system. Click Next.

19 Working with volumes Creating a volume 19 3 On the Volume Details page, specify the volume details and click Next. Field Volume Name Description Enter a name for the volume. The name can contain a maximum of 32 characters. Volume Size (in GB) Specify the size, in GB, for the volume. You can either type a value or select a number using the arrow buttons. File system type (Optional) Specify the file system type for the volume. The default value is xfs. This is an optional parameter. Select Datanode Select the datanode to be associated with the volume. All the volume snapshots will be maintained on the chosen datanode. The default value is Any, which means that HyperScale automatically chooses the best available datanode based on performance, workload distribution, and storage availability. Description (Optional) Enter some descriptive text that will help you identify this volume and its purpose.

20 Working with volumes Creating a volume 20 4 On the Storage Policies page, select the policy to apply to the volume and then click Next. 5 On the Volume Details page, review your choices. If you wish to make any changes, click Previous and go back to the earlier wizard pages. To add the volume to the configuration, click Finish. The Volumes page is updated with the new volume that you just added.

21 Working with volumes Creating a volume from a snapshot 21 Creating a volume from a snapshot To create a volume from a snapshot 1 Navigate to the Volumes page and click the volume whose snapshot you want to use to create a new volume. 2 On the volume details page, click the Snapshots tab. Identify the snapshot to use and click on the vertical ellipsis at the end of the row and then select Create Volume. Deleting a volume 3 The Create Volume wizard begins. From here, except for specifying the volume size, the steps for creating a volume are the same as those for creating one from scratch. See Creating a volume on page 18. Note the following: You cannot delete a volume if it belongs to a volume group.

22 Working with volumes Configuring Kubernetes/OpenShift pods to use HyperScale storage 22 To delete a volume 1 Navigate to the Volumes page. 2 In the list of volumes, for the volume that you want to delete, on the right side of the volume's table row, click on the vertical ellipsis and select Delete. 3 On the confirmation page, click Confirm to delete the volume. The console displays a pop-up message that confirms the volume deletion. Configuring Kubernetes/OpenShift pods to use HyperScale storage You can create HyperScale volumes from the HyperScale web management console and then attach them to the containers in the cluster. In addition to the user interface, HyperScale also provides a method to configure volumes using the Kubernetes Flexvolume plugin. Flexvolume is a volume plugin that enables storage providers to mount volumes in a Kubernetes cluster. Refer to the Flexvolume documentation for more information: volumes/flexvolume About HyperScale and Flexvolume HyperScale utilizes the Flexvolume plugin to provision HyperScale storage in a Kubernetes cluster environment. When you launch HyperScale on a node, the

23 Working with volumes Configuring Kubernetes/OpenShift pods to use HyperScale storage 23 HyperScale storage driver (vxflexvol) is installed in the following plugins directory on the node: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ The vxflexvol driver is then used to create and mount HyperScale volumes and make them available to the containers in the Kubernetes cluster. HyperScale volume parameters To be able to mount HyperScale volumes on the containers in a pod, the following volume parameters must be specified in a container manifest: Table 3-1 Parameter flexvolume driver HyperScale volume parameters Description The attributes specified under the flexvolume parameter define the storage type. These attributes take predefined values and are not user configurable. "veritas/vxflevol" This parameter defines that the volume created is of type HyperScale. fstype "xfs" This parameter defines xfs as the file system type for the HyperScale volume. options name The attributes specified under the options parameter define granular details about volumes and are user configurable. Specifies the name for the HyperScale volume. The volume name can include a maximum of 32 characters. size Specifies the size, in gigabyte (GB), for the HyperScale volume. To create a volume of 1 GB, you can specify the value as 1, or 1G, or 1 GB. policy Specifies the storage policy type associated with the HyperScale volume. You can define a custom policy or use any of the default policies (Gold, Silver, and Bronze) provided by HyperScale. Storage policy determines the performance, protection, and availability characteristics that get assigned to a volume. See About storage policies on page 42.

24 Working with volumes Configuring Kubernetes/OpenShift pods to use HyperScale storage 24 Sample pod manifest The following example shows a yaml configuration file for a pod (nginx) that uses a HyperScale volume: apiversion: v1 kind: Pod metadata: name: nginx-pod namespace: default spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerport: 80 volumemounts: - name: brnz-vol1 mountpath: /data volumes: - name: brnz-vol1 flexvolume: driver: "veritas/vxflexvol" fstype: "xfs" options: name: "brnz-vol1" size: "1G" policy: "BRONZE" Observe the details specified under the flexvolume and options attributes. The name of the HyperScale volume is brnz-vol1, the size is 1 GB and the storage policy associated is Bronze. Creating a HyperScale volume using a yaml manifest The following procedure shows how to use a yaml configuration file to provision HyperScale volumes in a pod.

25 Working with volumes Configuring Kubernetes/OpenShift pods to use HyperScale storage 25 To create a HyperScale volume for a pod 1 Create a yaml configuration file based on the sample pod manifest provided earlier and save it as a file named nginx.yaml in the root directory of a compute node. The sample specifies creating a sample pod (nginx) that uses a 1 GB HyperScale volume named brnz-vol1 that is associated with a Bronze storage policy. See the section called HyperScale volume parameters on page 23. See the section called Sample pod manifest on page Deploy the yaml file to create the pod. Run the following command on the compute node # kubectl create -f ngnix.yaml This command starts a nginx web service that is accessible to other pods in the cluster. 3 Verify that a HyperScale volume is created as specified in the yaml file. Run the following command on the compute node: # docker volume ls 4 You can also verify that the HyperScale volume is mounted on the compute node.

26 Chapter 4 Working with volume groups This chapter includes the following topics: About volume groups Creating a volume group Viewing volume group details Deleting a volume group Creating a volume group snapshot Creating a volume group from a volume group snapshot Deleting a volume group snapshot Volume group limitations About volume groups HyperScale introduces the concept of volume groups and as the name suggests, a volume group is a collection of volumes that are clubbed together to be represented as a single entity in the HyperScale configuration. Volume groups are also referred to as consistency groups and what that means is that the member volumes in a volume group are synchronized in such a way that their snapshot versions residing on a data node are always crash consistent at any given point in time. In case of a failure, data can be restored from the latest snapshot that was taken before the failure event, and the application can be brought back online. Volume groups are useful in cases where an application is deployed as multiple modules that run across a set of distributed containers on the same host. Each

27 Working with volume groups Creating a volume group 27 module generates data and needs dedicated storage. The application is said to run successfully only when the modules interact with and work together in tandem. A volume group provides crash-consistent storage for such distributed inter-dependent deployments on the same host. Creating a volume group You can create a volume group using the management console or from the command line. Note the following: Mounted volumes cannot be added to a volume group. You first add a volume to a volume group and then attach it to a container. A volume cannot belong to multiple volume groups. To create a volume group from the console 1 Navigate to the Volume Groups tab and click Add Volume Group. 2 On the Introduction page that displays the overall volume group creation workflow, click Next.

28 Working with volume groups Creating a volume group 28 3 On the Volume Group Details page, specify the volume group and volume details and then click Next. Field Volume Group Name Description Specify a name for the volume group. Select Volumes to add to Volume Group From the list of volumes displayed, select the volumes that you want to include in the volume group. 4 Review the volume group summary. If you wish to make any changes, click Previous and go back to the earlier wizard pages. To create the volume groups in the configuration, click Add. The Volume Group page displays the newly created volume group.

29 Working with volume groups Viewing volume group details 29 Viewing volume group details To view a volume group's details using the console 1 Navigate to the Volume Groups tab. 2 From the list of volumes groups, click on a volume group to see its details. The volume group pane displays various details about the selected volume group. The top of the plane displays the name of the volume group, the number of member volumes and user-defined snapshots created for the volume group. 3 Click Snapshots to view details about the snapshots created for the volume group. Deleting a volume group Note the following: Volume groups containing mounted volumes cannot be deleted. Volume groups with active snapshots cannot be deleted.

30 Working with volume groups Creating a volume group snapshot 30 To remove a volume group using the management console 1 Navigate to the Volume Groups tab. 2 In the list of volume groups, find the volume group you want to modify and on the right hand side of the volume group row, click the vertical ellipsis button and select Remove Volume Group. 3 On the confirmation dialog, click Confirm to remove the volume group from the configuration. Creating a volume group snapshot When you create a snapshot for a volume group, HyperScale creates two types of snapshots--a top-level parent snapshot for the volume group and separate snapshots of the individual member volumes in the volume group. Before you create a snapshot, ensure that the volumes that are part of the volume group are mounted in their containers.

31 Working with volume groups Creating a volume group from a volume group snapshot 31 To create a volume group snapshot using the console 1 Navigate to the Volume Groups tab. 2 In the list of volume groups, find the volume group you want to create a snapshot for and on the right hand side of the volume group row, click the vertical ellipsis button and select Create Snapshot. 3 Specify a name for the snapshot and then click Create. Creating a volume group from a volume group snapshot To create a volume group from a volume group snapshot using the console 1 Navigate to the Volume Groups tab. 2 From the list of volume groups displayed, click on the volume group name whose snapshot you wish to use to create a new volume group. 3 In the volume group details pane, click Snapshots. 4 From the list of snapshots displayed, find the snapshot you wish to use to create a new volume group and then on the right side in the snapshot row, click the vertical ellipsis button and select Create Volume Group. This launches the create volume group from snapshot wizard.

32 Working with volume groups Creating a volume group from a volume group snapshot 32 5 On the introduction page that displays the volume group creation workflow, click Next. 6 On the Volume Group Details page, specify the details for the new volume group and click Next. Field New Volume Group Name New Volume Name Prefix Description Specify a name for the volume group you want to create from the existing volume group snapshot. Specify a prefix that is added to the names of the volumes in the new volume group. These are the volumes that are cloned from the volumes in the snapshot.

33 Working with volume groups Deleting a volume group snapshot 33 7 On the Storage Policies page, click to select a storage policy that you wish to associate with the volumes in the volume group and then click Next. The storage policy defines the performance, availability, and resiliency characteristics for the volumes. 8 On the Summary page review the details and then click Create. If you wish to make any changes, click Previous and go back to the earlier wizard pages. To create a volume group based on the specified details, click Create. The new volume group should appear in the list of volume groups on the Volume Groups tab. Deleting a volume group snapshot Deleting a volume group snapshot deletes the top-level parent snapshot of the volume group as well as the individual member volume snapshots. To delete a volume group snapshot from the console 1 Navigate to the Volume Groups tab. 2 From the list of volume groups displayed, click on the volume group name whose snapshot you wish to delete. 3 In the volume group details pane, click Snapshots.

34 Working with volume groups Volume group limitations 34 4 From the list of snapshots displayed, find the snapshot you wish to delete and then on the right side in the snapshot row, click the vertical ellipsis button and select Delete Snapshot. 5 On the confirmation dialog, click Delete Snapshot to confirm the removal of the snapshot. Volume group limitations The following restrictions apply to volume groups: All the volumes that are members of a volume group must have the same snapshot frequency. Volume groups or consistency groups having member volumes with different snapshot frequency are not supported. Member volumes of a volume group can be mounted on containers that must be running on the same host. Volume groups serving containers running on multiple hosts is not supported.

35 Chapter 5 Working with data nodes This chapter includes the following topics: Adding a data node Removing a data node Adding a data node You can add a data node from the HyperScale graphical user interface (GUI) or from the command line. Refer to Veritas HyperScale for Containers Installation Guide for information on how to add a data node. Note: Wait for approximately 30 seconds, for node storage to be discovered before performing any volume operations. About adding data nodes This release of HyperScale for containers has been qualified with up to four data nodes.

36 Working with data nodes Adding a data node 36 Figure 5-1 Typical Veritas HyperScale for Containers configuration Management Network Public Network* (10 GbE) Compute Node 1 Compute Node 2 Compute Node 3 SSD SSD SSD HDD HDD HDD Data VLAN (10 GbE) Data Nodes Data network SSD SSD HDD HDD * Management traffic is routed over the public network. Data nodes are used to store point in time snapshots of the container volumes. Every 5 minutes, the compute nodes write new data to the data plane. This operation is known as Episodic Data Synchronization. For the Silver policy and above, if there are two data nodes in the environment, one data node is the primary node and the other is the secondary node. In this configuration, data is reflected from the primary data node to the secondary data node. If one of the data node fails, all the requests are served from the other data node.

37 Working with data nodes Removing a data node 37 For storage policy with resiliency factor 0, there is no reflection at the data node level. If the data node that is associated with a storage policy with resiliency factor 0 fails, you cannot access all the volume versions. Removing a data node To remove a data node from the cluster do the following: 1. Stop the HyperScale data node services on the data node to be removed. docker run --privileged -u 0 --rm -v /var/run/docker.sock:/var/run/docker.sock -e VSS_HOST_IP=<datanodepublicIPaddress> -e VSS_DATA_PARENT_INTF=<datainterfacename> -e VSS_EULA_ACCEPTANCE=YES veritas/hyperscale:2.0 stop 2.0 datanode Replace the input parameter with the appropriate values, as described in the following table: Input parameter VSS_HOST_IP VSS_EULA_ACCEPTANCE Description Specify the public IP address of the data node. Specify YES (in all caps) to acknowledge and accept the end user license agreement (EULA). To review the EULA, run the following command: docker run --rm veritas/hyperscale:2.0 eula VSS_DATA_PARENT_INTF Specify the name of the network interface that is associated with the private data network of the data node. Messages similar to the following indicate the progress status: Management IP : <public IP address> Data Transfer IP: over subnet /24 on interface <datainterfacename> Stopping Veritas HyperScale 2.0 on data node Stopping vxdatamgr... done Stopping vxdevmgr... done Stopping vxctlr... done Removing vxdatamgr... done

38 Working with data nodes Removing a data node 38 Removing vxdevmgr... done Removing vxctlr... done Removing network vx_default Removing network vx_priv 2. Wait till the data node service failure is detected, and the data node is marked as faulted in the HyperScale cluster. 3. Remove the data node from HyperScale using the HyperScale user interface/api as follows: Navigate to the Data Nodes page. In the list of data nodes, find the data node that you want to remove. On the right side of the data nodes table row, click on the vertical ellipsis and click Remove. On the confirmation page, click Confirm.

39 Chapter 6 Compute node and data node resiliency This chapter includes the following topics: Compute node resiliency Data node resiliency Compute node resiliency In the compute plane, HyperScale provides failure detection and resiliency. If any of the failures occur, an alert is displayed in the Alerts/Events area of the HyperScale dashboard. Failure detection Storage service failure If the service stops or fails an alert is raised. Storage media failure If the media throws an IO error, an alert is raised. Node failure If the node, or the system has either gone down, restarted or panicked, an alert is raised. Resiliency HyperScale maintains data resiliency on the compute nodes through a method called as reflection. When I/O occurs on a container, it is written on the source compute node, and also written (or reflected) on at least one other compute node

40 Compute node and data node resiliency Compute node resiliency 40 in the deployment. These additional compute nodes are called the reflection targets. The number of reflection targets used is the reflection factor. The reflection factor of a container, depends on the policy which is assigned to the volume. To use reflection, a container must have one of the following policies. You can also create custom policies that are tailored to your environment. Table 6-1 Policy Gold HyperScale policies and the reflection factors Reflection Factor 2 Data is written to two reflection targets. Silver 1 Data is written to one reflection target. Bronze Data reflection does not exist. Note: The customized policies also have the reflection factor similar to the standard policies. Source and target node failure If a node fails, HyperScale detects it and marks that node as faulted. If this node was a reflection target for a volume, HyperScale automatically detects it and restores the reflection target of the volume by selecting an available compute node and adding it as a new reflection target. If there is no compute node available, for adding it as a reflection target for a volume, it raises an alert and the volume remains in a degraded mode. If this node was a reflection source for a volume, it automatically detects it and recovers the data to ensure data consistency. If the application is being managed by an orchestrator then the application using this volume, will be failed over by the orchestrator to another node. After the failover, HyperScale ensures that the volume is also available. Storage media failure on the reflection target If a solid-state drive (SSD) or hard disk drive (HDD) fails on a reflection target, HyperScale detects it and restores the reflection factor for all the impacted volumes. HyperScale does this by selecting a compute node and adding it as a new reflection target. For adding it as a reflection target for a volume, in case a compute node is not available, it raises an alert and the volume remains in a degraded mode. Storage media failure on the reflection source

41 Compute node and data node resiliency Data node resiliency 41 Data node resiliency If a solid-state drive (SSD) or hard disk drive (HDD) fails on a reflection source, HyperScale detects it and recovers the data to ensure data consistency. Also, all the subsequent application IOs fail. Based on the alert raised, the administrator will failover the application to another compute node. After the failover, HyperScale ensures that the volume is also available. This release of HyperScale for containers has been qualified with up to four data nodes. The data node acts as a secondary storage tier for storing the vdisk versions. vdisk versions are periodically synchronized with the data node. If a compute node fails, the vdisk version is read from the data node. For a vdisk having the storage policy with resiliency enabled, data is reflected from one data node (primary) to another data node (secondary). The volume vdisks are load balanced amongst the data nodes in a round robin way. HyperScale provides resiliency at the data plane in case of the following data node failures. There is minimal impact on the applications and data management services such as snapshots and recoveries. Alerts are raised for all these failures in the Alerts/Events area of the HyperScale dashboard. Storage media failure If the solid-state drive (SSD) or hard disk drive (HDD) of a data node fails, the ongoing snapshots are aborted, and the data node goes into the Down state. You must remove the data node from the configuration. After replacing the faulty disk, add the same data node again in the configuration. Data network failure If the data network card or the data network cable of a data node fails, the data node cannot communicate with the controller. The data node goes into the Down state. After you fix the network communication issue, remove the data node from the configuration and add it again. Data node failure If a node running data node services fails, the vdisks associated with the failed data node are marked as resiliency degraded and its ownership is transferred automatically to a peer data node. The peer data node starts serving the next snapshots and recovery reads. Any ongoing requests, at the time of failure, restarts on the new primary data node after the ownership change. When the failed data node recovers and joins back, then the vdisks associated with the data node are automatically synced for updates and vdisks data node resiliency is restored.

42 Chapter 7 Ensuring quality of service with policies This chapter includes the following topics: About storage policies Using storage policies Adding a storage policy Deleting a storage policy About storage policies HyperScale includes the following default policies that define different levels of performance, availability, and protection: Gold Silver Bronze You can create custom policies tailored to your environment.

43 Ensuring quality of service with policies Using storage policies 43 Table 7-1 Default policies and their values Performance Protection Availability Policy IOPS Latency No. of Snapshots Frequency of Snapshots Resiliency Factor Gold 300 (min.) 1 (min.) 3 5 minutes 2 30,000 (max.) 10 (max.) Silver 200 (min.) 1 (min.) 3 5 minutes 1 20,000 (max.) 20 (max.) Bronze 100 (min.) 1 (min.) 3 5 minutes 0 10,000 (max.) 100 (max.) See Using storage policies on page 43. See Adding a storage policy on page 43. Using storage policies You can associate a storage policy with a volume using any of the following options: Use one of the default policies See About storage policies on page 42. Edit an existing policy Create a new policy See Adding a storage policy on page 43. Adding a storage policy HyperScale comes with three default storage policies (Gold, Silver, and Bronze) that define levels of performance, availability, and protection for a volume. See About storage policies on page 42. However, you can also create a custom policy to suit your needs. A storage policy includes the following sections:

44 Ensuring quality of service with policies Adding a storage policy 44 The Performance section defines the permissible IOPS and latency ranges for the volume. The Availability section defines the number of reflection targets (peer compute nodes) to replicate application writes to. The Protection section defines the interval at which periodic snapshots of the volume need to be taken and the number of snapshots to be maintained at any given time. To add a storage policy 1 Navigate to the Storage Policies page and click Add Storage Policy. 2 The Add Policy wizard begins with an overview of what policies are and the tunable parameters available while provisioning a volume. Click Next.

45 Ensuring quality of service with policies Adding a storage policy 45 3 On the Policy Details page, specify the policy parameters and then click Next. Field Select values from existing Storage Policy Select storage policy to view values Description Check this option if you wish to use the values from the default storage policies available or from an existing storage policy that was created earlier. Choose the desired storage policy from the drop-down list. After you select a policy, it parameter values are automatically populated in the remaining fields on the wizard page. This field is available only when the Select values from existing Storage Policy checkbox is selected. Policy Name Performance Enter a name for the storage policy. Define the performance characteristics for the policy. You can either type the values or specify them using the arrow buttons. IOPS Latency Define the minimum and maximum IOPs values for the policy. Define the minimum and maximum permissible latency for the policy.

46 Ensuring quality of service with policies Deleting a storage policy 46 Field Availability Resiliency factor Description Define the availability criteria for the storage policy. Define the number of reflection targets for volumes that are assigned this policy. A resiliency factor of two indicates that data is written to two additional compute nodes. Data Protection No. of snapshots Frequency of snapshots Define the data protection levels for the storage policy. Define the number of snapshots that are retained at any given point in time for the volume. Define the interval at which the snapshots are automatically triggered for the volumes that are assigned this policy. 4 On the Policy Details page, review your choices. To make changes, click Previous to go back to the policy parameter settings page. Click Add to create the storage policy. Deleting a storage policy The Storage Policies page displays the policy that you just added. You can only delete the storage policies that you create. You cannot delete the default policies distributed with HyperScale.

47 Ensuring quality of service with policies Deleting a storage policy 47 To delete a storage policy 1 Navigate to the Storage Policies page. 2 In the list of policies, find the policy you want to delete and on the right side of the policy row, click the vertical ellipsis button and select Delete. 3 On the confirmation screen, click Confirm. The Storage Policies page is updated and the selected policy does not appear in the list. If the policy still displays, refresh your browser.

48 Chapter 8 Protecting HyperScale volumes with NetBackup This chapter includes the following topics: About integrating NetBackup into your HyperScale environment NetBackup Media Server requirements NetBackup integration requirements HyperScale integration requirements Registering NetBackup with HyperScale Configuring a default NetBackup policy for HyperScale Creating storage policies in HyperScale using the configured backup policies Backing up volumes by assigning storage policies Restoring backups of a volume About integrating NetBackup into your HyperScale environment Integrating Veritas NetBackup into your Veritas HyperScale environment is a collaboration between the HyperScale infrastructure administrator and the NetBackup administrator. Steps must be performed on both the NetBackup Master/Media server and on HyperScale. For example, before you can register NetBackup as a backup provider with HyperScale, the NetBackup administrator must tell you the Media Server fully qualified domain name and provide you with the backup provider token.

49 Protecting HyperScale volumes with NetBackup About integrating NetBackup into your HyperScale environment 49 Similarly, before the NetBackup administrator can create HyperScale backup policies, the HyperScale infrastructure administrator needs to tell the backup administrator the frequency of the backups for each type of backup policy. The following table shows each backup task in order, and who performs it. Table 8-1 This task... HyperScale and NetBackup tasks Is performed by... Setting up the NetBackup environment Meeting installation requirements NetBackup administrator See NetBackup Media Server requirements on page 50. NetBackup administrator See NetBackup integration requirements on page 51. See HyperScale integration requirements on page 52. Registering NetBackup with HyperScale Configuring a default NetBackup policy for HyperScale HyperScale infrastructure administrator See Registering NetBackup with HyperScale on page 52. NetBackup administrator See Configuring a default NetBackup policy for HyperScale on page 54. Creating storage policies in HyperScale using the configured NetBackup policies HyperScale infrastructure administrator See Creating storage policies in HyperScale using the configured backup policies on page 58. Assigning storage policies for backing up volumes Restoring backups HyperScale infrastructure administrator See Backing up volumes by assigning storage policies on page 59. HyperScale infrastructure administrator See Restoring backups of a volume on page 61. This chapter describes the tasks of a HyperScale infrastructure administrator. It mentions the general tasks the NetBackup administrator must perform, but the specific steps are beyond the scope of this guide. For specific information about NetBackup, refer to Veritas NetBackup 8.0 Administrator's Guide: You can also refer to the NetBackup software compatibility list:

50 Protecting HyperScale volumes with NetBackup NetBackup Media Server requirements 50 Note: In this release, HyperScale is only qualified with NetBackup Version 8. NetBackup Media Server requirements The NetBackup administrator must set up a dedicated Media Server in the production environment. The Media Server is used as a backup proxy. Table 8-2 NetBackup Media Server (backup proxy) requirements Software / Hardware RAM Virtual CPUs Network speed Hardware type Network interface cards (NICs) Physical storage Requirement 32 GB minimum, 64 GB or higher recommended 4 cores minimum, 8 cores recommended 1 Gbps minimum, 10 Gbps recommended Physical machine 2 required Enough physical storage to maintain volume snapshots. HyperScale maintains a full copy of a data node's vdisks and metadata when it is exported to the external storage provider node. For example, if you export five volumes that have a total of 5 TB of data written to their vdisks, the external storage provider node requires 5 TB of storage. Network Port 8334 is open or the firewall on the backup proxy node is disabled. Verify this before you try to register NetBackup from the HyperScale user interface. Packages The following packages are required on the backup proxy node: rpcbind nfs-utils flask version pyopenssl VRTShsbkp RHEL7.x86_64.rpm

51 Protecting HyperScale volumes with NetBackup NetBackup integration requirements 51 In addition, the NetBackup administrator must configure the directory path that HyperScale uses for restore operations; this is known as the staging location. The NetBackup administrator must configure this path as /opt/vrtshsbkp/backup_selection/selection.txt when creating a backup policy in NetBackup for HyperScale. NetBackup integration requirements Note: This HyperScale release is only qualified with NetBackup Version 8.0. To integrate NetBackup with HyperScale, the NetBackup administrator must verify the following: The NetBackup Media Server and Master Server must be installed. When you install the servers, use the fully qualified domain names. Also use the fully qualified domain name when you register the Media Server with the Master Server. The Media Server must be running Red Hat Enterprise Linux (RHEL) 7.2. The "Minimal Install" is not supported; however, the RHEL software "Infrastructure Server" and above are supported. The firewall on the backup proxy node is disabled, or port 8443 is open. (This must be verified before you try to register NetBackup from the HyperScale user interface.) The date, time, and zone on the backup proxy node must be in sync with the date, time, and zone on the Master Server. If the dates and times are not sync, backups still occur, but they are not displayed on the HyperScale user interface. The VRTShsbkp.rpm is installed on the NetBackup Media Server. The rpm is included in the HyperScale installer, which you downloaded earlier using the following link: After this information is verified, the NetBackup administrator must communicate the following information to the HyperScale infrastructure administrator: The Media Server fully qualified domain name. The backup provider token. This token is generated by running the vxbackup get-token command on the Media Server.

52 Protecting HyperScale volumes with NetBackup HyperScale integration requirements 52 HyperScale integration requirements The HyperScale administrator must complete the following tasks for all the data nodes in the HyperScale cluster: Install NFS on the data nodes to display HyperScale volumes in NetBackup. Create a /etc/exports file on all the data nodes and add the following entry to specify the NFS parameters. /veritas x.x.x.x(rw,fsid=0,insecure,subtree_check,sync) /veritas/verfs x.x.x.x(rw,fsid=1,nohide,insecure,subtree_check,sync) where [x.x.x.x] is the IP address of the Media Server where VRTShsbkp RHEL7.x86_64.rpm is installed. Run the exportfs -rv command on all the data nodes after HyperScale is installed. Note: Ensure that the /etc/exportfs file is removed from all the data nodes when you uninstall the data node. If the NFS server terminates unexpectedly, HyperScale volumes and snapshots are not accessible and backup and restore operations fail. In such situations, restart the NFS server on the data nodes and run the following command on the backup proxy to verify that NFS communication is established with the data nodes: # mount -t nfs [datanode IP address]:/verfs /tmp Registering NetBackup with HyperScale Register NetBackup as a backup provider in HyperScale to back up HyperScale volumes using NetBackup policies and schedules. The HyperScale infrastructure administrator performs this task. Ensure that the NetBackup administrator has installed the VRTShsbkp.rpm on the NetBackup Media Server (also known as the backup proxy server) before you register NetBackup as a backup provider. Before you proceed, make sure that the NetBackup administrator has given you the following information: The NetBackup Media Server (backup proxy) management IP address. The public IP address of the backup proxy node. The backup provider token for NetBackup

53 Protecting HyperScale volumes with NetBackup Registering NetBackup with HyperScale 53 The NetBackup administrator must also disable the firewall on the backup proxy server node, or open port Before you register NetBackup, data nodes must be enabled. Registering NetBackup with HyperScale 1 Log in to the HyperScale management console if not already done so by typing the following URL in the address bar of your browser: where [ComputeNodeIPaddress] is the public IP address of any compute node in the HyperScale configuration. 2 In the left navigation pane, click Backup and Restore. 3 On the Backup and Restore page, click Add Backup Provider. 4 In the Add Backup Provider dialog box, specify the following details and then click Add. In the Backup Provider Name field, enter a name for the provider. In the Backup Vendor list, select NetBackup. In the Backup Server IP field, specify the public IP address of the backup proxy node. All HyperScale backup related communication between the backup proxy node and the HyperScale data node is carried over this public IP. In the Management IP field, specify the management IP address of the HyperScale compute node. In the HyperScale Admin Username and HyperScale Admin Password fields, enter the HyperScale administrator credentials and click Verify.

54 Protecting HyperScale volumes with NetBackup Configuring a default NetBackup policy for HyperScale 54 In the Backup Provider Token field, enter the backup provider token that you generated earlier on the NetBackup Media Server. 5 When the registration completes, refresh the Backup and Restore page. NetBackup is now listed as a backup provider for HyperScale. Configuring a default NetBackup policy for HyperScale Use the NetBackup Backup Policy Configuration Wizard to create a default backup policy with a 24X7 calendar schedule. For detailed information on how to create a backup policy using the Backup Policy Configuration Wizard, refer to the NetBackup documentation. Before you proceed, ensure that you have the Backup proxy server management IP address. To create a default NetBackup policy for HyperScale 1 From the NetBackup Administration Console, expand NetBackup Management > Policies and select Actions > New > New Policy. 2 On the Add a New Policy dialog box, specify a name for the HyperScale policy, select the Use Policy Configuration Wizard checkbox, and click OK to start the wizard. 3 On the Create a backup policy for screen, select File systems, databases, applications and then click Next. 4 On the Policy Name and Type screen, confirm the policy name, and from the Select the policy type drop-down list select Standard, and then click Next. Ensure that you select the policy type as Standard. 5 On the Client List screen, click Add and then do the following on the Add Client dialog box: In the Client name space, provide the management IP address of the backup proxy server. From the Hardware and operating system drop-down list, select Linux, RedHat

55 Protecting HyperScale volumes with NetBackup Configuring a default NetBackup policy for HyperScale 55 Click Add and then click OK. 6 On the Client List screen, verify that the backup proxy server management IP appears in the client list and then click Next. 7 On the Backup Selections screen, add /opt/vrtshsbkp/backup_selection/selection.txt as the staging location and click Next. 8 On the Backup Type screen, select Full Backup option and click Next. Ensure that no other option is selected. 9 Follow the remaining wizard steps and complete the workflow with the default values. 10 From the NetBackup Administration Console, launch the Policies Utility and edit the backup policy you created earlier. 11 In the Change Policy screen, ensure that the following options are selected on the Attributes tab: Follow NFS Cross mount points Allow multiple data streams

56 Protecting HyperScale volumes with NetBackup Configuring a default NetBackup policy for HyperScale In the Change Policy screen, on the Schedules tab, perform the following: Click New to add a new calendar schedule and then on the Add Schedule dialog box, specify a name for the schedule and select User Backup as the Type of backup.

57 Protecting HyperScale volumes with NetBackup Configuring a default NetBackup policy for HyperScale 57 On the Start Window tab, select a 24X7 time window. For Start day:, select Sunday and specify Start Time: as 00:00:00. For End day:, select Saturday and specify End Time: as 23:59:59.

58 Protecting HyperScale volumes with NetBackup Creating storage policies in HyperScale using the configured backup policies 58 Click Add and then click OK. 13 Confirm that a new calendar schedule is added to the backup policy you created and then click OK. Creating storage policies in HyperScale using the configured backup policies After the NetBackup administrator creates backup policies in NetBackup, you need to associate these backup policies with the storage policies in HyperScale. You can then assign these storage policies to the volumes you want to back up. To create a storage policy for backing up volumes: 1 Log in to the HyperScale management console. 2 In the left navigation pane, click Storage Policies. 3 In the Add Storage Policy wizard, the Introduction tab displays an overview of the storage policies. Click Next. 4 On the Policy Details tab, specify details to define performance, availability, and protection. 5 On the Backup Policy tab, click Yes to associate a backup policy configured in NetBackup with the HyperScale storage policy. 6 In the Select backup provider list, select the backup provider. This is the name that you specified when you registered NetBackup as a backup provider. The backup policies, which you created earlier in NetBackup for HyperScale are displayed. Select the backup policy that you want to associate with the storage policy and click Next. A new parameter called Backup is displayed, which shows the name of backup policy associated with the storage policy. 7 On the Summary tab, review the details and click Add. The following figures show how a backup policy that was configured in NetBackup is associated with the storage policy that you created.

59 Protecting HyperScale volumes with NetBackup Backing up volumes by assigning storage policies 59 Backing up volumes by assigning storage policies To back up volumes, create volumes with a storage policy that is associated with a backup policy. To back up a volume: 1 Log in to the HyperScale management console, if not already done so. 2 In the left navigation pane, click Volumes, and then click Add Volume.

60 Protecting HyperScale volumes with NetBackup Backing up volumes by assigning storage policies 60 3 In the Create Volume wizard, the Introduction tab displays an overview about how add a volume. Click Next. 4 On the Volume Details tab, specify the volume details such as the name, size, and the file system and click Next. 5 On the Storage Policy tab, select a storage policy for the volume. Ensure that you select a storage policy that is associated with a backup policy, and then click Next. 6 On the User Group tab, optionally associate user groups with the volume. 7 On the Summary tab, review the details and click Finish. The volume is created and volume backups are triggered as per the schedule and frequency configured in the backup policy. When a backup is complete, a notification is displayed in the Recent tasks in the upper-right corner of the HyperScale management console. The backup notification is displayed with a BnR: Backup prefix. The following figures shows a volume named nbu_vol, which is created with the NEW_POLICY storage policy.

61 Protecting HyperScale volumes with NetBackup Restoring backups of a volume 61 The following figure shows that the volume is backed up based on the backup policy called test which is associated with the NEW_POLICY storage policy. Restoring backups of a volume You can restore a backup as a volume. Restoring backups enables you to create volumes with previous point in time snapshots. To restore a backup: 1 Log in to the HyperScale management console, if not already done so. 2 In the left navigation pane, click Backup and Restore. 3 Click the backup provider and select the policy. The volumes that are backed up using the policy are displayed. 4 Find the volume for which you want to restore the backup and click Restore located in the far right corner of the row.

62 Protecting HyperScale volumes with NetBackup Restoring backups of a volume 62 5 When prompted for confirmation, click Yes. The backups for the selected volume are displayed on the Volumes tab in the [volumename]_r_timestamp format. For example, the backup for volume named nbu_vol is nbu_vol_r_ _4:42:9 Note: The time required to complete the Restore operation depends on the size of the backup. After the Restore operation is complete, a notification is displayed in the Recent tasks with the BnR:Restore prefix. 6 Click the backup that you want to restore and click Snapshots. 7 For the displayed snapshot, in the far right corner of the row click Create Volume. Note: The size of the snapshot is always displayed as 0 and does not represent the actual size of the volume. 8 Follow the steps in the Create Volume wizard and click Finish to create the volume. The backup is restored as a volume and displayed on the Volumes tab. You can now mount the volume and use it as per your need. The following figure shows the snapshot for the nbu_vol volume: