USING VERITAS ENTERPRISE VAULT WITH DELL EMC ELASTIC CLOUD STORAGE

Transcription

1 USING VERITAS ENTERPRISE VAULT WITH DELL EMC ELASTIC CLOUD STORAGE ABSTRACT This white paper explains how to use Dell EMC ECS with Veritas Enterprise Vault December, 2017 WHITE PAPER

2 The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. EMC 2, EMC, the EMC logo, are registered trademarks or trademarks of EMC Corporation in the United States and other countries. All other trademarks used herein are the property of their respective owners. Copyright 2016 EMC Corporation. All rights reserved. Published in the USA. October 2016 white paper part no H15309 EMC believes the information in this document is accurate as of its publication date. The information is subject to change without notice. EMC is now part of the Dell group of companies. Part Number H

3 TABLE OF CONTENTS INTRODUCTION... 5 Dell EMC ECS Streamer Overview... 5 Dell EMC ECS - Multi-protocol, API-accessible Storage... 6 Simple Multi-Tenancy... 6 Multi-site, Active-Active Architecture and Access... 6 Audience... 6 Terminology... 6 ECS CONFIGURATION DETAILS... 7 Namespace compliance... 7 Bucket Default Retention Setting... 7 ECS STREAMER INSTALLATION... 8 Download the Dell EMC ECS Streamer Driver... 8 ECS STREAMER CONFIGURATION... 9 Add Dell EMC ECS as a Storage Option in Enterprise Vault... 9 Configuring a Dell EMC ECS Streamer based Enterprise Vault Store Partition... 9 DETAILS OF THE ECS STREAMER DRIVER Network Ports Retention Retention Policies Capacity Information WORM and NON-WORM Support Safe Copy Check SUPPORTED ENVIRONMENTS Veritas Enterprise Vault Dell EMC Elastic Cloud Storage IP LOAD BALANCER REQUIREMENT Overview HAProxy Setup PERFORMANCE INFORMATION Testing Configuration Results APPENDIX A EXAMPLE HAPROXY CONFIG FILE

4 4

5 INTRODUCTION Dell EMC ECS Streamer Overview Veritas has developed a Storage Streamer API for Enterprise Vault (EV) which archive storage vendors must integrate with to allow Enterprise Vault to archive files/ s/etc. to their storage systems. Dell EMC has developed the ECS Storage Streamer driver to allow Veritas Enterprise Vault to archive to Dell EMC Elastic Cloud Storage (ECS). The Dell EMC ECS Streamer driver translates the Streamer API calls to the ECS S3 API. Customers can now create Vault Store partitions of type Dell EMC ECS Streamer within the Veritas Enterprise Vault VAC. These Vault Store partitions are associated with S3 buckets on the ECS cluster where files archived to the Vault Store will reside. DELL EMC ECS buckets reside within ECS namespaces that may be compliant or non-compliant namespaces. Both compliant and non-compliant namespaces allow retention to be propagated from Enterprise Vault to the ECS cluster, however compliant namespaces store archive data in tamper-resistant storage which meets strict SEC 17a-4(f) rules for electronic record-keeping. Figure 1 shows an example of a Dell EMC ECS based storage solutions for Veritas Enterprise Vault. Figure 1. Veritas Enterprise Vault support of Dell EMC Elastic Cloud Storage The Veritas Enterprise Vault system has archive policies that archive files to a Vault Store in which an EMC ECS based partition has been defined. The ECS Streamer driver uses the Dell EMC S3 API to store/access objects in the S3 bucket on the Dell EMC ECS cluster. Dell EMC ECS systems are clusters of 4 or more individual ECS nodes and currently in order to allow all nodes to be accessed via the Dell EMC Streamer driver a Network IP Load Balancer should be deployed between the Veritas Enterprise Vault server and the Dell EMC ECS cluster. 5

6 Dell EMC ECS - Multi-protocol, API-accessible Storage ECS is a massively scalable, software-defined object storage platform that enables any organisation to store, access and manipulate unstructured data as objects. ECS provides support for industry standard APIs such as Amazon S3. In addition ECS extends the Amazon S3 API with support for retention, byte range updates and atomic appends. Simple Multi-Tenancy ECS delivers a flat software architecture ideal for multi-tenant environments. Configuring, provisioning, creating a namespace and managing a multitenant platform has never been easier. Key metrics and reporting are provided for capacity, object count, objects created, objects deleted and inbound/outbound bandwidth. These activities are all visible via the ECS GUI and through the REST API. Multi-site, Active-Active Architecture and Access ECS features a truly geo-efficient architecture that stores, distributes and protects data both locally and geographically. This eliminates any single point of failure and provides seamless failover from site to site with no impact to the business. ECS automatically maximises throughout, maintains high availability and data durability, and increases capacity and the reliability of applications. In terms of geographic limitations there are none providing users with a single global namespace with anywhere access to content Today s modern applications demand a different architecture. Managing both cost and accessibility as storage environments grow and become more complex is one of the biggest challenges organisations face today. Developers are finding it easier to go to public cloud alternatives putting data at risk and increasing management costs. ECS provides a cloud-scale storage architecture that is specifically designed to support modern applications with unparalleled availability, protection, simplicity and scale. Audience This document is intended for Storage Administrators and/or Dell EMC Professional Services personnel who wish to learn how to deploy and configure Dell EMC s ECS object storage with Veritas Enterprise Vault. Terminology Abbreviation ECS EV S3 LB VDC Bucket Description Elastic Cloud Storage Veritas Enterprise Vault Simple Storage Service (API) IP Network Load Balancer ECS Virtual Data Centre A bucket is a logical unit of storage on an Dell EMC ECS system in which objects (including their metadata) are stored Table 1. Terminology 6

7 ECS CONFIGURATION DETAILS How the ECS is configured (e.g. number of ECS nodes, number of VDC s, GEO protection and application location access) is decided by the Solution Architect designing the ECS configuration and for the most part Veritas Enterprise Vault is unaware of the design. However, the Solution Architect should take into account the compliance requirements of the Customers Enterprise Vault solution. Namespace compliance If the customer wishes to have an SEC compliant solution then the Namespace that the buckets that all Enterprise Vault partitions will use should be configured as compliant When creating the Namespace in the ECS Management GUI ensure the Compliance option is selected Bucket Default Retention Setting Create the bucket you are creating for the EV Vault Store partition in the compliant Namespace. Ensure that Bucket Retention is set to a minimum of 1 second. This must be set or an error will occur when you attempt to create the bucket. 7

8 Administrators should take care when setting ECS bucket level retention. Enterprise Vault will expect to be able to delete expired content and although EV does cleanly handle errors on deleting content that is still under ECS retention, it will post an error in the Windows Event Log on each failure. Always set the bucket level retention to 1 second for Enterprise Vault Partitions. ECS STREAMER INSTALLATION Download the Dell EMC ECS Streamer Driver Download the ECS Streamer driver ahead of the installation. The ECS Streamer installer can be found on Once you have downloaded the Dell EMC ECS Streamer driver onto the Veritas Enterprise Vault server, double-click the installation file and it will install itself. Click Next to installer the ECS Streamer driver. If the ECS Streamer driver is already installed on the server it will be re-installed by this process. The ECS Streamer driver should be installed on all Enterprise Vault Servers. If you are re-installing the ECS Streamer driver you may be required to reboot your Enterprise Vault server. 8

9 ECS STREAMER CONFIGURATION Add Dell EMC ECS as a Storage Option in Enterprise Vault Refer to the procedure for this documented on Veritas support web site at this is only required to be performed on one Enterprise Vault server instance. Configuring a Dell EMC ECS Streamer based Enterprise Vault Store Partition In the VAC, expand the Enterprise Vault Site and Vault Store Groups and Right-Click on the Vault Store you wish to create a new Vault Store Partition for and click New->Partition then Next. Choose an appropriate Name and Description and click Next> Choose Dell EMC Elastic Cloud Storage from the Storage Type: drop down 9

10 Then Next > Enter the connectivity details for the ECS cluster to be connected to and select Test to check the configuration details. Property Host Port Bucket Name Access key Secret key Safe copy check Use proxy Namespace Description Enter the IP address or FDQN of the LB service. See HOST PROPERTY DISCUSSION BELOW Default Enter here the port number being used for data connection to the ECS. Enter the name of the ECS bucket to be used Enter the ECS access key (object user) to be used Enter the access key s secret key 0 or 2 See the Safe Copy Check discussion below. Do not use option 1 at present Set to 0 if no proxy is being used or port 8888 for use with a proxy Enter the name of the namespace in which the bucket named above resides Click the Test button to perform a check of the configuration settings. If you get an error recheck the settings as entered. The namespace property is only used in the dtquery request (used when safe copy check is 2, see the section below). The namespace property is irrelevant to the actual archiving process and it is not used in S3 object requests. The object user is globally unique and belongs to a namespace, therefore the namespace is implicitly identified. If the Test is successful then select NEXT > 10

11 In the next GUI form, select the appropriate WORM setting for the partition. The ECS Streamer driver has been validated for NON- WORM and well as WORM Enterprise Vault partitions, check or un-check this radio button appropriately. Complete the following Forms as required. HOST PROPERTIES With version of the Dell ECS Streamer driver some additions were made to the syntax of the Host property to allow the administrator to set some internal settings of the Dell ECS Streamer driver. These new options are - MAX_TRIES, - DISABLE_WRITE_TEST, - MAX_CONNS, - DTHOST - RDT - SHOWCHUNKS In order to use the options, the Host IP (or name) must come first and the other options can be specified using a semicolon separator in <name>=<value> format e.g ;MAX_TRIES=2;MAX_CONNS=25;DISABLE_WRITE_TEST=1 MAX_TRIES When any server failure response is received, the streamer will attempt a total of 5 tries by default before returned bad status to Evault. There is a standard exponential back-off algorithm between retries. This value can be overridden by the MAX_TRIES option which must be greater than 0 and less than 5. MAX_CONNS This controls the maximum number of simultaneous safe copy checks that can occur at one time. The default is 65 as of rev of the Streamer Driver, customers should not need to change this number however is may be set between 2 and 100 DISABLE_WRITE_TEST 11

12 When creating a Vault Store Partition for the first time when there is a Temporary Site Outage (TSO), the tests performed when the user clicks on the Test Button will fail because of the TSO. We recommend that you do not create Vault Store Partitions whilst a TSO is in effect but if you must then setting DISABLE_WRITE_TEST=1 will disable the test. DTHOST=<hostname> or <IP> This allows the user to specify the endpoint for the DTQUERY geo replication checking to a different host than the one specified for the Vault Store partition. This should only be changed under the instruction of Dell EMC support personnel. RDT=1 When set this will set "&random=<someuniqueguid>" to the end of the DTQUERY URL to make sure the URL response is never cached. This <someuniqueguid> is never reused to ensure the request URL is unique. Although additional flags are now used in the request constructor to prevent caching, but this variation is still available for use. SHOWCHUNKS=1 This will include the chunkids and chunk ranges for files written in the DTRACE log, only if the files was archived to the bucket owning site. Do not enable this unless told to by Dell EMC support personnel. If enabled there will be an effect on performance. 12

13 DETAILS OF THE ECS STREAMER DRIVER Network Ports The following network ports are used to communicate between the Enterprise Vault Server and ECS. Port No Port 9020 or 9021 Default http connection or default https port. TCP to the ECS Required to perform the Enterprise Vault safe copy check. TCP to the ECS If the customer uses an IP Load Balancer they may choose to use different HTTP or HTTPS ports from the Enterprise Vault server to the IP Load Balancer. Ports used should be configured in any IP Load Balancer for TCP to the ECS nodes. At present the port # 9101 cannot be mapped to another port by any Load Balancer being used. Retention Enterprise Vault allows users to associate retention policies to archived objects/ files/ s etc. The ECS Streamer driver provides Enterprise Vault with the capability of associating these retention policies with the objects stored on the ECS. The ECS object store will enforce the objects retention and will not allow objects to be deleted until its retention has expired. Objects archived to ECS cannot have their retention shortened or extended by Enterprise Vault. Enterprise Vault will not any apply retention periods to objects stored in a Dell ECS Streamer Partition that is configured as NON- WORM. Retention Policies The Dell EMC ECS Streamer driver does not support Dell EMC ECS Retention Policies. Capacity Information The ECS Streamer driver does not at this stage implemented the Enterprise Vault Storage Streamer API Capacity method. Dell EMC ECS is a scale-out architecture and each EV Vault Store Partition has available to it all the capacity of the entire ECS cluster. ECS also provides the end-user capacity consumption metrics which will display current stats e.g. 13

14 WORM and NON-WORM Support Veritas defines WORM and NON-WORM as follows WORM Enterprise Vault will set the retention period defined in an Enterprise Vault Retention Category to the objects written to an ECS Streamer Partition. NON-WORM Enterprise Vault will not set the retention period defined in an Enterprise Vault Retention Category to the objects written to an ECS Streamer Partition. When a NON-WORM partition is created this allows the following - Even if a Vault Store is using a Retention Category that has a retention period, users using the Enterprise Vault Search Browser can select a document(s) for deletion and as there is no-retention on the object(s) on the ECS, the ECS will allow the deletion to succeed. Administrators for example use this to set documents to be Retained Forever in Enterprise Vault so that the Enterprise Vault Expiration Task will not be run on the Partition but still allow users to delete individual documents. - Enterprise Vault Administrators can change the retention period of a Retention Category and can shorten it. Subsequent runs of the Enterprise Vault Expiration Task will act on the new shortened retention period and will attempt to delete the content which will succeed as there is no retention on the object on the ECS. Dell EMC ECS 3.x is supported as a WORM or NON-WORM enabled Enterprise Vault archive device for EV11 and EV12. When any Veritas Vault Store Partition is created and uses the ECS Streamer driver, check the radio box below to enable WORM storage, un-check to use NON-WORM. This settings cannot be changed once the Partition has been created. 14

15 Safe Copy Check Enterprise Vault has a Safe Copy Check feature which can be enabled by customers at the Vault Store level. This check is to ensure that content archived by Enterprise Vault is not replaced in its original location by an HTM link until it has been backed up to a second archive device via the devices replication functionality. This approach is commonly used by customers who use archive devices with replication functionality that have an eventual consistency model. With Centera, for example, it can be several minutes before replication has created the second copy of the archive content and the Safe Copy Check is commonly used. ECS however implements a strong consistency model and the latencies that exist in the Centera replication queues do not exist in ECS. Customer can be assured that an object that is written to one Virtual Data Centre in a Replication Group will also be available to all other Virtual Data Centres in that Replication Group. The more advanced multi-site architecture implemented by ECS therefore eliminates the need for the Enterprise Vault Safe Copy verification. The concept of a safe copy means that an archived object is geo replicated on a secondary VDC. However, within a VDC, archived files are always stored in a resilient manner (i.e. Erasure coding and a copy). Please read the ECS storage documentation for more information. Note at present the ECS S3 API doesn t support safe copy check however the ECS Streamer Driver has been written to accommodate this in future revisions of the ECS S3 API. Customers who wish to perform this check will need to specify option 2 in the table following. This will cause the ECS Streamer Driver to use dtquery to perform the check. It is anticipated that in the future when the ECS S3 API supports the safe copy check it will be more efficient and will use the same port as the EMC S3 API. If safe copy check is set to 2 the dtquery request to validate a safe archive copy (i.e. Geo replication) will be sent. The test will only check that a valid response is received. The ECS streamer does not control the message in the failure alert window. ECS streamer will attempt to log a specific reason. Should a customer wish to use the Enterprise Vault safe copy check they can enable this as before on the Vault Store and then choose option 2. This is set on the Partition setup Management GUI The following table shows the 3 possible values for the Safe Copy Check parameter. Safe copy check value Has the file been backed up 0 Always return true. The archived object is always considered safe. 1 Reserved for future use. Do not use at present 2 Uses the ECS dtquery API check to determine status. This will require that port 9101 is open in any firewall between the Enterprise Vault server and the ECS. Notes 1. When using the dtquery option, port 9101 must be open in any firewall between the Enterprise Vault Server and ECS cluster. 2. There is a restriction that port 9101 is fixed and so any LB must be set up with this taken into consideration. 3. Ensure that ECS Streamer driver or above is installed. 4. ECS buckets used by Enterprise Vault should not be file system enabled. 15

16 SUPPORTED ENVIRONMENTS Veritas Enterprise Vault Veritas as a detailed support matrix for all Enterprise Vault including 3 rd party Software and Hardware components at Dell EMC Elastic Cloud Storage Veritas EV10, EV11 and EV12 support Dell EMC Elastic Cloud Storage from revision 2.2 upwards. STREAMER RELEASE INFORMATION Streamer Release # Description Default MAX_CONNS changed to 65 Added optional SHOWCHUNKS confuration parameter to assist debugging Added optional RDT parameter to ensure requests are not cached Added option DTHOSTS parameter to assist with debugging Issue caused by caching in the Windows HTTP library resolved Modified the Test() function. Since Evault itself also calls the Test functionality, it is not reasonable to fail the test if the dtquery call fails. This information is logged but does not fail the connectivity test. This portion of the test is only done when safecopy==2. Unfortunately dtquery has problems during TSO. So any call to the Test functionality by Evault when safecopy==2 during TSO, would cause archiving to stop. During archiving any failure in dtquery simply sets the safe copy boolean to false and archiving continues Added a fix for a potential Read() problem where Windows may requires a larger buffer 16

17 IP LOAD BALANCER REQUIREMENT Overview The Dell EMC ECS Streamer driver uses the Dell EMC S3 API to communicate with the backend ECS storage cluster. At present the configuration of the Dell EMC ECS Streamer driver takes only one IP address so in order for the driver to utilise all the nodes in the ECS cluster an IP load balancer is required. Dell EMC has no preference in the IP load balancer used, it may be hardware based or software based. The Dell EMC ETD-SA group used the open source software load balancer HAProxy in their labs to perform the Veritas Enterprise Vault self-certification testing. Future revisions of the ECS Streamer driver may address the requirement to have an external IP load balancer by implementing load balancing in the driver itself. HAProxy Setup HAProxy is an open source IP Load Balancer which is available free. It can be installed on a minimal Linux host (Virtual or Hardware based) with as little of 4GB of RAM, single CPU, 50GB Disk and a single NIC. For example with CentOS, once you have installed the base CentOS distribution (available at you can install HAProxy with the following commands # yum y install haproxy Next edit the file /etc/haproxy/haproxy.cfg, Appendix A contains an example of the HAPRoxy used in the ETD SA lab for testing. Once you have configured the haproxy.cfg file you can start haproxy with the command # service haproxy start # chkconfig haproxy on This service start is nor persistent over a reboot, so you should include these commands in the systems init file. You can obtain service details by running the command # service haproxy status If you see errors indicating that haproxy could not bind to its ports use this command and restart haproxy # setenforce 0 # service haproxy restart # service haproxy status Note; This is included purely as an illustration of how HAProxy was used in the running of the Veritas Enterprise Vault Self Certification tests in our lab. Customer implementations should consider issues like single point of failure etc in designing and implementing LB configuration for Production use. 17

18 PERFORMANCE INFORMATION At present no performance testing has been performed in order to provide details that can be used to assist architects sizing and designing Enterprise Vault solutions. However, as part of the Veritas Enterprise Vault Self Certification testing performed by Dell EMC, some basic throughput measurements were taken. Testing Configuration The following system configurations were used for testing with Enterprise Vault revisions tested. EV10 EV11 EV12 SQL Server DB 8 x vcpu, 32Gb RAM 3 x 100GB SAN Disk 10GbE Nic 8 x vcpu, 32Gb RAM 3 x 100GB SAN Disk 10GbE Nic 8 x vcpu, 32Gb RAM 3 x 100GB SAN Disk 10GbE Nic Windows 2008R2 STD SQL Server 2008R2 Windows 2008R2 STD SQL Server 2008R2 Windows 2012R2 STD SQL Server 2014 EVServer 1 8 x vcpu, 32Gb RAM 3 x 100GB SAN Disk 10GbE Nic 8 x vcpu, 32Gb RAM 3 x 100GB SAN Disk 10GbE Nic 8 x vcpu, 32Gb RAM 3 x 100GB SAN Disk 10GbE Nic Windows 2008R2 STD Windows 2008R2 STD Windows 2012R2 STD EVServer 2 8 x vcpu, 32Gb RAM 3 x 100GB SAN Disk 10GbE Nic 8 x vcpu, 32Gb RAM 3 x 100GB SAN Disk 10GbE Nic 8 x vcpu, 32Gb RAM 3 x 100GB SAN Disk 10GbE Nic Windows 2008R2 STD Windows 2008R2 STD Windows 2012R2 STD Results EV10 EV11 EV12 Ingest Files/Hour Ingest Mbytes/hour Read Test Files/Hour Expiration Test Files/hr File sizes used in these tests was on average 64Kbytes. 18

19 APPENDIX A EXAMPLE HAPROXY CONFIG FILE The settings specific to the ECS in our test lab are highlighted # # Global settings # global log local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon # turn on stats unix socket stats socket /var/lib/haproxy/stats # Default SSL material locations ca-base /etc/ssl/certs crt-base /etc/ssl/private # Default ciphers to use on SSL-enabled listening sockets. # For more information, see ciphers(1ssl). ssl-default-bind-ciphers keecdh+arsa+aes:krsa+aes:+aes256:!kedh:!low:!exp:!md5:!anull:!enull ssl-default-bind-options no-sslv3 # # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block # defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except /8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 5000 timeout client timeout server timeout http-keep-alive 10s timeout check 10s maxconn 3000 # # main frontend which proxys to the backends # frontend ecs05_s3_http_in :9020 default_backend ecs05_s3_http # # round robin balancing between the various backends # backend ecs05_s3_http balance leastconn # Set the X-Forwarded-For header with source IP option forwardfor # Use the s3 Ping ViPR extension to make sure node is alive option httpchk GET /?ping HTTP/1.1\r\nHost:\ haproxy\r\nx-emc-namespace:\ foo # Workaround for AWS JS SDK and Storage-2177 reqrep ^(X-Amz-Date:)(.*)$ x-amz-date:\2 # List of data nodes. S3 runs on port server ecs05-01-s3-http :9020 check server ecs05-02-s3-http :9020 check server ecs05-03-s3-http :9020 check server ecs05-04-s3-http :9020 check 19

20 listen stats :1936 mode log http global maxconn 10 clitimeout srvtimeout contimeout timeout queue 100s 100s 100s 100s stats enable stats hide-version stats refresh 30s stats show-node stats auth admin:password stats uri /haproxy?stats 20