DataMan. version 3.7.0

Transcription

1 DataMan version 3.7.0

2 Contents DataMan User Guide 1 Overview 1 Introduction 1 DataMan 1 Technical Specifications 1 Hardware Requirements 1 Software Requirements 2 Ports 2 Other Configuration 2 Data Encryption 2 DataMan Installation 3 Component Installation 3 License Installation 4 Creating Endpoints 4 Adding an AWS Glacier Endpoint 6 Adding an AWS S3 Endpoint 6 Adding a SSH Host Endpoint 6 Adding Credentials 7 Transferring Files 7 DataMan Settings 8 CLI Installation 9 CLI Initialization 9 Store Storage Service Credentials 9 Configuration 10 Create Endpoints 10 REST API Specification 10 Command Line Client 11

3 Initialization 11 List Encryption Keys 12 Store Encryption Key 12 Create a Storage Endpoint 12 List Endpoints 12 Create and Upload a AWS Glacier Archive 13 List Archives 13 List Contents of an archive 13 Creating a Recurring transfer 14 Downloading an Archive 14 Searching for files 14 Job Status 15 Scaling DataMan 15

4

5 DataMan User Guide Overview The DataMan provides CycleServer plugins that allow managing archiving and retrieval of user data to Cloud Storage. DataMan makes it easy to manage backup, archival and transfer of data residing in various storage services and mediums. It supports Amazon's Glacier for archival and provides encryption for data archived using user provided keys. Customizable plugins compress and then encrypt the files before uploading them to Glacier. By default AES Encryption is performed using the key specified by the user. A REST API is provided to perform various operations on encryption keys that allow users to store and retrieve encryption keys and perform archival ans retrieval of folders residing on the local file system. Once the compressed archive is uploaded metadata about the archive such as the key used for encryption, source location and the location of a generated manifest file is stored in CycleServer. The manifest file contains file system specific information about all the files in the archive such as file ownership, permissions, various creation and/or modification timestamps etc as obtained from the underlying file system. A command line client utilizing the REST API is provided. Introduction This document serves as the user guide for the DataMan product versions 2.x and 3.x. The document also provides technical specifications of DataMan. DataMan DataMan at its core is based on the Cycle telemetry engine operating on a nosql datastore. It has a pluggable architecture and makes use of components to support various operations. It also provides a rich query interface and a rich alerts and reporting framework that can be used to create custom reports and alerts on the datastore information, such as user actions, scheduled data transfers and collected metadata. The DataMan component supports data transfer and management operations by operating on data transfer requests created by adding records to the datastore. These records contain both user information and information needed to perform the data transfer. All operations performed by the DataMan component are also written to the datastore to provide an audit trail. This information is made available via a REST API and command line clients which will be detailed in later sections. Technical Specifications Hardware Requirements 1

6 We recommend running CycleServer with DataMan on a Linux machine with 4-8 cores, 8-16 GB of RAM and at least 500GB of free space for optimal performance. The amount of RAM required is proportional to the number of parallel streams used to transfer data. The recommended setup is for 8 parallel streams with a 64GB part size. Every additional stream needs an additional 1-2GB of RAM. The disk space required by DataMan is a function of the largest file size to upload/download. The amount of free scratch space is should be at least 3 times the largest file to upload/download. Software Requirements DataMan requires a Linux system capable of running Oracle Java 1.7. The required Java Runtime Environment is bundled with CycleServer. All RHEL 5.x and its clones, Ubuntu and SLES 11+ are supported platforms. Root access is required during installation for user creation and enabling automated startup. The cycle_server user and group must be created prior to installation if default user creation facilities of Linux are inadequate to create user accounts. Python 2.7 is needed for execution of the Client. Ports In order to communicate with clients CycleServer requires ports 8080 and 8443 to be open to machines on which clients are installed. For Amazon Glacier transfers outgoing traffic to the Amazon Glacier endpoints must be allowed and for S3 transfers outgoing traffic to S3 endpoints must be allowed. Other Configuration CycleServer must have read access to the filesystem from which files need to be uploaded to Glacier. In addition if data deletion on inventory is enabled write access is required. Special scripts required to query ACL values of files must adhere to ACL input format described in the user guide. Write access is needed to the filesystem to which data must be restored. Additionally any administrative privileges required to perform ACL changes are required to restore ACLs on downloaded files. Data Encryption DataMan encrypts each file to be archived post compression using strong AES 256-bit encryption. The Encryption is performed using the standard Java Cryptography Extensions (JCE) security libary provided with the Oracle Java Runtime Environment. The JCE library provide standards based, industry strength cryptographic methods providing tools for strong encryption and avoiding the pitfalls of using weak encryption keys. The JCE SecureRandom class ensures that cryptographically secure random keys are generated and used for encryption of data. 2

7 Users can generate or import encryption keys and use them to encrypt data when uploading to Amazon Web Services Glacier or S3. In order to improve security due to encryption it is possible create a key family and specify the key family name during encryption. Specifying a key family to be used results in a new key to be generated for each transfer request using that key family. Recurring syncs, for example, can use a key family causing a new key to be generated and used each time the sync is scheduled to run. When uploading data, encryption is performed in parallel on fixed size chunks of data. The encryption keys are seeded with a platform dependent random number generator. On Linux based systems /dev/urandom is used to obtain the random bytes for use in generating the encryption keys and on Microsoft Windows the Microsoft Crypto API is used to request random numbers. The random numbers on both Linux and Windows are generated using entropy from various sources on the system such as clock ticks, network and disk I/O, key strokes if applicable, thermal variations of hardware components etc. The system blocks until enough entropy is available before returning the next set of random bytes, thus ensuring high quality random data. This ensures that the random numbers are impossible to reproduce without physical access to the system and knowledge of the numerous physical events that went into generation of the random data. The JCE libraries also provide flexibility to replace the built in implementation of the Random number generator with an implementation that makes use of hardware randomization if available. DataMan provides a way to export the encryption keys out of DataMan for backup purposes. Administrative users can specify a key backup script that can be run at scheduled intervals to backup keys provided to it as a json file to a secure location. The cycle_server account and the key backup script must be secured in such a way that they cannot be tampered with leading to a security breach. DataMan Installation Install DataMan by unpacking the compressed archive provided and execute the install.sh script on Linux and install.cmd on Windows. On Linux the install script must be executed as root. DataMan is currently supported on Linux, Windows and Mac OS X platforms. If DataMan is installed from a prepackaged DataMan installation proceed to License Installation. If the components are provided separately proceed to the component installation step before configuring the DataMan installation from the web interface. Component Installation If DataMan is being installed on a preexisting CycleServer installation, then follow this step to install DataMan. Copy the file_sync component zip file provided to the $CS_HOME/components directory. Next copy the DataMan component zip file into the same location. Wait a few minutes for the components to be installed. Finish CycleServer 3

8 installation by pointing your browser to the CycleServer web interface as provided at the end of the CycleServer installation. By default this is host>:8080 or License Installation At the end of the insttaltion wizard you will be taken to the Apps page, where you can install alicense to activate all the DataMan features. On the apps page, if you already have a license, paste the license text in the text box provided under the License tab as shown below. Figure 1. License Installation If you do not already have a license, copy the nodeid shown and it to sales@cyclecomputing.com to receive a license. Upon receiving the license, paste the license text in the space provided on this page. Creating Endpoints Once the license is installed, click on the Data link on the top menu to proceed to the DataMan home page. The first time the page is opened, it will start with the "Add Endpoint" forms on both the left and right pane as shown in Figure 2. 4

9 Figure 2. Initial Endpoint Add forms The left pane automatically defaults to the localhost endpoint which is used to view and operate on data residing on the DataMan host. The Local Host check box must be enabled to configure endpoints on the DataMan host. The Allowed Paths text box allows you to add paths that DataMan can browse and retrieve and store data to. This is set to /tmp by default and should be changed to point to the data staging area on the DataMan host. If only Cloud storage operations will be performed and no data will be downloaded, the defaults can be accepted. Once you click Save the localhost endpoint named DataMan will be created and the file browser pane will show the allowed paths as in Figure 3. 5

10 Figure 3. Local Endpoint Configured The right pane can be used to similarly add endpoints of different types. The configuration options for the other endpoints are described in the following sections. Adding an AWS Glacier Endpoint To add a Glacier endpoint, type in a name for the endpoint on the Add Endpoint form and then select Amazon Glacier from the Endpoint Type drop down menu. If this is the first time configuring an AWS endpoint or to use a new set of credentials click on the plus sign next to the Credential dropdown to add a new Credential. More detail on adding credentials is presented in section Adding Credentials. After adding a credential, select the AWS Glacier Region in which the vault is located. Please note that a vault should already be present the the region of your choice before adding the DataMan endpoint. After selecting the region, the Vault selection dropdown will be automatically populated with the vaults available in that region. Select the desired vault and click Save. Adding an AWS S3 Endpoint To add an AWS S3 endpoint, use the endpoint form to select Amazon S3 Endpoint type, pick or enter a new set of AWS credentials and select the bucket to use from the drop down menu. Please note that for endpoint creation to work from the endpoint creation form, the credentials must have list buckets privileges for Amazon S3. Adding a SSH Host Endpoint To add an ssh host endpoint, select Add Endpoint from the title menu on one of the endpoint browser panes. This willreplace the pane with the Endpoint Creation form. Name the endpoint appropriately and select Host from the endpoint type drop down. If not already unchecked, uncheck the Local host check box and fill in the remote host's host name or IP address in the Hostname field. Also fill in the account name to login as and select the SSH Private Key that will be used to connect to the host from the Credential dropdown as shown in Figure 4. 6

11 Figure 4. Adding SSH Host Endpoint Adding Credentials Credentials can be added either from the Endpoint Add form by clicking the plus sign next to the Credential text box or by navigating to the Credential type (under Other types) by browsing to <dataman-url>:<dataman-port>/types and selecting All from the component selection menu. Transferring Files Data is transferred between endpoints by using the data transfer buttoms located between the left and right endpoint browser panes. The simple transfer button can be used to start an immediate transfer from one endpoint to another. It should however be noted that an immediate transfer simply submits the transfer for execution as soon as a worker is available to perform the transfer. The number of workers can be configured by editing the Queue records and changing the workers attribute to teh desired number of workers. The default setting is 1 and is recommended for most purposes. 7

12 DataMan Settings In order to tune DataMan transfers a number of settings can be adjusted. The settings page can be navigated to using the user menu on the top right of the screen and selecting Settings. On the settings page double clicking on the DataMan entry brings up the DataMan settings form as shown in Figure 5. Figure 5. DataMan Settings form Proxy Server Configuration To enable using a proxy server for AWS operations check the Use Proxy checkbox and put the details of the proxy server in the followign fields. SSL for AWS Operations The Enable SSL checkbox enables the use of https when connecting to the AWS Storage endpoints. It is enabled by default and must be unchecked to fall back to using http. Although not recommended it may be needed if there are too many "Peer not authenticated" or HTTP connection errors during transfers. Server Side Encryption The S3 encryption checkbox is used to enable server side encryption when transferring data to S3. Please note that this is not the same as using encryption via DataMan, which preencrypts the data before transferring it to S3 or Glacier. The Server Side Encryption enables data to be encrypted using AWS managed keys when at rest in the S3 bucket. S3 Uploader This settings modifies the number of threads used to transfer data to S3. Recommended value of this setting depends on your network bandwidth and the number of CPU cores available on the DataMan host. We recommend that this value not be set greater than 32 to prevent http connection errors during uploads. 8

13 S3 Part Size This is the size of a single chunk of a multi-part upload that is uploaded using a single thread to S3. The chunk size is a power of 2 and specified in bytes. Amazon S3 places a upper limit of on the number of parts, so the chunk size should be chosen appropriately based on the largest file to be uploaded. For instance if the largest file to be uploaded is larger than 320GB and less than 640GB, the part size must be 64 MB ( ). If the largest file size is smaller than 320 GB, it is recommended that the part size be 32 MB. For very small files a part size of 8 or 16 MB can be used. Please note that when converting from MB to bytes calculate the number of bytes as (size in MB) * 1024 * 1024 to obtain a power of 2 size in bytes. Change Checkers Change checkers are used to determine if a file has been changed and needs to be copied from one S3 bucket to another. Change checkers perform HEAD operations on S3 objects to retrieve metadata that can be used to determine if the object has changed. DataMan compares the ETag attribute on the S3 object to the MD5 checksum of the file disk or to the ETag of the source S3 object. If the ETag is not an MD5 checksum, as is the case with multi-part uploads, DataMan looks for the metadata attribute "user-md5", which is set by DataMan to be the MD5 checksum of the file. If this attribute is not found or is different, DataMan overwrites the object with the new one. For host to S3 uploads it will also set the "user-md5" metadata attribute. Amazon S3 automatically prefixes the metadata attribute with "x-amz-meta-", so the attribute on the S3 console will appear as "x-amz-meta-user-md5". If objects have been uploaded to S3 using multi-part uploads, the metadata attribute should be added to the objects for reliable syncing, without relying on the ETag. CLI Installation Unzip the CLI bundle to the location of your choice. Place the dataman binary found in the bin folder on your PATH for convenient access to dataman commands. CLI Initialization Intialize dataman using the dataman initialize command. Provide your username and password when prompted: $ dataman initialize Welcome to DataMan! CycleServer URL: [ CycleServer username: [akulshrestha] admin CycleServer password: Generating CycleServer key... DataMan configuration stored in /Users/akulshrestha/.cycle/config.ini Store Storage Service Credentials 9

14 Use the storeawscredentials to store the Glacier AWS secret key and access key to be used by CycleServer: dataman storeawscreds -a <accesskey> -s <secretkey> Configuration Use the dataman config command to configure dataman network settings and notification settings. The Web interface should be used in preference to this command. Answer the questions on the prompts and finish configuration: $ dataman config list for archive status notification: @demo.com Dataman application settings configured. Local Endpoint Created. Advanced configuration options can be invoked by using the dataman config all command. Use this to specify the number of upload streams, size of each upload chunk, command to be executed to gather additional file attributed etc. Create Endpoints The dataman createendpoint command is used to create Endpoints for AWS Glacier vaults and AWS S3 buckets. To configure DataMan to use a AWS Glacier vault as an endpoint: $./dataman createendpoint -t Glacier Endpoint Name: test AWS Glacier Region: us-east-1 AWS Glacier Vault Name: test DataMan Endpoint Created. For S3 buckets: $./dataman createendpoint -t S3 Endpoint Name: s3test AWS S3 Bucket: com.mybucket.bucketname DataMan Endpoint Created. Dataman is now configured and ready to use. REST API Specification DataMan provides a RESTful interface for configuration, key management and data transfer operations. The configuration operations interact with the CycleServer datastore to store configuration items used by DataMan to perform data transfers. The data transfer operations store transfer requests in the datastore which are then scheduled for execution. Key Management List Keys 10

15 GET /ads/datamanager.encryptionkey&format=json Store Key Autogenerated Key: GET /datamanager/rest?o=store&keyname=name Specified Key: GET /datamanager/rest?o=store&keyname=name&key=bda12deadbeef431 Archive Operations List Archives GET /ads/datamanager.archive List Archive Contents GET /glacier/rest?o=listarchive&a=archivename Upload an archive GET /glacier/rest?o=upload&k=keyname&a=archivename&v=vault&p=/upload/from&t=tags Download an archive or a file from an archive GET /glacier/rest?o=download&k=keyname&a=archivename&v=vault&d=/download/to GET /glacier/rest?o=download&k=keyname&a=archivename&v=vault&d=/download/to&f=file Command Line Client Users will interact with DataMan using the command line client called dataman. A help screen is provided by invoking dataman help which lists the various commands and their options. (Note: On Windows \ must be escaped in paths - C:\\manifests): $ dataman help Initialization 11

16 The initialize command is used to perform one time configuration of the command line client enabling it to securely communicate with the CycleServer instance hosting the Cycle DataMan REST API.: $ dataman initialize List Encryption Keys The listencryptionkeys command shows all keys stored in CycleServer.: $ dataman listencryptionkeys KeyName:Symmetric Key examplekey _1: c d8a476f203a0772c7185eb7b6a1795bf3586ac3d5d Store Encryption Key The storeencryptionkeys command allows users to store a key into CycleServer to encrypt and decrypt files with. If no key is provided then a key is automatically generated and stored. The keyname is used as a handle to identify the key to be used with other commands.: $ dataman storeencryptionkeys <keyname> [key] Create a Storage Endpoint Storage endpoints refer to data sources and destinations which may include traditional file systems, file storage services, archival services etc.: $ dataman createendpoint -t Glacier Endpoint Name: glacier AWS Glacier Region: us-east-1 AWS Glacier Vault Name: examplevault1 DataMan Endpoint Created. $ dataman createendpoint -t S3 Endpoint Name: s3example AWS S3 Bucket: com.cyclecomputing.software DataMan Endpoint Created. List Endpoints Lists all available endpoints.: $ dataman listendpoints Glacier Name : Vault : Region 12

17 glacier : examplevault1 : us-east-1 $ dataman listendpoints S3 Name : Bucket s3example : com.cyclecomputing.software Create and Upload a AWS Glacier Archive The upload command creates a compressed and encrypted archive and uploads it to Amazon Glacier.: $ dataman upload -c vaultname -k keyname -d /path/to/upload -t "tag1=value1,tag2=value2" archivename List Archives The listarchives command is used to query CycleServer for the archives that have been uploaded to Glacier.: $ dataman listarchives examplevault1 Name : Status : Key Used examplearchive _6 : Inventoried : testkey-1 examplearchive _5 : Inventoried : testkey-1 examplearchive _7 : Inventoried : testkey-1 List Contents of an archive To see the contents of an archive: $ dataman listarchive arch _1 Wed Mar 6 13:00: arch _3.manifest Mon Feb 4 22:48: b6e480a a5-c17e2b manifest Wed Mar 6 16:47: arch _9.manifest Mon Feb 4 21:21: file1.dat Mon Feb 4 21:36: file3.dat Fri Feb 15 12:45: arch _1.manifest Mon Feb 4 22:13: e7a38c79-275a b b manifest Mon Feb 4 21:37: file4.dat Tue Feb 26 12:51: arch _4.manifest Mon Feb 4 21:35: file2.dat Fri Mar 8 12:19: arch _1.manifest Tue Feb 5 00:13: arch _1.manifest Wed Mar 6 13:55: arch _6.manifest total

18 Creating a Recurring transfer A recurring sync can be created to backup new or changed files to glacier. MD5 checksums of files are used to determine if they are changed. The -s option can be used to specify an interval in seconds or a Quartz cron schedule string: $ dataman createrecurringsync -n sync1 -s 60 -e glacier -a winsync1 -k wintest1 -d "C:\testData" Schedule strings are of the format "ss mm hh dd mon dayofweek". Examples of schedule strings. 1. Every 20 seconds: "0/20 * * * *?" 2. Every other minute, starting at 15 seconds past the minute: "15 0/2 * * *?" 3. Every other minute, between 8am and 5pm: "0 0/ * *?" 4. At 10am on the 1st and 15th days of the month: "0 0 10am 1,15 *?" 5. Every 30 seconds on Weekends (Saturday and Sunday): "0,30 * *? * SAT,SUN" Downloading an Archive The download command requests an archive to be downloaded, downloads it when the request has been fulfilled by Amazon Glacier, decrypts, uncompress and explodes it to the provided location. Only inventoried archives from inventoried vaults can be downloaded.: $ dataman download -c vaultname -k keyname -d /path/to/explode/to archivename $ dataman download -c vaultname -k keyname -d /path/to/explode/to -f filename archivename Searching for files The dataman search command can be used to locate files within archives or archives with specified tags.: $ dataman search -f file1.dat arch_unenc_nomd5_512parts8mb_ _5#11: file1.dat arch_unenc_nomd5_256parts_ _4#10: file1.dat arch_unenc_nomd5_64parts_ _2#9: file1.dat arch_unenc_nomd5_64parts_ _1#8: file1.dat arch_unenc_32parts_ _4#7: file1.dat arch_unenc_ _3#6: file1.dat arch _3#5: file1.dat arch _1#3: file1.dat= $ dataman search -t project=demo arch_unenc_nomd5_256parts_ _4#10: project=demo,owner=demouser 14

19 Job Status To show the status of a download request or an inventory request: $ dataman jobstatus arch _1 : archivedownload : COMPLETED : J0VEbh68gKA_AD --- : inventory : COMPLETED : LbDi-mSHh8V56L --- : inventory : COMPLETED : E8gP3B_sHdte6n --- : inventory : EXPIRED : 6c5qHIGBaaHeJU arch _1 : archivedownload : COMPLETED : wptsffd7xtiirs Scaling DataMan If the aggregate bandwidth available for Glacier uploads is greater than the sustained bandwidth usage of a single DataMan node, multiple nodes should be used. In order to determine the bandwidth usage efficiency the following factors must be taken into consideration. Data Preprocessing When dealing with large volumes of data, it must often be pre-processed to package and optionally compress the data. This can be a time consuming process and the throughput depends on resources available on the node that pre-processes the data. For compression of large files, parallel compression tools such as pigz are recommended on multi-core machines. If the upload speed to Glacier is faster than the aggregate bandwidth of the preprocessing pipeline, it is recommended that more than one node be used to preprocess data to be presented for upload to DataMan. The goal of the pre-process pipeline should be to always have one additional dataset available for upload before the previous upload finishes. Upload Bandwidth Single DataMan nodes will often peak out below the net available bandwidth to the Archival Service endpoint. This can be due to various reasons, such as interface speeds, limited number of cores on the node or limited I/O bandwidth. If MD5 checksum computation is enabled, additional time is needed to compute the checksum, before data is uploaded. Due to the nature of the checksum algorithm, it runs on a single core and is not parallelized. Hence for large files this becomes a bottleneck and reduces the net upload bandwidth to Glacier. If prepackaging of data creates a few large files, the 15

20 checksum computation is a significant overhead. In this case, multiple DataMan nodes must be used to make best use of the bandwidth. In the perfect scenario at least one of the nodes must be uploading data while the other is computing checksums or waiting on data to become available from the pre-processing pipeline. The DataMan nodes should be individually tuned to determine the right upload part size and concurrency required to achieve the highest upload speeds possible for that node. If the peak bandwidth is less than 80% of the total bandwidth available to the archival service endpoint, then additional DataMan nodes are needed to make full use of the available bandwidth. Please note that this is only valid on a congestion free network. For a network with substantial traffic adding more DataMan nodes may not offer any benefit. On congested networks the network traffic patterns must be evaluated before making any recommendation. Additional DataMan nodes can also be used to maximize bandwidth use if the MD5 checksum computation takes more than half the time required to upload the dataset. When adding another node it is beneficial to start the next upload on the new DataMan node once the first node has started uploading data to the archival service. When data sets are of similar or comparable size, this can be done by putting the uploads on a schedule that allows one node to be computing the checksum while the other is uploading data. 16