DataMan. version 3.5.1

Transcription

1 DataMan version 3.5.1

2 Contents DataMan User Guide 1 Overview 1 Introduction 1 CycleServer 1 DataMan 1 Technical Specifications 2 Hardware Requirements 2 Software Requirements 2 Ports 2 Other Configuration 2 Data Encryption 2 DataMan Installation 3 Component Installation 3 License Installation 4 CLI Installation 4 Initialization 4 Store Storage Service Credentials 4 Configuration 4 Create Endpoints 5 REST API Specification 5 Command Line Client 6 Initialization 6 List Encryption Keys 6 Store Encryption Key 7 Create a Storage Endpoint 7 List Endpoints 7 Create and Upload a AWS Glacier Archive 7 List Archives 8

3 List Contents of an archive 8 Creating a Recurring transfer 8 Downloading an Archive 9 Searching for files 9 Job Status 9 Scaling DataMan 10 AWS S3 Operations 11 Server Side Encryption 11 Versioning Support 11

4

5 DataMan User Guide Overview The DataMan provides CycleServer plugins that allow managing archiving and retrieval of user data to Cloud Storage. DataMan makes it easy to manage backup, archival and transfer of data residing in various storage services and mediums. It supports Amazon's Glacier for archival and provides encryption for data archived using user provided keys. Customizable plugins compress and then encrypt the files before uploading them to Glacier. By default AES Encryption is performed using the key specified by the user. A REST API is provided to perform various operations on encryption keys that allow users to store and retrieve encryption keys and perform archival ans retrieval of folders residing on the local file system. Once the compressed archive is uploaded metadata about the archive such as the key used for encryption, source location and the location of a generated manifest file is stored in CycleServer. The manifest file contains file system specific information about all the files in the archive such as file ownership, permissions, various creation and/or modification timestamps etc as obtained from the underlying file system. A command line client utilizing the REST API is provided. Introduction This document serves as the user guide for the DataMan product versions 2.x and 3.x. The document also provides technical specifications of DataMan. CycleServer CycleServer at its core is a telemetry engine on top of a nosql datastore. It has a pluggable architecture and makes use of components to support various operations. It also provides a rich query interface and a rich alerts and reporting framework that can be used to create custom reports and alerts on the datastore information such as user actions, data transfer volumes etc. DataMan The DataMan component supports data transfer and management operations by operating on data transfer requests created by adding records to the CycleServer datastore. These records contain both user information and information needed to perform the data transfer. All operations performed by the DataMan component are also written to the datastore to provide an audit trail. This information is made available via a REST API and command line clients which will be detailed in later sections. 1

6 Technical Specifications Hardware Requirements We recommend running CycleServer with DataMan on a Linux machine with 4-8 cores, 8-16 GB of RAM and at least 500GB of free space for optimal performance. The amount of RAM required is proportional to the number of parallel streams used to transfer data. The recommended setup is for 8 parallel streams with a 64GB part size. Every additional stream needs an additional 1-2GB of RAM. The disk space required by DataMan is a function of the largest file size to upload/download. The amount of free scratch space is should be at least 3 times the largest file to upload/download. Software Requirements DataMan requires a Linux system capable of running Oracle Java 1.7. The required Java Runtime Environment is bundled with CycleServer. All RHEL 5.x and its clones, Ubuntu and SLES 11+ are supported platforms. Root access is required during installation for user creation and enabling automated startup. The cycle_server user and group must be created prior to installation if default user creation facilities of Linux are inadequate to create user accounts. Python 2.7 is needed for execution of the Client. Ports In order to communicate with clients CycleServer requires ports 8080 and 8443 to be open to machines on which clients are installed. For Amazon Glacier transfers outgoing traffic to the Amazon Glacier endpoints must be allowed and for S3 transfers outgoing traffic to S3 endpoints must be allowed. Other Configuration CycleServer must have read access to the filesystem from which files need to be uploaded to Glacier. In addition if data deletion on inventory is enabled write access is required. Special scripts required to query ACL values of files must adhere to ACL input format described in the user guide. Write access is needed to the filesystem to which data must be restored. Additionally any administrative privileges required to perform ACL changes are required to restore ACLs on downloaded files. Data Encryption DataMan encrypts each file to be archived post compression using strong AES 256-bit encryption. The Encryption is performed using the standard Java Cryptography Extensions (JCE) security libary provided with the Oracle Java Runtime Environment. The JCE library provide standards based, industry strength cryptographic methods providing tools for strong 2

7 encryption and avoiding the pitfalls of using weak encryption keys. The JCE SecureRandom class ensures that cryptographically secure random keys are generated and used for encryption of data. Users can generate or import encryption keys and use them to encrypt data when uploading to Amazon Web Services Glacier or S3. In order to improve security due to encryption it is possible create a key family and specify the key family name during encryption. Specifying a key family to be used results in a new key to be generated for each transfer request using that key family. Recurring syncs, for example, can use a key family causing a new key to be generated and used each time the sync is scheduled to run. When uploading data, encryption is performed in parallel on fixed size chunks of data. The encryption keys are seeded with a platform dependent random number generator. On Linux based systems /dev/urandom is used to obtain the random bytes for use in generating the encryption keys and on Microsoft Windows the Microsoft Crypto API is used to request random numbers. The random numbers on both Linux and Windows are generated using entropy from various sources on the system such as clock ticks, network and disk I/O, key strokes if applicable, thermal variations of hardware components etc. The system blocks until enough entropy is available before returning the next set of random bytes, thus ensuring high quality random data. This ensures that the random numbers are impossible to reproduce without physical access to the system and knowledge of the numerous physical events that went into generation of the random data. The JCE libraries also provide flexibility to replace the built in implementation of the Random number generator with an implementation that makes use of hardware randomization if available. DataMan provides a way to export the encryption keys out of DataMan for backup purposes. Administrative users can specify a key backup script that can be run at scheduled intervals to backup keys provided to it as a json file to a secure location. The cycle_server account and the key backup script must be secured in such a way that they cannot be tampered with leading to a security breach. DataMan Installation Install CycleServer by unpacking the compressed archive provided and execute the install.sh script on Linux and install.cmd on Windows. On Linux the install script must be executed as root. The DataMan toolkit is currently supported on Linux, Windows and Mac OS X platforms. If DataMan is installed from a prepackaged DataMan installation proceed to License Installation. If the components are provided separately proceed to the component installation step before configuring the CycleServer installation from the web interface. Component Installation 3

8 If DataMan is being installed on a preexisting CycleServer installation, then follow this step to install DataMan. Copy the file_sync component zip file provided to the $CS_HOME/components directory. Finish CycleServer installation by pointing your browser to the CycleServer web interface as provided at the end of the CycleServer installation. By default this is host>:8080 or After creating the user click through the installation wizards to accept defaults. Once this is done install the DataMan component by placing the component zip file provided in the $CS_HOME/components directory. License Installation Open the CycleServer GUI URL and navigate to /datamanager/rest. If a license is not installed you will be redirected to the Apps page. On the apps page, if you already have a license, paste the license text in the text box provided under the License tab. If you do not already have a license, copy the nodeid shown and it to sales@cyclecomputing.com to receive a license. Upon receiving the license, paste the license text in the space provided on this page. CLI Installation Unzip the CLI bundle to the location of your choice. Place the dataman binary found in the bin folder on your PATH for convenient access to dataman commands. Initialization Intialize dataman using the dataman initialize command. Provide your username and password when prompted: $ dataman initialize Welcome to DataMan! CycleServer URL: [ CycleServer username: [akulshrestha] admin CycleServer password: Generating CycleServer key... DataMan configuration stored in /Users/akulshrestha/.cycle/config.ini Store Storage Service Credentials Use the storeawscredentials to store the Glacier AWS secret key and access key to be used by CycleServer: dataman storeawscreds -a <accesskey> -s <secretkey> Configuration 4

9 Use the dataman config command to configure dataman network settings and notification settings. The Web interface should be used in preference to this command. Answer the questions on the prompts and finish configuration: $ dataman config list for archive status notification: @demo.com Dataman application settings configured. Local Endpoint Created. Advanced configuration options can be invoked by using the dataman config all command. Use this to specify the number of upload streams, size of each upload chunk, command to be executed to gather additional file attributed etc. Create Endpoints The dataman createendpoint command is used to create Endpoints for AWS Glacier vaults and AWS S3 buckets. To configure DataMan to use a AWS Glacier vault as an endpoint: $./dataman createendpoint -t Glacier Endpoint Name: test AWS Glacier Region: us-east-1 AWS Glacier Vault Name: test DataMan Endpoint Created. For S3 buckets: $./dataman createendpoint -t S3 Endpoint Name: s3test AWS S3 Bucket: com.mybucket.bucketname DataMan Endpoint Created. Dataman is now configured and ready to use. REST API Specification DataMan provides a RESTful interface for configuration, key management and data transfer operations. The configuration operations interact with the CycleServer datastore to store configuration items used by DataMan to perform data transfers. The data transfer operations store transfer requests in the datastore which are then scheduled for execution. Key Management List Keys GET /ads/datamanager.encryptionkey&format=json Store Key Autogenerated Key: GET /datamanager/rest?o=store&keyname=name 5

10 Specified Key: GET /datamanager/rest?o=store&keyname=name&key=bda12deadbeef431 Archive Operations List Archives GET /ads/datamanager.archive List Archive Contents GET /glacier/rest?o=listarchive&a=archivename Upload an archive GET /glacier/rest?o=upload&k=keyname&a=archivename&v=vault&p=/upload/from&t=tags Download an archive or a file from an archive GET /glacier/rest?o=download&k=keyname&a=archivename&v=vault&d=/download/to GET /glacier/rest?o=download&k=keyname&a=archivename&v=vault&d=/download/to&f=file Command Line Client Users will interact with DataMan using the command line client called dataman. A help screen is provided by invoking dataman help which lists the various commands and their options. (Note: On Windows \ must be escaped in paths - C:\\manifests): $ dataman help Initialization The initialize command is used to perform one time configuration of the command line client enabling it to securely communicate with the CycleServer instance hosting the Cycle DataMan REST API.: $ dataman initialize List Encryption Keys 6

11 The listencryptionkeys command shows all keys stored in CycleServer.: $ dataman listencryptionkeys KeyName:Symmetric Key examplekey _1: c d8a476f203a0772c7185eb7b6a1795bf3586ac3d5d Store Encryption Key The storeencryptionkeys command allows users to store a key into CycleServer to encrypt and decrypt files with. If no key is provided then a key is automatically generated and stored. The keyname is used as a handle to identify the key to be used with other commands.: $ dataman storeencryptionkeys <keyname> [key] Create a Storage Endpoint Storage endpoints refer to data sources and destinations which may include traditional file systems, file storage services, archival services etc.: $ dataman createendpoint -t Glacier Endpoint Name: glacier AWS Glacier Region: us-east-1 AWS Glacier Vault Name: examplevault1 DataMan Endpoint Created. $ dataman createendpoint -t S3 Endpoint Name: s3example AWS S3 Bucket: com.cyclecomputing.software DataMan Endpoint Created. List Endpoints Lists all available endpoints.: $ dataman listendpoints Glacier Name : Vault : Region glacier : examplevault1 : us-east-1 $ dataman listendpoints S3 Name : Bucket s3example : com.cyclecomputing.software Create and Upload a AWS Glacier Archive 7

12 The upload command creates a compressed and encrypted archive and uploads it to Amazon Glacier.: $ dataman upload -c vaultname -k keyname -d /path/to/upload -t "tag1=value1,tag2=value2" archivename List Archives The listarchives command is used to query CycleServer for the archives that have been uploaded to Glacier.: $ dataman listarchives examplevault1 Name : Status : Key Used examplearchive _6 : Inventoried : testkey-1 examplearchive _5 : Inventoried : testkey-1 examplearchive _7 : Inventoried : testkey-1 List Contents of an archive To see the contents of an archive: $ dataman listarchive arch _1 Wed Mar 6 13:00: arch _3.manifest Mon Feb 4 22:48: b6e480a a5-c17e2b manifest Wed Mar 6 16:47: arch _9.manifest Mon Feb 4 21:21: file1.dat Mon Feb 4 21:36: file3.dat Fri Feb 15 12:45: arch _1.manifest Mon Feb 4 22:13: e7a38c79-275a b b manifest Mon Feb 4 21:37: file4.dat Tue Feb 26 12:51: arch _4.manifest Mon Feb 4 21:35: file2.dat Fri Mar 8 12:19: arch _1.manifest Tue Feb 5 00:13: arch _1.manifest Wed Mar 6 13:55: arch _6.manifest total Creating a Recurring transfer A recurring sync can be created to backup new or changed files to glacier. MD5 checksums of files are used to determine if they are changed. The -s option can be used to specify an interval in seconds or a Quartz cron schedule string: $ dataman createrecurringsync -n sync1 -s 60 -e glacier -a winsync1 -k wintest1 -d "C:\testData" 8

13 Schedule strings are of the format "ss mm hh dd mon dayofweek". Examples of schedule strings. 1. Every 20 seconds: "0/20 * * * *?" 2. Every other minute, starting at 15 seconds past the minute: "15 0/2 * * *?" 3. Every other minute, between 8am and 5pm: "0 0/ * *?" 4. At 10am on the 1st and 15th days of the month: "0 0 10am 1,15 *?" 5. Every 30 seconds on Weekends (Saturday and Sunday): "0,30 * *? * SAT,SUN" Downloading an Archive The download command requests an archive to be downloaded, downloads it when the request has been fulfilled by Amazon Glacier, decrypts, uncompress and explodes it to the provided location. Only inventoried archives from inventoried vaults can be downloaded.: $ dataman download -c vaultname -k keyname -d /path/to/explode/to archivename $ dataman download -c vaultname -k keyname -d /path/to/explode/to -f filename archivename Searching for files The dataman search command can be used to locate files within archives or archives with specified tags.: $ dataman search -f file1.dat arch_unenc_nomd5_512parts8mb_ _5#11: file1.dat arch_unenc_nomd5_256parts_ _4#10: file1.dat arch_unenc_nomd5_64parts_ _2#9: file1.dat arch_unenc_nomd5_64parts_ _1#8: file1.dat arch_unenc_32parts_ _4#7: file1.dat arch_unenc_ _3#6: file1.dat arch _3#5: file1.dat arch _1#3: file1.dat= $ dataman search -t project=demo arch_unenc_nomd5_256parts_ _4#10: project=demo,owner=demouser Job Status To show the status of a download request or an inventory request: $ dataman jobstatus arch _1 : archivedownload : COMPLETED : J0VEbh68gKA_AD 9

14 --- : inventory : COMPLETED : LbDi-mSHh8V56L --- : inventory : COMPLETED : E8gP3B_sHdte6n --- : inventory : EXPIRED : 6c5qHIGBaaHeJU arch _1 : archivedownload : COMPLETED : wptsffd7xtiirs Scaling DataMan If the aggregate bandwidth available for Glacier uploads is greater than the sustained bandwidth usage of a single DataMan node, multiple nodes should be used. In order to determine the bandwidth usage efficiency the following factors must be taken into consideration. Data Preprocessing When dealing with large volumes of data, it must often be pre-processed to package and optionally compress the data. This can be a time consuming process and the throughput depends on resources available on the node that pre-processes the data. For compression of large files, parallel compression tools such as pigz are recommended on multi-core machines. If the upload speed to Glacier is faster than the aggregate bandwidth of the preprocessing pipeline, it is recommended that more than one node be used to preprocess data to be presented for upload to DataMan. The goal of the pre-process pipeline should be to always have one additional dataset available for upload before the previous upload finishes. Upload Bandwidth Single DataMan nodes will often peak out below the net available bandwidth to the Archival Service endpoint. This can be due to various reasons, such as interface speeds, limited number of cores on the node or limited I/O bandwidth. If MD5 checksum computation is enabled, additional time is needed to compute the checksum, before data is uploaded. Due to the nature of the checksum algorithm, it runs on a single core and is not parallelized. Hence for large files this becomes a bottleneck and reduces the net upload bandwidth to Glacier. If prepackaging of data creates a few large files, the checksum computation is a significant overhead. In this case, multiple DataMan nodes must be used to make best use of the bandwidth. In the perfect scenario at least one of the nodes must be uploading data while the other is computing checksums or waiting on data to become available from the pre-processing pipeline. The DataMan nodes should be individually tuned to determine the right upload part size and concurrency required to achieve the highest upload speeds possible for that node. If the peak bandwidth is less than 80% of the total bandwidth available to the archival 10

15 service endpoint, then additional DataMan nodes are needed to make full use of the available bandwidth. Please note that this is only valid on a congestion free network. For a network with substantial traffic adding more DataMan nodes may not offer any benefit. On congested networks the network traffic patterns must be evaluated before making any recommendation. Additional DataMan nodes can also be used to maximize bandwidth use if the MD5 checksum computation takes more than half the time required to upload the dataset. When adding another node it is beneficial to start the next upload on the new DataMan node once the first node has started uploading data to the archival service. When data sets are of similar or comparable size, this can be done by putting the uploads on a schedule that allows one node to be computing the checksum while the other is uploading data. AWS S3 Operations DataMan supports various operations on Amazon S3 Endpoints, such as copying data from the DataMan host to a S3 bucket, downloading objects from a S3 bucket, copying obects between S3 buckets and updating metadata attributes on existing objects. Some of the specific features that can be enabled via the DataMan GUI are: Server Side Encryption Server side encryption can currently be enabled globally which affects all objects created after the setting is turned on. To enable Server side encryption, navigate to the Settings page by clicking on User menu in the top right corner and select settings. Double click on DataMan to bring up the settings pane and use the Server Side Encryption checkbox to enable or disable SSE for new objects. Note that turning on or off SSE will not modify existing obects in the bucket. Versioning Support Enabling versioning support from the DataMan settings dialog allows displaying all verisons of objects in versioning enabled buckets. Copying and downloading of versioned S3 objects is also possible just like regular objects once versioning is enabled. Once versioning is enabled the S3 Object name is followed by the Version Id so that the various versions can be seen in the GUI. 11