DataMan. version 5.5.7

Transcription

1 DataMan version 5.5.7

2 Contents DataMan User Guide 1 Introduction 1 DataMan 1 Technical Specifications 2 Hardware Requirements 2 Software Requirements 2 Ports 2 Other Configuration 2 DataMan Installation 2 Component Installation 3 License Installation 3 Data Encryption 4 Creating Endpoints 4 Adding an AWS S3 Endpoint 6 Adding a Microsoft Azure Endpoint 6 Adding a Google Cloud Platform Endpoint 6 Adding an SSH Host Endpoint 6 Adding Credentials 7 Creating AWS Credentials 7 Creating GCP Credentials 7 Creating Azure Credentials 8 Transferring Files 8 Scheduled Transfer 8 Recurring Transfer 8 Monitoring Transfer Details 9 Transfer Agents 10 DataMan Settings 11

3 CLI Installation 15 CLI Initialization 16 Store Storage Service Credentials 16 Create Endpoints 16 Command Line Client 16 Initialization 16 List Encryption Keys 17 Create Encryption Keys 17 Create a Storage Endpoint 17 List Endpoints 17 Create and Upload a AWS Glacier Archive 18 List Archives 18 List Contents of an archive 18 Creating a Recurring transfer 18 Downloading an Archive 19 Searching for files 19 Job Status 19 REST API Specification 20 Scaling DataMan 21 Transfer Listeners 22 Release Notes 23 DataMan version DataMan version DataMan version DataMan version DataMan version DataMan version DataMan version DataMan version

4 DataMan version DataMan version DataMan version DataMan version

5 DataMan User Guide Introduction DataMan is a high performance data transfer and management tool that allows you to configure various data endpoints which act as the source or destination of transferred data. DataMan also monitors some of these endpoints to provide detailed stored data usage reporting. DataMan supports setting up filesystems and cloud storage endpoints and can transfer data between any of these endpoints. This document serves as the User Guide for the DataMan product versions. The document also provides technical specifications of DataMan. DataMan DataMan is based on the Cycle telemetry engine operating on a nosql datastore. It has a pluggable architecture and makes use of components to support various operations. It provides a rich query interface. It also provides a rich alerts and reporting framework that can be used to create custom reports, or alerts, on the datastore information. Information such as user actions, scheduled data transfers, and collected metadata are available through the alerts and reporting framework. The DataMan component supports data transfer, and management, operations by operating on requests created by adding records to the datastore. These records contain both user information, and metadata, needed to perform the data transfer. All operations performed by the DataMan component are written to the datastore to provide an audit trail. This information is made available via a REST API, command line clients, and Event Viewer Interface. DataMan provides CycleServer plugins that allow management, archival, and retrieval of user data in Cloud Storage. DataMan makes it easy to manage backup, archive, and transfer data residing in various storage services and mediums. It supports Amazon's Glacier for archiving, and provides encryption for archived data with user-provided keys. Customizable plugins may compress or encrypt files before uploading them to Glacier. By default, AES Encryption is performed using a key specified by the user. A RESTful API is provided to perform various operations on encryption keys allowing users to store or retrieve encryption keys, and perform archival or retrieval of folders residing on the local file system. Once the compressed archive is uploaded then archive metadata, such as the key used for encryption, source location, and the location of a generated manifest file is stored in CycleServer. The manifest file contains file system specific information about all the files in the archive such as file ownership, permissions, creation and modification timestamps, etc. as obtained from the underlying file system. A command line client utilizing the REST API is also provided. 1

6 Technical Specifications Hardware Requirements We recommend running CycleServer with DataMan on a Linux machine with 4-8 cores, 8-16 Gb. of RAM, and at least 50 Gb. of free disk space prior to installation. For heavy use environments, at least 500 Gb. of free disk space is recommended for optimal performance. The amount of RAM required is proportional to the number of parallel streams used to transfer data. The recommended setup is for 8 parallel streams with a 64Gb. part size. Every additional stream needs an additional 1-2 Gb. of RAM. The disk space required by DataMan is a function of the largest file size to upload/download. The amount of free scratch space should be at least 3 times the largest file to upload/download. Software Requirements The required Java Runtime Environment is bundled with CycleServer. RHEL/CentOS 6.x and newer clones, Ubuntu and newer, and SLES 11 are supported platforms. Root access is required during installation for user creation and enabling automated startup. The cycle_server user and group must be created prior to installation if default user creation facilities of Linux are inadequate to create user accounts. Python 2.7 is needed for execution of the client. Ports In order to communicate with clients, CycleServer requires ports 8080 and 8443 to be open for clients to be able to connect. For Amazon Glacier transfers outgoing traffic to the Amazon Glacier endpoints must be allowed. For S3 transfers outgoing traffic to S3 endpoints must be allowed. For Microsoft Azure transfers outgoing traffic to Azure block storage endpoints must be allowed. For Google Cloud Storage transfers outgoing traffic to Google Could Storage must be allowed. Other Configuration CycleServer must have read access to the source file system for files to be uploaded to Glacier. Additionally, if data deletion on inventory is enabled then write access is required on the source file system. Special scripts required to query ACL values of files must adhere to ACL input format described in this user guide. Write access is needed to the file system where data will be restored. Furthermore, any administrative privileges required to perform ACL changes are required to restore ACLs on downloaded files. DataMan Installation 2

7 Install DataMan by unpacking the compressed archive provided then execute the install.sh script on Linux, or install.cmd on Windows. On Linux, the install script must be executed as root. DataMan is currently supported on Linux, Windows, and Mac OS X platforms. If DataMan is installed from a pre-packaged DataMan installation, proceed to License Installation. If the components are provided separately, proceed to the Component Installation step before configuring the DataMan installation from the web interface. Component Installation If DataMan is being installed on a preexisting CycleServer installation, then follow this step to install DataMan. Copy the file_sync component zip file provided to the $CS_HOME/components directory. Next copy the DataMan component zip file into the same location. Wait a few minutes for the components to be installed. Finish CycleServer installation by navigating your browser to the CycleServer web interface address provided at the end of the CycleServer installation. By default this is host>:8080 or License Installation At the end of the installation wizard, you will be taken to the apps page, where you can install a license to activate all the DataMan features. On the apps page, if you already have a license, paste the license text in the text box provided under the License Key tab as shown below. 3

8 If you do not already have a license, copy the nodeid shown and it to sales@cyclecomputing.com to receive a license. Upon receiving the license, paste the license text in the space provided on this page. Data Encryption DataMan encrypts each file to be archived post-compression using strong AES 256-bit encryption. The encryption is performed using the standard Java Cryptography Extensions (JCE) security libary provided with the Oracle Java Runtime Environment. The JCE library provides standards-based strong encryption tools that avoid the pitfalls of weak encryption keys. The JCE SecureRandom class ensures that cryptographically secure random keys are generated and used for encryption of data. Users can generate or import encryption keys and use them to encrypt data when uploading to Amazon Web Services Glacier, S3, Microsoft Azure Block Storage, or Google Cloud Storage. When uploading data, encryption is performed in parallel on fixed size chunks of data. The encryption keys are seeded with a platform-dependent random number generator. On Linux-based systems /dev/urandom is used to obtain the random bytes used in generating the encryption keys. On Microsoft Windows, the Microsoft Crypto API is used to request random numbers. The random numbers on both Linux and Windows are generated using entropy from various sources on the system such as clock ticks, network and disk I/O, key strokes, and thermal variations of hardware components, etc. The system blocks until enough entropy is available before returning the next set of random bytes thus ensuring high quality random data. This ensures that the random numbers are impossible to reproduce without physical access to the system and knowledge of the numerous physical events that went into generation of the random data. The JCE libraries provide flexibility to replace the built-in implementation of random number generation with an implementation that makes use of hardware randomization, if available. DataMan provides a way to export the encryption keys out of DataMan for backup purposes. Administrative users may specify a key backup script. The script can be run at scheduled intervals to backup keys to a secure location. The cycle_server account and the key backup script must be secured in such a way that will not lead to a security breach. Creating Endpoints Once the license is installed, click on the Data link on the top menu and select Transfer Manager to proceed to the DataMan home page. The first time the page is opened, it will start with the "Add Endpoint" forms on both the left and right pane as shown below. 4

9 The left pane automatically defaults to the localhost endpoint, which is used to view and operate on data residing on the DataMan host. The Local Host check box must be enabled to configure endpoints on the DataMan host. The Allowed Paths text box allows you to add paths that DataMan can browse, retrieve, and store data to. This is set to /tmp by default and should be changed to point to the data staging area on the DataMan host. If only cloud storage operations will be performed and no data will be downloaded, the defaults can be accepted. Once you click "Save", the localhost endpoint named DataMan will be created and the file browser pane will show the allowed paths: 5

10 The right pane can be used to similarly add endpoints of different types. The configuration options for the other endpoints are described in the following sections. In order for Glacier inventory jobs to work then one must create another local endpoint called "localhost". Select the Add Endpoint option to create an endpoint and then give it the name localhost. Adding an AWS S3 Endpoint In order to add an AWS S3 endpoint, use the endpoint form to select the Amazon S3 Endpoint type, pick or enter a new set of AWS credentials, and select the bucket to use from the drop down menu. Please note that for endpoint creation to work from the endpoint creation form, the credentials must have list buckets privileges for Amazon S3. Adding a Microsoft Azure Endpoint In order to add an Microsoft Azure endpoint, use the endpoint form to select the Windows Azure Block Storage type. Pick or enter a new set of Azure credentials, enter the account name, and the name of the container to use. Adding a Google Cloud Platform Endpoint In order to add a Google Cloud Platform endpoint, use the endpoint form to select the Google Cloud Storage type, pick or enter a new set of GCP credentials, and enter a bucket name to use. Adding an SSH Host Endpoint In order to add an ssh host endpoint, use the endpoint form to select Host type. Name the endpoint and make sure the Local checkox is unchecked. Also fill in the account name to login as and select the SSH Private Key that will be used to connect to the host from the credential dropdown as shown below. You can use the Advanced button to provide specific SSH options in the List format if they are needed to connect to the host over ssh. 6

11 Adding Credentials Credentials can be added either from the "Add Endpoint" form by clicking the plus sign next to the "Credential" drop-down, or by navigating to the "Credential" type (organized under "Other" category heading) by browsing to <dataman-url>:<dataman-port>/types and selecting "All" from the component selection menu. Creating AWS Credentials In order to add AWS credentials, click the green plus sign next to the "Credential" drop-down in the "Add Endpoint" window. This opens the "Create Credential" form. Add your AWS Access Key and Secret Key, select your default region, and choose a descriptive name. Once entered then one can confirm the validity of credentials with the "Test" button. Click "Save" to save your credentials. One may now use the credentials to create a Glacier or S3 endpoint. Creating GCP Credentials 7

12 In order to add GCP credentials, click the green plus sign next to the "Credential" drop-down in the "Add Endpoint" window. This opens the "Create Credential" form. Enter a descriptive name for the credential, such as "My GCP Account". Add your GCP and private key. DataMan currently only supports Google Service accounts. Service accounts must be created on the Google Cloud web console. The Client and Private key are tied to the Service account and can be retrieved from the web console. Once entered, one may test the validity of credentials with the "Test" button. Click "Save" to save your credentials. One may now use the credentials to create a Google Cloud Storage endpoint. Creating Azure Credentials In order To add Azure credentials, click the green plus sign next to the "Credential" drop-down in the "Add Endpoint" window. This opens the "Create Credential" form. Create a descriptive name for the credential, such as "My Azure Account". Add the access key and click "Save". One may now use the credentials to create an Azure Block Storage endpoint. Transferring Files Data is transferred between endpoints by using the data transfer buttons located between the left and right endpoint browser panes. The simple transfer button can be used to start an immediate transfer from one endpoint to another. It should be noted that an immediate transfer simply submits the transfer for execution as soon as a worker is available to perform the transfer. The number of workers can be configured by editing the Queue records and changing the workers attribute to the desired number of workers. The default setting is 1 and is recommended for most purposes. Scheduled Transfer In order to setup a scheduled transfer, use the file browsers for each endpoint to find the desired source and destination. Click the arrow with clock transfer button to open transfer options window. From the Transfer Type drop-down, select "Scheduled". This will display the basic scheduling options. Expanding the Advanced menu will display the Encryption Key field. Select an key from the drop-down or click the green plus sign to create a key. Recurring Transfer In order to setup a recurring transfer, use the file browsers for each endpoint to find the desired source and destination. Click the arrow with clock transfer button to open the transfer options window. From the Transfer Type drop-down, select "Recurring". This will display the basic recurring transfer options in a drop-down menu. Schedules can be specified either by 8

13 selecting the various GUI options for daily and weekly transfers or by using the Interval mode, which schedules the transfer to be recurring after the specified Interval has elapsed. Intervals can be specified in seconds, minutes, or hours. Valid values for Interval in seconds, or minutes, is 1 through 60. Expanding the Advanced menu will display the Encryption Key field. Select any key from the drop-down or click the green plus sign to create a key. Monitoring Transfer Details When an immediate transfer is initiated or a scheduled or recurring transfer begins, transfer details are displayed in the bottom pane of the Transfer Manager page. One may need to click the refresh button for an accurate transfer progress reading. Double-clicking on a transfer displays further transfer details. Clicking on the Scheduled Transfers tab displays a queue of future transfers that have already been scheduled. 9

14 Double-clicking a queued transfer displays further details about the scheduled transfer. Transfer Agents It is possible to use external hosts as transfer agents. The external hosts split transfers by directory and then use multiple hosts to perform the transfers. To create a transfer agent, go to the Data browser and select Transfer Agent and click "Create" on the data table menu. 10

15 The transfer agent form is identical to a Host Endpoint form and takes SSH credentials to connect to the host and requires a host name and a user to connect as. The Managed Endpoints attribute accepts a list of endpoints that the Agent is allowed to operate on. This attribute is relevant when a shared filesystem is mounted across multiple nodes and these nodes are used as transfer agents to transfer data to and from such a shared file system. The endpoint list can contain Cloud storage endpoints and FileSystem Endpoints. Please note that FileSystem endpoints are different from Host Endpoints and only take the PathMappings attribute, which must be the mountpoint to the shared file system and must be the same across all nodes that will act as a transfer agent for this endpoint. DataMan Settings In order to tune DataMan transfers, a number of settings can be adjusted. Navigate to the settings page by clicking on the user drop-down menu on the top right of the screen and selecting Settings. 11

16 On the settings page, double clicking on the DataMan entry displays the DataMan settings form as shown below. 12

17 13

18 Proxy Server Configuration In order to use a proxy server for AWS operations, check the Use Proxy checkbox and enter the details of the proxy server in the Proxy Server, Proxy Port, Proxy Username, and Proxy Password textboxes. Default: Unchecked. SSL for AWS Operations The Enable SSL checkbox enables the use of https when connecting to the AWS Storage endpoints. It is enabled by default and must be unchecked to fall back to using http. Although not recommended it may be needed if there are too many "Peer not authenticated" or HTTP connection errors during transfers. Default: Enabled. Glacier Part Retry The Glacier Part Retry field indicates the number of times the upload of an individual part is retried before the upload is declared a failure. Default value: 5. Server Side Encryption The S3 encryption checkbox is used to enable server side encryption when transferring data to S3. Note that this is not the same as using encryption via DataMan, which pre-encrypts the data before transferring it to S3 or Glacier. The Server Side Encryption enables data to be encrypted using AWS managed keys when at rest in the S3 bucket. Default: Unchecked. Skip SSE Check If unchecked, when performing a sync, DataMan will check to ensure destination objects recieve server side encryption. When checked, DataMan will skip performing the encryption check. Default: Unchecked. S3 Overwrite When checked, if an object is found with the same key as an object you are trying to upload, the key is overwritten if the new object is different. Default: Checked. S3 Uploaders This settings modifies the number of threads used to transfer data to S3. Recommended value of this setting depends on your network bandwidth and the number of CPU cores available on the DataMan host. To prevent http connection errors during uploads, we recommend that this value not be set greater than 32. Default: 16. S3 Part Size This is the chunk size of multi-part uploads which are uploaded using a single thread to S3. The chunk size is a power of 2 and specified in bytes. Amazon S3 places an upper limit of on the number of parts, so the chunk size should be chosen appropriately based on the largest file to be uploaded. For instance, if the largest file to be uploaded is larger than 320GB and less than 640GB, the part size must be 64 MB ( ). If the largest file size is smaller than 320 GB, it is recommended that the part size be 32 MB. For very small files, a part size of 8 or 16 MB can be used. Please note that when converting from MB to bytes calculate the number of bytes as (size in MB) * 1024 * 1024 to obtain a power of 2 size in bytes. Default: 67,108,864. Change Checkers Change checkers are used to determine if a file has been changed and needs to be copied from one S3 bucket to another. Change checkers perform HEAD operations on S3 objects to retrieve metadata that can be used to determine if the object has changed. DataMan compares the ETag attribute on the S3 object to the MD5 checksum of 14

19 the file disk or to the ETag of the source S3 object. If the ETag is not an MD5 checksum, as is the case with multi-part uploads, DataMan looks for the metadata attribute "user-md5", which is set by DataMan to be the MD5 checksum of the file. If this attribute is not found or is different, DataMan overwrites the object with the new one. For host to S3 uploads it will also set the "user-md5" metadata attribute. Amazon S3 automatically prefixes the metadata attribute with "x-amz-meta-", so the attribute on the S3 console will appear as "x-amz-meta-user-md5". If objects have been uploaded to S3 using multi-part uploads, the metadata attribute should be added to the objects for reliable syncing, without relying on the ETag. Default: 8. Skip Check Force copies all objects without checking for their existence at the destination. Default: Unchecked. S3 Copy Number of parallel copy requests S3 will send to Amazon. Only applies to S3 to S3 copies. Default: 8. Copy Part Size The size of each part in a multipart S3 copy operation. Default: 67,108,864. S3 Multipart Maximum object size before multipart copy is used. Default: 104,857,600. Compute MD5 When checked, the checksum of a file will be computed before uploading to S3 so a change comparison can be made during subsequent uploads. Required for syncs to work properly. Default: Checked. S3 Versions Shows version number of your bucket in the data browser if version numbers are enabled in the bucket. Default: Unchecked. Glacier Part Size The size of each part in a multipart Glacier copy operation. Default: 67,108,864. Glacier Uploaders Number of parallel copy requests Glacier will send to Amazon. Only applies to Glacier to Glacier copies. Default: 8. Delete data after Delete locally stored data after it is uploaded to Glacier. Default: Unchecked. Encrypt Data Encrypt data before uploading to Glacier. Default: Checked. Compute MD5 When checked, the checksum of a file will be computed before uploading to Glacier so a change comparison can be made during subsequent uploads. Required for syncs to work properly. Default: Checked. CLI Installation Unzip the CLI bundle to the location of your choice. Place the dataman binary found in the bin folder on your PATH for convenient access to dataman commands. 15

20 CLI Initialization Intialize dataman using the dataman initialize command. Provide your username and password when prompted: $ dataman initialize Welcome to DataMan! CycleServer URL: [ CycleServer username: [akulshrestha] admin CycleServer password: Generating CycleServer key... DataMan configuration stored in /Users/akulshrestha/.cycle/config.ini Store Storage Service Credentials Use credential create to store the Glacier AWS secret key and access key to be used by CycleServer: dataman credential create -t AWS -n <credential name> -a <accesskey> -s <secretkey> Create Endpoints The dataman endpoint create command is used to create Endpoints for AWS Glacier vaults and AWS S3 buckets. To configure DataMan to use a AWS Glacier vault as an endpoint: $ dataman endpoint create -t Glacier -n <endpoint name> -r <region> -b <bucket/vault name> DataMan Endpoint Created. For S3 buckets: $ dataman endpoint create -t S3 -n <endpoint name> -r <region> -b <bucket/vault name> DataMan Endpoint Created. The DataMan endpoint is now configured and ready to use. Command Line Client Users will interact with DataMan using the command line client called dataman. A help screen is provided by invoking dataman help which lists the various commands and their options. (Note: On Windows \ must be escaped in paths - C:\\manifests): $ dataman help Initialization 16

21 The initialize command is used to perform one time configuration of the command line client enabling it to securely communicate with the CycleServer instance hosting the Cycle DataMan REST API.: $ dataman initialize List Encryption Keys The encryptionkey list command shows all keys stored in CycleServer.: $ dataman encryptionkey list KeyName:Symmetric Key examplekey _1: c d8a476f203a0772c7185eb7b6a1795bf3586ac3d5d Create Encryption Keys The encryptionkey create command allows users to store, or create, a key in CycleServer to encrypt and decrypt files. If no key is provided, then a key is automatically generated and stored. The keyname is used as a handle to identify the key to be used with other commands.: $ dataman encryptionkey create <keyname> [key] Create a Storage Endpoint Storage endpoints refer to data sources and destinations which may include traditional file systems, file storage services, archival services etc.: $ dataman endpoint create -t Glacier -n <endpoint name> -r <region> -b <vault name> DataMan Endpoint Created. $ dataman endpoint create -t S3 -n <endpoint name> -r <region> -b <bucket name> DataMan Endpoint Created. List Endpoints Lists all available endpoints: $ dataman endpoint list Name : Vault : Region glacier : examplevault1 : us-east-1 Lists endpoints by type: 17

22 $ dataman endpoint list -t S3 Name : Bucket s3example : com.cyclecomputing.software Create and Upload a AWS Glacier Archive The transfer command may create a compressed and encrypted archive and uploads it to Amazon Glacier.: $ dataman transfer -t GLACIERTRANSFERTYPE -c <vault name> -k <key name> -s <source endpoint> -d <destination endpoint> List Archives The archive list command is used to query CycleServer for the archives that have been uploaded to Glacier.: $ dataman archive list Name : examplearchive _6 : examplearchive _5 : examplearchive _7 : Status : Key Used Inventoried : testkey-1 Inventoried : testkey-1 Inventoried : testkey-1 List Contents of an archive To see the contents of an archive: $ dataman archive list arch _1 Wed Mar 6 13:00: arch _3.manifest Mon Feb 4 22:48: b6e480a a5-c17e2b manifest Wed Mar 6 16:47: arch _9.manifest Mon Feb 4 21:21: file1.dat Mon Feb 4 21:36: file3.dat Fri Feb 15 12:45: arch _1.manifest Mon Feb 4 22:13: e7a38c79-275a b b manifest Mon Feb 4 21:37: file4.dat Tue Feb 26 12:51: arch _4.manifest Mon Feb 4 21:35: file2.dat Fri Mar 8 12:19: arch _1.manifest Tue Feb 5 00:13: arch _1.manifest Wed Mar 6 13:55: arch _6.manifest total Creating a Recurring transfer 18

23 A recurring sync can be created to backup new or changed files to Glacier. MD5 checksums of files are used to determine if they are changed. The -c option can be used to specify an interval in seconds or a Quartz cron schedule string: $ dataman recurringsync create -n sync1 -c 60 -t GLACIERTRANSFERTYPE -s winsync1 -d wintest1 -l "C:\testData" Schedule strings are of the format "ss mm hh dd mon dayofweek". Examples of schedule strings. 1. Every 20 seconds: "0/20 * * * *?" 2. Every other minute, starting at 15 seconds past the minute: "15 0/2 * * *?" 3. Every other minute, between 8am and 5pm: "0 0/ * *?" 4. At 10am on the 1st and 15th days of the month: "0 0 10am 1,15 *?" 5. Every 30 seconds on Weekends (Saturday and Sunday): "0,30 * *? * SAT,SUN" Downloading an Archive The transfer command may request an archive to be downloaded. When the request has been fulfilled by Amazon Glacier, decrypts, uncompresses, and explodes it to the provided location. Only inventoried archives from inventoried vaults can be downloaded.: $ dataman transfer -t GLACIERTRANSFERTYPE -c <vault name> -k <key name> -l /path/to/explode/to -s <source endpoint> -d <local endpoint> $ dataman transfer -t GLACIERTRANSFERTYPE -c <vault name> -k <key name> -l /path/to/explode/to -p filename -s <source endpoint> -d <local endpoint> Searching for files The dataman search command can be used to locate files within archives or archives with specified tags.: $ dataman search -f file1.dat arch_unenc_nomd5_512parts8mb_ _5#11: file1.dat arch_unenc_nomd5_256parts_ _4#10: file1.dat arch_unenc_nomd5_64parts_ _2#9: file1.dat arch_unenc_nomd5_64parts_ _1#8: file1.dat arch_unenc_32parts_ _4#7: file1.dat arch_unenc_ _3#6: file1.dat arch _3#5: file1.dat arch _1#3: file1.dat= $ dataman search -t project=demo arch_unenc_nomd5_256parts_ _4#10: project=demo,owner=demouser Job Status 19

24 To show the status of a download request or an inventory request: $ dataman status arch _1 : archivedownload : COMPLETED : J0VEbh68gKA_AD --- : inventory : COMPLETED : LbDi-mSHh8V56L --- : inventory : COMPLETED : E8gP3B_sHdte6n --- : inventory : EXPIRED : 6c5qHIGBaaHeJU arch _1 : archivedownload : COMPLETED : wptsffd7xtiirs REST API Specification DataMan provides a RESTful interface for configuration, key management, and data transfer operations. The configuration operations interact with the CycleServer datastore to store configuration items used by DataMan to perform data transfers. The data transfer operations store transfer requests in the datastore which are then scheduled for execution. Key Management List Keys GET /ads/datamanager.encryptionkey&format=json Store Key Autogenerated Key: GET /datamanager/rest?o=store&keyname=name Specified Key: GET /datamanager/rest?o=store&keyname=name&key=bda12deadbeef431 Archive Operations List Archives GET /ads/datamanager.archive List Archive Contents GET /glacier/rest?o=listarchive&a=archivename Upload an archive 20

25 GET /glacier/rest?o=upload&k=keyname&a=archivename&v=vault&p=/upload/from&t=tags Download an archive or a file from an archive GET /glacier/rest?o=download&k=keyname&a=archivename&v=vault&d=/download/to GET /glacier/rest?o=download&k=keyname&a=archivename&v=vault&d=/download/to&f=file Scaling DataMan If the aggregate bandwidth available for Glacier uploads is greater than the sustained bandwidth usage of a single DataMan node, multiple nodes should be used. In order to determine the bandwidth usage efficiency, the following factors must be taken into consideration: Data Preprocessing When dealing with large volumes of data, it must often be pre-processed to package and optionally compress the data. This can be a time consuming process and the throughput depends on resources available on the node that pre-processes the data. For compression of large files, parallel compression tools such as pigz are recommended on multi-core machines. If the upload speed to Glacier is faster than the aggregate bandwidth of the preprocessing pipeline, it is recommended that more than one node be used to preprocess data to be presented for upload to DataMan. The goal of the pre-process pipeline should be to always have one additional dataset available for upload before the previous upload finishes. Upload Bandwidth Single DataMan nodes will often peak out below the net available bandwidth to the Archival Service endpoint. This can be due to various reasons, such as interface speeds, limited number of cores on the node or limited I/O bandwidth. If MD5 checksum computation is enabled, additional time is needed to compute the checksum, before data is uploaded. Due to the nature of the checksum algorithm, it runs on a single core and is not parallelized. Hence, for large files this becomes a bottleneck and reduces the net upload bandwidth to Glacier. If prepackaging of data creates a few large files, the checksum computation is a significant overhead. In this case, multiple DataMan nodes must be used to make best use of the bandwidth. In the perfect scenario at least one of the nodes must be uploading data while the other is computing checksums or waiting on data to become available from the pre-processing pipeline. 21

26 The DataMan nodes should be individually tuned to determine the right upload part size and concurrency required to achieve the highest upload speeds possible for that node. If the peak bandwidth is less than 80% of the total bandwidth available to the archival service endpoint, then additional DataMan nodes are needed to make full use of the available bandwidth. Please note that this is only valid on a congestion-free network. For a network with substantial traffic, adding more DataMan nodes may not offer any benefit. On congested networks, the network traffic patterns must be evaluated before making any recommendation. Additional DataMan nodes can also be used to maximize bandwidth use if the MD5 checksum computation takes more than half the time required to upload the dataset. When adding another node it is beneficial to start the next upload on the new DataMan node once the first node has started uploading data to the archival service. When data sets are of similar or comparable size, this can be done by putting the uploads on a schedule that allows one node to be computing the checksum while the other is uploading data. Transfer Listeners DataMan supports writing plugins that are executed based on transfer events such as individual file transfer completion or failure or entire transfer completion or failure. Listener plugins must set the Implements attribute to be DataTransferStateListener and can subscribe to the states that they are interested in receiving events for. For example, if individual file upload events are of interest, the plugin must subscribe to the events from the "Transferring" state. Other attributes pertaining to the plugin can be set on the DataTransferStateListener attribute as a nested record. Currently supported attributes are: Attribute States (Required) FunctionName (Optional) Description The list of states that the listener will listen to. These can be either Transferring, Completed, or Failed. The name of the plugin function to invoke upon receiving the listener event. It defaults to handletransferevent. The plugin function is provided with a record containing the details of the transfer event, such as FileName transferred, number of bytes transferred, number of files transferred, etc. An example plugin configuration is as follows: Implements = DataTransferStateListener DataTransferStateListener := [States={"Transferring","Completed"}; FunctionName="handleTransferEvent"] An example of the plugin that implements the handletransferevent method is as follows: 22

27 from application import logger def handletransferevent(record): logger.info("triggerring transfer event handler for %s" % record.getasstring("filename")) An example of the record that is sent to plugin method as an argument is as follows: [ Bytes=1024.0; FileCount=1.0; FileName="dir1/file1001.txt"; State="Transferring" ] Release Notes DataMan version New Features: Dataman can now sign URLs for files in Azure Storage. Resolved Issues: AWS S3 signed URL expiration times are now validated properly to 7 days or less. Symlink access is now validated consistently for file browser and file transfers. Copying a blob from one storage account in Azure to another no longer fails with authentication errors. Fixed an issue where the remote host endpoint UI wouldn't save the credential drop down. Initial endpoint configuration forms have been corrected for the standalone Dataman installation. Reduced memory/cpu usage when transferring larger files. Fixed an issue which would cause S3 uploads to fail sporadically. Fixed an issue where canceling a transfer could prevent future transfers from being processed. Canceling transfers is now more responsive. Pogo can now be terminated promptly with ctrl+c. DataMan version New Features: Amazon S3 transfers to the China region (cn-north-1) are now supported. 23

28 Amazon S3 Server Side Encryption can now be configured separately on each endpoint. Resolved Issues: Pogo no longer fails to upload to Amazon S3 in Windows. Updating storage buckets in the background is now much faster and requires far fewer list operations. Transfers to/from the filesystem used too much memory. The global Amazon S3 Server Side Encryption setting had no effect. DataMan version Resolved Issues: Transfers from remote hosts to cloud storage endpoints did not work. Allowed paths containing "/" caused directory navigation issues on local host endpoints. Failures transferring to local host no longer display "Unknown Cause". Transfer requests are now validated even when allowed paths are defined on an endpoint. Transfers to or from the root path on endpoints did not work. Remote host recursive transfers did not create the right destination path. Breadcrumb links on the data navigator were broken. The "prior to" time filtering option on the data navigator did not work correctly. Fixed various issues when using Google credentials with a client ending in iam.gserviceaccount.com. DataMan version New Features: New setting to adjust the transfer thread count and part size. Resolved Issues: Transfers to the root directory of a host endpoint work properly again. 24

29 Remote host endpoints did not support multiple allowed paths. Private key credentials were not validated properly. Data encryption key dropdown did not allow a selection of "None". Active recurring syncs no longer prevent unrelated recurring syncs from running. Pogo config does not ignore existing instance creds. Transferring to the root directory of a host endpoint was not validated against the allowed paths. Certificate-based credentials may now be used when creating Azure Storage endpoints. Endpoint browsers wouldn't always refresh after transfers completed. Update documentation to reflect dataman CLI help output. Transfer performance dashboard charts now display correctly again. DataMan version New Features: Dataman now supports specifying the hostname/ip address of the S3 service. Resolved Issues: The Advanced section of the form did not appear when creating a new s3 endpoint. Pogo previously would emit an unnecessary unicode warning message on Windows installations. S3 encrypted uploads would fail with the message: "Unkown Cause". Running pogo config in Amazon instances would fail. Pogo would fail if transferring from one S3 account to another. DataMan version DataMan version New Features: Updated Java AWS SDK to version Google endpoint creation form now shows a dropdown of existing buckets and allows bucket creation. 25

30 Pogo is now capable of printing a list of buckets in Google Cloud Storage. pogo ls gs://. Resolved Issues: DataMan transfers using Pogo can now be cancelled using the cancel button. The 'Transfers' table now automatically refreshes more quickly and reliably. Multi-node transfers are reported under a single transfer instead of one for each node. Transfers of 0 byte files now correctly report their status when complete. Pogo wasn't autocorrecting region when default_region setting was present for S3 transfers. Transfers no longer show 0 for speed and uploaded bytes until metrics are available. Transfers are correctly acted upon after a DataMan restart. Pogo couldn't transfer files larger than 2GB on Windows. Fixed many unicode issues in Pogo. Pogo config would fail if a cyclecloud configuration wasn't present. Pogo list command with bad credentials would print unfriendly messages to the console. DataMan version New Features: DataMan now uses a more secure two-tier encryption scheme for encrypting data to AWS S3, Azure Blob Storage, and Google Cloud Storage. Pogo now supports optimized multipart S3 - S3 copies for files larger than 5 GB with MD5 set. Pogo can now encrypt/decrypt with the same encryption used in DataMan. Pogo transfers to/from S3 now support S3 "server-side encryption" if the endpoint is configured with server_side_encryption = True. Added support for scheduled remote transfers to Glacier. Added options for more fine-grained recurring transfers (hourly, every N seconds). CycleCloud nodes using recipe[dataman::worker] will auto-register themselves as Data.Workers for use as transfer agents. Pogo will automatically detect EC2 instance creds and use them for instances started with an Instance Role providing S3 access. Added the ability to test Google Cloud Credentials before saving them. 26

31 Azure storage endpoints can now use the Azure Management Certificate credentials. Resolved Issues: DataMan will no longer run out of heap space when syncing a file system with more than 5 million files. TotalSize is now correctly shown when no files are transferred in a sync. Fixed OOM errors when walking large directories. DataMan transfers on Windows no longer fail due to additional '/' at the beginning of the source path. DataMan version New Features: Improved Pogo performance in many situations. Pogo now supports direct copies if the source and destination are the same provider (with the exception of S3). Pogo batch deletions now supported for Google Cloud Storage. Signed URLs can now be generated for downloading objects in Amazon S3 Endpoints. FDT status reporting for ongoing transfers now includes instantaneous transfer rate. Transfers now report instantaneous wire speed instead of average over lifetime. DataMan supports a prototype implementation of multi-node transfer. Resolved Issues: Error conditions that occur during file browsing are now displayed in the file browser pane. Folder files created using the console are no longer shown in the data browser. The endpoint creation/editing form now correctly saves the Endpoint. Glacier Archive inventory status checking did not work properly. Remote Glacier upload part sizes were too small by default. EndTime is now correctly reported after FDT transfers finish. DataMan version

32 New Features: Pogo is now multithreaded for improved performance. DataMan now uses Pogo to initiate S3 transfers on remote hosts. Host to Host transfers using FDT are now supported. DataMan version New Features: Google Storage is now supported in DataMan using Service Account credentials. Active transfers can now be cancelled. Cross account S3 to S3 transfers are now supported. Pogo now has support for Google Cloud Storage. Pogo will now continue multi-file operations should a single file fail. Resolved Issues: Failed part download from AWS S3 caused file corruption. DataMan version New Features: Support transfer listeners on progress and completion or failure. Support list based comparisons for S3 to S3 copy operations. Transfer table auto-refreshes quickly (and faster when there are outstanding transfer requests). DataMan GUI now has a single tab to view all transfer activity. Performance dashboard now uses common transfer schema to report performance metrics. Resolved Issues: Pogo: Recursive list properly handles AWS folder objects. 28