DataMan. version 6.5.4

Transcription

1 DataMan version 6.5.4

2 Contents DataMan User Guide 1 Introduction 1 DataMan 1 Technical Specifications 1 Hardware Requirements 1 Software Requirements 2 Ports 2 DataMan Installation 2 Component Installation 2 License Installation 2 Data Encryption 3 Endpoint Creation 4 Endpoint Types 4 Configuring a Local Endpoint 4 Adding Cloud Provider Credentials 6 Adding Endpoints 6 Transferring Data 9 On-Demand Transfers 9 Scheduled Transfers 9 Adding Credentials 10 Creating AWS Credentials 10 Creating GCP Credentials 11 Creating Azure Credentials 11 Transferring Files 11 Scheduled Transfer 11 Recurring Transfer 11 Monitoring Transfer Details 12

3 Transfer Listeners 14 Release Notes 15 DataMan version DataMan version DataMan version DataMan version DataMan version DataMan version DataMan version

4

5 DataMan User Guide Introduction DataMan is a high performance data transfer and management tool that allows you to configure various data endpoints which act as the source or destination of transferred data. DataMan also monitors some of these endpoints to provide detailed stored data usage reporting. DataMan supports setting up filesystems and cloud storage endpoints and can transfer data between any of these endpoints. This document serves as the User Guide for the DataMan product versions. The document also provides technical specifications of DataMan. DataMan DataMan is based on the Cycle telemetry engine operating on a nosql datastore. It has a pluggable architecture and makes use of components to support various operations. It provides a rich query interface. It also provides a rich alerts and reporting framework that can be used to create custom reports, or alerts, on the datastore information. Information such as user actions, scheduled data transfers, and collected metadata are available through the alerts and reporting framework. The DataMan component supports data transfer, and management, operations by operating on requests created by adding records to the datastore. These records contain both user information, and metadata, needed to perform the data transfer. All operations performed by the DataMan component are written to the datastore to provide an audit trail. This information is made available via a REST API, command line clients, and Event Viewer Interface. DataMan provides CycleServer plugins that allow management, archival, and retrieval of user data in Cloud Storage. DataMan makes it easy to manage backup, archive, and transfer data residing in various storage services and mediums. It provides encryption for archived data with user-provided keys. By default, AES Encryption is performed using a key specified by the user. A RESTful API is provided to perform various operations on encryption keys allowing users to store or retrieve encryption keys, and perform archival or retrieval of folders residing on the local file system. A command line client utilizing the REST API is also provided. Technical Specifications Hardware Requirements We recommend running CycleServer with DataMan on a Linux machine with 4-8 cores, 8-16 Gb. of RAM, and at least 50 Gb. of free disk space prior to installation. For heavy use environments, at least 500 Gb. of free disk space is recommended for optimal performance. 1

6 The amount of RAM required is proportional to the number of parallel streams used to transfer data. The recommended setup is for 16 parallel streams with a 16MB part size. Software Requirements The required Java Runtime Environment is bundled with CycleServer. RHEL/CentOS 6.x and newer clones, Ubuntu and newer, and SLES 11 are supported platforms. Root access is required during installation for user creation and enabling automated startup. The cycle_server user and group must be created prior to installation if default user creation facilities of Linux are inadequate to create user accounts. Python 2.7 is needed for execution of the client. Ports In order to communicate with clients, CycleServer requires ports 8080 and 8443 to be open for clients to be able to connect. For S3 transfers outgoing traffic to S3 endpoints must be allowed. For Microsoft Azure transfers outgoing traffic to Azure block storage endpoints must be allowed. For Google Cloud Storage transfers outgoing traffic to Google Could Storage must be allowed. DataMan Installation Install DataMan by unpacking the compressed archive provided then execute the install.sh script on Linux, or install.cmd on Windows. On Linux, the install script must be executed as root. DataMan is currently supported on Linux, Windows, and Mac OS X platforms. If DataMan is installed from a pre-packaged DataMan installation, proceed to License Installation. If the components are provided separately, proceed to the Component Installation step before configuring the DataMan installation from the web interface. Component Installation If DataMan is being installed on a preexisting CycleServer installation, then follow this step to install DataMan. Copy the file_sync component zip file provided to the $CS_HOME/components directory. Next copy the DataMan component zip file into the same location. Wait a few minutes for the components to be installed. Finish CycleServer installation by navigating your browser to the CycleServer web interface address provided at the end of the CycleServer installation. By default this is host>:8080 or License Installation At the end of the installation wizard, you will be taken to the apps page, where you can install a license to activate all the DataMan features. On the apps page, if you already have a 2

7 license, paste the license text in the text box provided under the License Key tab as shown below. If you do not already have a license, copy the nodeid shown and it to sales@cyclecomputing.com to receive a license. Upon receiving the license, paste the license text in the space provided on this page. Data Encryption DataMan can encrypt files using strong AES 256-bit encryption. The encryption is performed using the standard Java Cryptography Architecture (JCA) security library provided with the Oracle Java Runtime Environment. The JCA library provides standards-based strong encryption tools that avoid the pitfalls of weak encryption. The SecureRandom class ensures that cryptographically secure random keys are generated and used for the encryption of data. When uploading data, encryption is performed in parallel as to minimize the impact on transfer performance. Users can generate or import encryption keys and use them to encrypt data when uploading to Amazon S3, Microsoft Azure Block Storage, or Google Cloud Storage. DataMan provides a way to export the encryption keys out of DataMan for backup purposes. Administrative users may specify a key backup script. The script can be run at scheduled intervals to backup keys to a secure location. The cycle_server account and the key backup script must be secured in such a way that will not lead to a security breach. 3

8 Endpoint Creation Endpoints are logical connections to storage providers and can be any of the following: Endpoint Types Type Amazon S3 Azure Block Storage Google Cloud Storage Host Description Amazon's Simple Storage Service Microsoft Azure's storage service Google Cloud's object storage service Data residing on servers accessible to the CycleCloud Data Manager. These can be local directories or other mounted file systems. To view and manage endpoints, select Data then Transfer Manager from the CycleCloud menu bar. Configuring a Local Endpoint On the left side of the Transfer Manager screen, you can see that an initial local endpoint has been created by the installer, and is configured to use the /tmp directory. Click "Save" to create this endpoint. Adding Another Local Directory To add another directory to the local endpoint, click the endpoint name next to the gear in the top left, and select Edit Endpoint. 4

9 Click the + to generate a new line, then enter the path. Click Save to continue. Note Any directories in the "Allowed Paths" must be accessible by the user running CycleCloud. Host Endpoint Attributes Attribute Name Type Allowed Paths Encryption Key Local Hostname Description Name of the endpoint The type of this endpoint (see table above) A list of system paths accessible by this endpoint The encryption key used to encrypt all files transferred to this endpoint When checked, indicates that folders are local to the host Hostname for host endpoints 5

10 SSH User Credential SSH Port Pogo Path Remote login name The SSH keypair to use when logging into a remote host Port for connections via SSH The path to the Pogo executable on the remote host Adding Cloud Provider Credentials You will need to add credentials for each of the cloud provider accounts you wish to use with the CycleCloud Data Manager. Click on the host name in the upper left and select Add Endpoint Click on the green plus sign next to Credential, which will open a new window called Create Credential Enter a descriptive name for the credential Select the appropriate Type: AWS: Amazon Azure: Azure Access Key Google: Google Cloud Private Key Enter your Access and Secret Keys Select your default region Confirm your entries by clicking Test Click "Save" to save your credentials. You can now use the credentials to create an Amazon S3, Azure Block Storage, or Google Cloud Storage endpoint. Adding Endpoints Adding an AWS S3 Endpoint After you've created AWS credentials, you can set up an AWS S3 endpoint. From the Add Endpoint window, enter a descriptive name in the "Name" field. Select "Amazon S3" from the "Type" drop-down, then select the credentials you created above. Next, select the desired bucket from the drop-down or click the green plus sign to create a new bucket. Click "Save" to save the endpoint. 6

11 Note If you do not have an S3 bucket created, you can click the green plus button next to the bucket dropdown to create one. Adding a Microsoft Azure Endpoint Once you've added Azure credentials, you can set up an Azure Storage endpoint. From the New Endpoint window, enter a descriptive Name Select Azure Storage as the Type Enter the Allowed Path. Use the + to add additional paths if needed. Select the Encryption Key Choose the appropriate Azure Credential Select the Storage Account and Storage Container via the dropdown menus Click Save to add your Azure endpoint 7

12 Adding a GC Endpoint Once you've added Google Cloud credentials, you can set up a Google Cloud Storage endpoint. From the Add Endpoint window, enter a descriptive name in the "Name" field. Select "Google Cloud Storage" from the "Type" drop-down. Select the GC credentials you created earlier. Select a bucket from the drop-down or click the green plus sign to create a new bucket. Click "Save" to save the endpoint. Adding a Remote Host Endpoint For server-to-server transfers, you can create another "Host" type endpoint. However, configuring a remote host endpoint is outside of the scope of this guide. For more information, please see the CycleCloud User Guide. 8

13 Transferring Data CycleCloud supports both on-demand and scheduled data transfers between endpoints. Both of these modes are managed via the Transfer Manager page, which is accessed from the Data dropdown menu. On-Demand Transfers To begin an on-demand transfer, select the appropriate endpoint on the left side of the page. On the right, select the cloud provider endpoint you created. Use the file browsers for each endpoint to find the data to transfer and the desired destination. Click the right arrow to upload to cloud storage, or the left arrow to download to your endpoint. Note If the source is a directory, Data Manager will recursively transfer all files and subdirectories. Scheduled Transfers Scheduled transfers are done the same way as on-demand transfers, but use the clock arrows instead. This will allow you to transfer files: Immediate: transfer right away Scheduled: transfer once, at a set time Recurring: transfer on a regular schedule If you wish to encrypt your data, you can import or create a new encryption key in the Advanced section of the Scheduler. Once your scheduled or recurring transfer has been created, click Go to schedule it. To view, edit, or delete your scheduled transfer(s), click the Scheduled Transfers tab at the bottom of the page. 9

14 Adding Credentials Credentials can be added either from the "Add Endpoint" form by clicking the plus sign next to the "Credential" drop-down, or by navigating to the "Credential" type (organized under "Other" category heading) by browsing to <dataman-url>:<dataman-port>/types and selecting "All" from the component selection menu. Creating AWS Credentials In order to add AWS credentials, click the green plus sign next to the "Credential" drop-down in the "Add Endpoint" window. This opens the "Create Credential" form. Add your AWS Access Key and Secret Key, select your default region, and choose a descriptive name. Once entered then one can confirm the validity of credentials with the "Test" button. Click "Save" to save your credentials. One may now use the credentials to create an S3 endpoint. If you are want to restrict privileges to a minimum necessary set, this IAM policy is recommended: { } "Action": [ "s3:listallmybuckets" ], "Effect": "Allow", "Resource": "*" { "Action": [ "s3:abortmultipartupload", "s3:deleteobject", "s3:getobject", "s3:getobjectversion", "s3:putobject", "s3:listbucket", "s3:createbucket", "s3:deletebucket", "s3:getbucketlocation", "s3:listbucketmultipartuploads", "s3:listmultipartuploadparts" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::com.example.bucket", "arn:aws:s3:::com.example.bucket/*" 10

15 } ] Creating GCP Credentials In order to add GCP credentials, click the green plus sign next to the "Credential" drop-down in the "Add Endpoint" window. This opens the "Create Credential" form. Enter a descriptive name for the credential, such as "My GCP Account". Add your GCP and private key. DataMan currently only supports Google Service accounts. Service accounts must be created on the Google Cloud web console. The Client and Private key are tied to the Service account and can be retrieved from the web console. Once entered, one may test the validity of credentials with the "Test" button. Click "Save" to save your credentials. One may now use the credentials to create a Google Cloud Storage endpoint. Creating Azure Credentials In order to add Azure credentials, click the green plus sign next to the "Credential" drop-down in the "Add Endpoint" window. This opens the "Create Credential" form. Create a descriptive name for the credential, such as "My Azure Account". Add the access key and click "Save". One may now use the credentials to create an Azure Block Storage endpoint. Transferring Files Data is transferred between endpoints by using the data transfer buttons located between the left and right endpoint browser panes. The simple transfer button can be used to start an immediate transfer from one endpoint to another. It should be noted that an immediate transfer simply submits the transfer for execution as soon as a worker is available to perform the transfer. The number of workers can be configured by editing the Queue records and changing the workers attribute to the desired number of workers. The default setting is 1 and is recommended for most purposes. Scheduled Transfer In order to setup a scheduled transfer, use the file browsers for each endpoint to find the desired source and destination. Click the arrow with clock transfer button to open transfer options window. From the Transfer Type drop-down, select "Scheduled". This will display the basic scheduling options. Expanding the Advanced menu will display the Encryption Key field. Select an key from the drop-down or click the green plus sign to create a key. Recurring Transfer 11

16 In order to setup a recurring transfer, use the file browsers for each endpoint to find the desired source and destination. Click the arrow with clock transfer button to open the transfer options window. From the Transfer Type drop-down, select "Recurring". This will display the basic recurring transfer options in a drop-down menu. Schedules can be specified either by selecting the various GUI options for daily and weekly transfers or by using the Interval mode, which schedules the transfer to be recurring after the specified Interval has elapsed. Intervals can be specified in seconds, minutes, or hours. Valid values for Interval in seconds, or minutes, is 1 through 60. Expanding the Advanced menu will display the Encryption Key field. Select any key from the drop-down or click the green plus sign to create a key. Monitoring Transfer Details When an immediate transfer is initiated or a scheduled or recurring transfer begins, transfer details are displayed in the bottom pane of the Transfer Manager page. One may need to click the refresh button for an accurate transfer progress reading. Double-clicking on a transfer displays further transfer details. 12

17 Clicking on the Scheduled Transfers tab displays a queue of future transfers that have already been scheduled. Double-clicking a queued transfer displays further details about the scheduled transfer. 13

18 Transfer Listeners DataMan supports writing plugins that are executed based on transfer events such as individual file transfer completion or failure or entire transfer completion or failure. Listener plugins must set the Implements attribute to be DataTransferStateListener and can subscribe to the states that they are interested in receiving events for. For example, if individual file upload events are of interest, the plugin must subscribe to the events from the "Transferring" state. Other attributes pertaining to the plugin can be set on the DataTransferStateListener attribute as a nested record. Currently supported attributes are: Attribute Description 14

19 States (Required) FunctionName (Optional) The list of states that the listener will listen to. These can be either Transferring, Completed, or Failed. The name of the plugin function to invoke upon receiving the listener event. It defaults to handletransferevent. The plugin function is provided with a record containing the details of the transfer event, such as FileName transferred, number of bytes transferred, number of files transferred, etc. An example plugin configuration is as follows: Implements = DataTransferStateListener DataTransferStateListener := [States={"Transferring","Completed"}; FunctionName="handleTransferEvent"] An example of the plugin that implements the handletransferevent method is as follows: from application import logger def handletransferevent(record): logger.info("triggerring transfer event handler for %s" % record.getasstring("filename")) An example of the record that is sent to plugin method as an argument is as follows: [ Bytes=1024.0; FileCount=1.0; FileName="dir1/file1001.txt"; State="Transferring" ] Release Notes DataMan version New Features: Azure endpoints now have auto-populated dropdowns for the Storage Account and Container fields. Added Monitor Limit field to endpoints, which specifies the maximum time the monitor should spend on an endpoint in a 24-hour period. Added Preserve field to endpoints, which will save an unmonitored endpoint's data records (which would otherwise be deleted). Resolved Issues: Endpoint Encryption Key field wouldn't display previously selected values or newly created keys. 15

20 The Data Navigator page would time out on large datasets and fail to display any data. DataMan version Resolved Issues: Adding/removing an endpoint would only update the one endpoint list that was interacted with. Pogo's config command would not create non-default ini files. Pogo's url command would not give an indication if there are no config sections present. Pogo would display inaccurate initial transfer rates. Pogo wasn't reporting a transfer rate when reporting transfer metrics to DataMan. Pogo would occasionally leave parts of multi-part transfers behind on GCS. Pogo would make too many requests when listing Azure storage accounts. Pogo config sections with conflicting matches attributes would not be indicated. DataMan version Resolved Issues: Empty files would cause transfers between cloud providers to block indefinitely. DataMan version New Features: S3 transfers can now be performed even with a minimal policy in place. Encryption keys are now hex encoded for easier portability. Endpoint forms now use standardized parameters across endpoint types. Resolved Issues: The "Transfer Manager" page did not work properly if there were any endpoints without a type defined. 16

21 Localhost endpoints with only a single Allowed Paths entry wouldn't skip directly to that entry. Pogo would print a stack-trace when a non-existent config file was specified. Fixed an issue that prevented users from selecting the root folder. Certain Allowed Path values would result in unexpected transfer results. The status column for canceled transfers would appear animated even though they have reached a final state. Scheduled syncs without a selected source would attempt to transfer the root of the endpoint instead of the displayed folder. Transfers could sometimes reach the Completed state without a TotalSize being set. Signed urls generated by DataMan were not using HTTPS. DataMan version New Features: CycleCloud's DataMan feature now support browser uploads to any endpoint. Actions on Transfers and Scheduled Transfers tables are streamlined. Resolved Issues: Transfers table did not accurately reflect the current status or rate of transfers in some cases. Pogo did not correctly perform certificate validation for S3. Remote host endpoints could not delete files properly. Host-to-host transfers could not be canceled. Host-to-host transfers were not reported accurately. The S3 endpoint's credential field was not visible when adding an endpoint. DataMan version New Features: DataMan now supports AzureRM credential type. Users are now prompted to confirm endpoint deletion. DataMan version

22 New Features: DataMan now makes proper use of AWS credentials with AWSUseInstanceProfile set. Pogo file system configuration sections now support follow_links to copy symlink directories as if they were normal files. Resolved Issues: Deleted directories are now properly reflected by the data navigator sunburst. 18