Moving data to the cloud using the MoveToCloud script

Transcription

1 What this tutorial is about IBM dashdb's movetocloud script provides an easy way of getting your data to the cloud in either SoftLayer Swift object storage or Amazon S3. The script is ideal for uploading large objects over 5 GB that cannot be directly loaded into cloud storage services. In this tutorial, we ll focus on using the movetocloud script to move data into Swift object storage, but the procedure to move data into S3 is similar. What you should be able to do At the end of this tutorial, you should be able to use the movetocloud script to load large objects into Swift. What you need before you start A delimited ASCII or CSV file located on a UNIX or Linux machine A Swift account with free space A Bluemix account with dashdb deployed as a service A system installed with UNIX or Linux, Perl version 5.10 or later, curl, and gzip Page 1

2 Tutorial instructions 1. Download the movetocloud script. a. Log in to the dashdb UI. b. Click Load > Load from Cloud. c. In the Load from cloud pane, click Move to cloud. d. Click the movetocloud script link to download the script. e. Extract the downloaded file. Copy the movetocloud.pl script file and the creds.txt file to the machine where the file to be uploaded is located. f. Use the following commands to verify that you have Perl v5.10 or higher, curl, and Page 2

3 gzip installed: perl -version curl -version gzip -version If you are missing any of these prerequisites, install them before continuing. g. Navigate to the directory where the movetocloud.pl script is located, and grant execute permissions on it: chmod +x movetocloud.pl h. Review the man page for the command: movetocloud.pl man Note: If you can t see the man page using this command, you ll need to call the script by adding a Perl input (perl movetocloud.pl man). You ll also need to add the Perl input for the rest of the commands in this tutorial. 2. Update the creds.txt credentials file with your SoftLayer credentials. a. Log in to your SoftLayer Swift account. b. Navigate to the object storage and container where you want to upload your file, and click View Credentials. For this tutorial, we ll use a container called dashdbloaddemo. Page 3

4 c. In the creds.txt file, update the following fields with values from your Swift credentials: accesskey - Swift user name secretkey - Swift API key 3. Test your connection to Swift: movetocloud.pl -test target service::url::container::path creds creds.txt where: service - Represents the cloud storage name. For this tutorial, use softlayer. url - Public authentication endpoint from your SoftLayer credentials or the equivalent service tag You can use the movetocloud -list command to see the list of supported endpoints and their service tags. For this tutorial, we ll use the endpoint which corresponds to the service tag dallas. Page 4

5 container - Name of the target container. For this tutorial, use dashdbloaddemo. path - Target path to the container. For this tutorial, use the root directory (/). creds.txt - Name of the credentials file Using the specified sample values, the command to test the connection is: movetocloud.pl -test target softlyer:: shdbloaddemo::/ creds creds.txt 4. Upload your file to Swift: movetocloud.pl -source source_path target service::url::container::path creds creds.txt -verbose where: source_path - Path to the source file. For this tutorial, we ll use the value filesforupload/inventory_levels.csv service::url::container::path - Same values as step 3 -verbose - Returns additional messages about the status of the upload You can also specify the nocompression option on the command to disable compression of the target file. Using the specified sample values, the command to upload the file to Swift is: movetocloud.pl source filesforupload/inventory_levels.csv target softlyer:: shdbloaddemo::/ creds creds.txt -verbose Page 5

6 5. Verify your file was uploaded successfully. The target file is renamed with a.gz file extension, and if your original file was larger than 5 GB, an associated.gz.parts folder is created in the object store container. The folder contains the file chunks created by the movetocloud script. If your original file was larger than 5 GB, you ll see both the.gz file and.gz.parts folder in your Swift object store. Otherwise you ll see just the.gz file. To load this file into dashdb, you ll need to select Load from Cloud and specify the.gz file as your source file. Page 6