Object Storage Service. Third-Party Client (S3cmd) User Guide. Issue 03 Date

Transcription

1 Third-Party Client (S3cmd) User Guide Issue 03 Date

2 Contents Contents 1 Overview Interoperability Matrix Set-Up Preparing Tools Creating Access Keys (AKs and SKs) Interconnection with OBS Bucket Management Creating a Bucket Obtaining the Bucket List Viewing Storage Information About a Bucket Viewing Basic Information About a Bucket Configuring a Bucket Policy Configuring Static Website Hosting Configuring CORS Configuring Logging Configuring a Lifecycle Rule Deleting a Bucket Object Management Uploading an Object Obtaining the Object List Synchronizing an Object Copying an Object Moving an Object Viewing Basic Information About an Object Configuring ACL Permissions for an Object Configuring an Expiration Rule for an Object Downloading an Object Deleting an Object Multipart Upload Task Management Viewing a Multipart Upload Task Viewing the Part List Issue 03 ( ) ii

3 Contents Suspending a Multipart Upload Task Other Operations Modifying the Configuration File Operation Example FAQs Can S3cmd Upload an Object in Multiparts? Does S3cmd Have a Limit on the Size of an Object to Be Uploaded? Does S3cmd Have a Limit on the Size of an Object to Be Downloaded? Can S3cmd Batch Upload Objects? Can S3cmd Batch Delete Objects? Does S3cmd Support Resumable Upload? Troubleshooting A Connection to OBS Fails to Be Established Through HTTPS A Bucket or Object Whose Name Contains a Period (.) Fails to Be Accessed Through HTTPS...44 A Change History Issue 03 ( ) iii

4 1 Overview 1 Overview S3cmd is an open-source, free, and easy-to-use command line tool written in Python for uploading, retrieving, and managing data in Amazon Simple Storage Service (Amazon S3) and other cloud storage providers that use the S3 protocol. It is best suited for users familiar with command line. It is also ideal for batch processing and automated tasks. You can use S3cmd to access Object Storage Service (OBS) except for using the official clients provided by OBS. With S3cmd you and upload, download, delete, and set permissions for OBS buckets and objects. S3cmd is applicable to both Linux and MAC operating systems. This document uses S3cmd as an example to describe how to use S3cmd on SUSE Linux. S3cmd is compatible with most OBS application programming interfaces (APIs). This document provides an interoperability matrix and command use cases for your reference. For details, see 2 Interoperability Matrix. If procedures of upgrading S3cmd provided in this document do not suit your needs, visit the S3cmd official website ( for more instructions. Issue 03 ( ) 1

5 2 Interoperability Matrix 2 Interoperability Matrix Table 2-1 S3cmd interoperability matrix Operation Description Link to the Section Create a bucket. Creates a bucket on OBS Creating a Bucket Obtain the bucket list. View storage information about a bucket. View basic information about a bucket. Configure a bucket policy. Configure static website hosting. Configure CORS. Configure logging. Configure a lifecycle rule. Delete a bucket. Upload an object. Obtains all buckets you have on OBS. Views storage information about a specific bucket. Views basic information about a specific bucket. Manages (configures, updates, and deletes) bucket policies. Manages (configures and deletes) static website hosting configuration. Manages (configures, updates, and deletes) CORS rules of buckets. Manages (that is, enables and disables) the logging function for buckets. Manages (configures, updates, and deletes) lifecycle rules for buckets. Deletes a specific bucket from OBS. Uploads an object to a specific bucket Obtaining the Bucket List Viewing Storage Information About a Bucket Viewing Basic Information About a Bucket Configuring a Bucket Policy Configuring Static Website Hosting Configuring CORS Configuring Logging Configuring a Lifecycle Rule Deleting a Bucket Uploading an Object Issue 03 ( ) 2

6 2 Interoperability Matrix Operation Description Link to the Section Obtain the object list. Synchronize an object. Copy an object. Move an object. View basic information about an object. Configure ACL permissions for an object. Configure an expiration rule for an object. Download an object. Delete an object. View a multipart upload task. View the part list. Suspend a multipart upload task. Obtains the list of objects in a specific bucket. Synchronizes an object between the local client and OBS. Creates a copy of an existing object on OBS. Moves an object between buckets or in a bucket on OBS. Views basic information about a specific object. Configures ACL permissions of a specific user for a specific object. Configures an expiration rule for a specific object. Downloads an object from a bucket on OBS to a local client. Deletes a specific object stored on OBS. Views a multipart upload task of a specific bucket. Views the list of multipart upload tasks. Suspends a multipart upload task Obtaining the Object List Synchronizing an Object Copying an Object Moving an Object Viewing Basic Information About an Object Configuring ACL Permissions for an Object Configuring an Expiration Rule for an Object Downloading an Object Deleting an Object Viewing a Multipart Upload Task Viewing the Part List Suspending a Multipart Upload Task Issue 03 ( ) 3

7 3 Set-Up 3 Set-Up 3.1 Preparing Tools Visit the S3cmd official website ( for instructions on preparing the S3cmd installation environment and software package. Install S3cmd following descriptions in file INSTALL in the software package. After the S3cmd installation is complete, you can check whether the installation is successful. The verification procedure is as follows: Run the following command: s3cmd -help If the S3cmd help information is displayed, S3cmd is successfully installed. 3.2 Creating Access Keys (AKs and SKs) This section describes how to create access keys (AKs and SKs) in OBS Console. A pair of AK and SK is used to encrypt the signature of a request, ensuring that the request is secure and integral, and that identities of the request sender and receiver are correct. Context AKs and SKs support the authentication mechanism of Identity and Access Management (IAM). Detailed explanations are as follows: An Access Key ID (AK) defines a user that accesses the OBS system. An AK belongs to only one user, but one user can have multiple AKs. The OBS system recognizes the users that access the system by their AKs. A Secret Access Key (SK) is the key used by users to access OBS. It is the authentication information generated based on the AK and the request header. SKs and AKs are in one-to-one mapping. Issue 03 ( ) 4

8 3 Set-Up Restrictions and Limitations Each user can create up to two valid AK/SK pairs. Prerequisites You have registered and activated a cloud account. Step 4 Log in to OBS Console. In the upper right corner of the page, click the username and select My Credential. On the My Credential page, click the Access Keys tab, and then click Add Access Key below the displayed access key list. In the Add Access Key dialog box that is displayed, enter the password and its verification code. If you have not bound an address or mobile number, you need to enter only the password. If you have bound an address and a mobile number, you can select the verification either by address or mobile number. Step 5 Step 6 Click OK. In the Download Access Key dialog box that is displayed, click OK to save the access keys to your browser's default download path. To prevent the access keys from being leaked, keep it secure. If you click Cancel in the dialog box, the access keys will not be downloaded, and you cannot download them later. You can re-create an access key if you need to use it. Step 7 Open the downloaded credentials.csv file to obtain the access keys (AK and SK). Follow-up If the AK and SK are used abnormally (including loss and leakage) or they are no longer needed, click Delete on the My Credential page to delete them. When deleting the AK and SK, you need to enter the login password and either an or mobile verification code. After the AK and SK are deleted, they will be permanently deleted and cannot be restored. 3.3 Interconnection with OBS After S3cmd is successfully installed, configure the interconnection between OBS and S3cmd. You can use S3cmd to access OBS only after being authenticated by OBS. Issue 03 ( ) 5

9 3 Set-Up Prerequisites You have prepared resources (including S3cmd installation and the access key) needed for the interconnection. (Optional) If you use client-side encryption, install the GPG encryption software beforehand and obtain the software's installation path. Log in to the host where S3cmd is installed as user root. Run the following command to perform basic configuration for S3cmd: s3cmd --configure Step 4 Enter the AK, SK, and region information following the instructions in the command output. Table 3-1 describes the parameters. You can press Enter to skip optional parameters. Table 3-1 Parameters Parameter Description Mandatory or Optional Access Key The AK to access OBS. Mandatory Secret Key The SK to access OBS. Mandatory Default Region Encryption password Path to GPG program Use HTTPS protocol HTTP Proxy server name The default region that indicates where a bucket is located. The encryption password needed when you use client-side encryption to transmit data. The path of the GPG encryption software when you use client-side encryption to transmit data. Whether to use the Hypertext Transfer Protocol Secure (HTTPS) protocol. If yes, set the value of this parameter to Yes. If no, set the value of this parameter to No. Whether to enable the Hypertext Transfer Protocol (HTTP) proxy when HTTPS is not enabled. If this parameter is null, the HTTP proxy is not used. Mandatory Optional Optional Optional Optional After the previous basic parameters are configured, the current configuration result is generated for your confirmation. The following information is displayed at the end of the command output: Issue 03 ( ) 6

10 3 Set-Up Test access with supplied credentials? [Y/n] Step 5 Enter n to quit verifying the access key. By default, S3cmd sends a request to Amazon S3. If you enter Y instead, a message is displayed, indicating an AK and SK authentication failure. The following information is displayed at the end of the command output: Save settings? [y/n] Step 6 Enter y to save the current configuration. After the current configuration is saved, S3cmd will automatically generate file.s3cfg under directory $HOME. The file includes all configuration information about S3cmd. Step 7 Run the following command to open file.s3cfg: vi ~/.s3cfg Step 8 Step 9 0 Press i to go to the edit mode. Locate parameter host_base and modify the domain name address of OBS as follows host_base = Domain name address of OBS Press Esc, input :wq!, and press Enter to save and exit the vi editor. (Optional) After the configuration, run the following command to view the configuration information about S3cmd: s3cmd --dump-config 1 Run the following command to configure S3cmd: s3cmd --configure The value of a configured parameter is displayed in the square brackets ([]) after this parameter. 2 3 Press Enter to skip the AK and SK configuration items until the following message is displayed: Test access with supplied credentials? [Y/n] Enter Y to start authenticating the interconnection with OBS. If the information in the red-line box, as shown in Figure 3-1, is displayed in the command output, it indicates that S3cmd has connected to OBS successfully. Figure 3-1 Verifying the interconnection Issue 03 ( ) 7

11 3 Set-Up 4 Enter y to save the current configuration. Issue 03 ( ) 8

12 4.1 Bucket Management Creating a Bucket Prerequisites This section helps you use S3cmd to create an OBS bucket. You have installed and configured S3cmd. Run the following command to create an OBS bucket: s3cmd mb s3://my-bucket my-bucket is the name of the bucket that you want to create. The name of each bucket is globally unique on OBS and complies with the following rules: Contains 3 to 63 characters, including lowercase letters, digits, hyphens (-), and periods (.), and starts with a digit or letter Cannot be an IP address Cannot start or end with a hyphen (-) or period (.) Cannot contain two consecutive periods (.), for example, my..bucket Cannot contain periods (.) and hyphens (-) adjacent to each other, for example, my-.bucket or my.- bucket If the following command output is displayed, it indicates that an OBS bucket is successfully created. Bucket 's3://my-bucket/' created (Optional) Run the following command to obtain the bucket list and check whether the name of the newly created bucket is in the bucket list: Issue 03 ( ) 9

13 s3cmd ls Obtaining the Bucket List You can run the command to obtain the bucket list so that you can view all your buckets on OBS. Run the following command to obtain the OBS bucket list: s3cmd ls You can only view the buckets created by yourself Viewing Storage Information About a Bucket Bucket owners or users who have the s3:listbucket permission (the permission is configured in the bucket policy) can view the storage information about the bucket, including bucket size and number of objects. For details about how to configure a bucket policy, see section Configuring a Bucket Policy. Run the following command to view the storage information about a specific bucket: s3cmd du s3://my-bucket my-bucket is the name of the bucket whose storage information you want to view. In the command output, the storage information about the bucket is displayed in the following format: Bucket size Number of objects s3://my-bucket/ The unit of the bucket size is KB Viewing Basic Information About a Bucket Bucket owners or users that have the following permissions for the buckets can view the basic information about the bucket, including bucket locations, expiration rules, bucket policies, CORS rules, and ACL rules. Issue 03 ( ) 10

14 s3:getbucketlocation s3:getlifecycleconfiguration s3:getbucketpolicy s3:getbucketcors s3:getbucketacl You can obtain these permissions by configuring a bucket policy. For details about how to configure a bucket policy, see section Configuring a Bucket Policy. Run the following command to obtain the basic information about a specific bucket: s3cmd info s3://my-bucket my-bucket is the name of the bucket whose basic information you want to view. In the command output, the basic information about the bucket is displayed in the following format: s3://my-bucket/(bucket): Location: Payer: None Expiration Rule: policy: cors: ACL: Table 4-1 describes metadata parameters. Table 4-1 Object metadata parameters Metadata Location Payer Expiration Rule policy cors ACL Description The bucket location, that is, the region. The payment policy of the bucket requester. OBS does not support the ability to modify the payment policy. The value of this parameter is None. The expiration rule of a specific object in the bucket. The bucket policy that defines the access control policies of resources (buckets and objects) on OBS. The bucket cross-origin resource sharing (CORS) rule. The bucket Access Control List (ACL) rule. Issue 03 ( ) 11

15 4.1.5 Configuring a Bucket Policy On OBS, you can use a bucket policy to control access to buckets and objects in a stricter manner. For example, you can specify a bucket that a specific OBS account can access, limit a specific address's access to a specified bucket, and more. You can configure a bucket policy for your own bucket. Run the following command to create a file that stores a bucket policy. vi policy-file-name policy-file-name is the name of the file that stores a bucket policy. The value can be user-defined. Press i to go to the edit mode and configure a bucket policy in the following format: { } "Version": " ", "Id": "PolicyId", "Statement": [ { "Sid": "StmtId", "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::domainid:root" ] }, "Action": [ "s3:get*" ], "Resource": [ "arn:aws:s3:::my-bucket" ] } ] Table 4-2 describes parameters in a bucket policy. Table 4-2 Parameters in a bucket policy Parameter Description Mandatory or Optional Version The version that is consistent with Amazon S3. The value can only be Optional Id The bucket policy ID. The value must be unique. Optional Issue 03 ( ) 12

16 Parameter Description Mandatory or Optional Statement The description of the bucket policy. The statement defines complete permission control. Each bucket policy can have multiple statements, and each statement contains the following parameters: Sid Effect Principal NotPrincipal Action NotAction Resource NotResource Condition In each statement, you must select either Principal or NotPrincipal, either Action or NotAction, and either Resource or NotResource. Mandatory Sid The statement ID. Optional Effect Principal NotPrincipal The effect of the bucket policy. The statement can be sent to accept or reject requests. Allow: to accept requests. Deny: to reject requests. The grantee or user on whom the bucket policy statement takes effect. A wildcard character (*) indicates all users. Principal supports authorization of Domain in the following three formats: AWS: [domainid] AWS: [arn:aws:iam::domainid:root] CanonicalUser:[domainid] When authorizing User, Principal is in the AWS: [arn:aws:iam::domainid:user/userid] format. The unauthorized user excluded from the users that match the bucket policy. The value format is the same as Principal. Mandatory Optional Optional Issue 03 ( ) 13

17 Parameter Description Mandatory or Optional Action NotAction Resource NotResource Condition The OBS operation on which the bucket policy statement takes effect. The value is an array configured with many operations that are separated by commas (,). The value is case-insensitive and supports a wildcard character (*) that indicates all operations, for example, "Action": ["List*","Get*"]. The set of operations excluded from the operations that match the bucket policy. The value format is the same as Action. The target resource on which the bucket policy statement takes effect. If the target resource is a bucket, the format of Resource is arn:aws:s3:::my-bucket. If the target resource is an object, the format of Resource is arn:aws:s3:::my-bucket/*. NOTICE The target bucket must be an existing bucket consistent with the one specified in the command. Otherwise, you cannot successfully configure a bucket policy. The set of resources excluded from the resources that match the bucket policy. The condition that defines restrictions of this bucket policy. Optional Optional Optional Optional Optional Step 4 Step 5 Press Esc, input :wq!, and press Enter to save and exit the vi editor. Run the following command to configure this bucket policy for a specific bucket: s3cmd setpolicy policy-file-name s3://my-bucket policy-file-name is the name of the file that stores the bucket policy and my-bucket is the name of the bucket for which you want to configure this bucket policy. If the following information is displayed in the command output, it indicates that the bucket policy is successfully configured for the bucket. s3://my-bucket/: Policy updated Step 6 (Optional) Run the following command to view bucket basic information and check the bucket policy configuration: s3cmd info s3://my-bucket Issue 03 ( ) 14

18 Example A bucket policy can define many access control policies. The following provides several configuration examples. These examples introduce how to control access permissions on OBS by configuring bucket policies in different scenarios. 1. Assigning specific users the permission to obtain objects in specific buckets In the following example, the account (whose Domain ID is 783fc6652cf246c096ea836694f71855) is assigned the permission to obtain all objects in bucket examplebucket. { } "Version": " ", "Id": "Policy1", "Statement": [ { "Sid": "Stmt1", "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::783fc6652cf246c096ea836694f71855:root" ] }, "Action": [ "s3:getobject" ], "Resource": [ "arn:aws:s3:::examplebucket/*" ] } ] 2. Assigning the read-only permission to anonymous users The following example introduces how to assign the permission to anonymous users to obtain all objects in the bucket. This permission allows all users to read objects and is very helpful when you configure a bucket as a website and hope that everyone can read objects stored in the bucket. { } "Version":" ", "Id":"Policy2", "Statement":[ { "Sid":"Stmt2", "Effect":"Allow", "Principal": "*", "Action":["s3:GetObject"], "Resource":["arn:aws:s3:::examplebucket/*"] } ] 3. Configuring access permissions of specific website addresses By default, all OBS resources are private and can only be accessed by accounts that create them. OBS is the storage resource pool of a website (suppose that the website's domain name is and resources such as videos and images are stored in bucket examplebucket). If you want to allow the objects in bucket examplebucket to be accessed from configure a bucket policy based on the following example: { "Version":" ", "Id":"Policy3", "Statement":[ { Issue 03 ( ) 15

19 } ] } "Sid":"Stmt3", "Effect":"Allow", "Principal":{"AWS":["*"]}, "Action":["s3:GetObject"], "Resource":["arn:aws:s3:::examplebucket/*"], "Condition": {"StringEquals":{"aws:Referer":[" } Follow-up If you need to modify the bucket policy, you can modify the original bucket policy file or create a bucket policy file, and then run the bucket policy configuration command. By doing so, the bucket policy can be updated. You can delete an access policy of a bucket as follows if necessary: Run the following command to delete the bucket policy: s3cmd delpolicy s3://my-bucket If the following information is displayed in the command output, it indicates that this bucket policy is successfully deleted. s3://my-bucket/:policy deleted (Optional) Run the following command to view bucket basic information and check the bucket policy configuration: s3cmd info s3://my-bucket Configuring Static Website Hosting Prerequisites Bucket owners or users who have the following permissions on the buckets can configure static website hosting for the buckets and access the static websites using the bucket domain names. s3:putbucketwebsite s3:getbucketwebsite s3:deletebucketwebsite You can obtain these permissions by configuring a bucket policy. For details about how to configure a bucket policy, see section Configuring a Bucket Policy. The web page files required by the static website have been uploaded to the specified bucket. For details about how to upload objects, see section Uploading an Object. Your static website files must contain one default home page file and can contain one default 404 page. When a client attempts to access the static website using the bucket domain name, the browser can resolve the default website resources and present them to the end users. Issue 03 ( ) 16

20 Run the following command to configure static website hosting for a bucket: s3cmd ws-create --ws-index=website_index --ws-error=website_error s3://my-bucket --ws-index=website_index is the website's default home page and website_index varies with the actual home page file name. Only files in the current bucket can be used as the index page (home page) of the static website. If related file names are not entered, the default home page settings are not enabled. OBS only allows files such as index.html in the root directory of a bucket to function as the default homepage. That is to say, do not set the default homepage with a multi-level directory structure (for example, /page/index.html). --ws-error=website_error is the parameter used to set the website's default 404 page and website_error varies with the actual file name of the 404 error page. Only files in the current bucket can be used as the default 404 (Not Found) page of the static website. This error page is returned when incorrect path is being accessed. If this parameter is not used, the default 404 page settings are not enabled. my-bucket is the bucket for which static website hosting is to be configured. If the following information is displayed in the command output, it indicates that the static website is successfully set: Bucket 's3://my-bucket/': website configuration created. (Optional) Run the following command to obtain related information, such as the domain name used to access the bucket website: s3cmd ws-info s3://my-bucket Check whether the following information is displayed in the command output: Bucket s3://my-bucket/: Website configuration Website endpoint: Index document: index.html Error document: error.html Step 4 (Optional) Open the browser and enter information in the address bar in the following format: name.domain name used to access the hosted static website on OBS Follow-up Configuration of static website hosting will take effect within 2 minutes. If you fail to access the static website, try again later. You can disable static website hosting for a bucket as follows if necessary: Run the following command to delete the static website hosting configuration of a bucket as follows: s3cmd ws-delete s3://my-bucket If the following information is displayed in the command output, it indicates that the static website hosting configuration is successfully deleted: Issue 03 ( ) 17

21 Bucket 's3://my-bucket/': website configuration deleted. (Optional) Run the following command to view the static website information: s3cmd ws-info s3://my-bucket If the following information is displayed in the command output, it indicates that static website hosting is not configured for the bucket: Bucket s3://my-bucket/: Unable to receive website configuration Configuring CORS Cross-Origin Resource Sharing (CORS) is a standard mechanism put forward by World Wide Web Consortium (W3C) and enables client-side cross-origin requests. Bucket owners or users who have the following permissions on the buckets can configure CORS. After a CORS rule is configured for a bucket stored on OBS, the bucket policy and ACL permissions on the bucket's objects still prevail. s3:putbucketcors s3:getbucketcors You can obtain these permissions by configuring a bucket policy. For details about how to configure a bucket policy, see section Configuring a Bucket Policy. Run the following commands to create a file that stores a CORS rule. vi cors-file-name cors-file-name is the name of the file that stores a CORS rule and can be user-defined. Press i to go to the edit mode and configure a CORS rule in the following format: <CORSConfiguration> <CORSRule> <ID>id</ID> <AllowedMethod>method</AllowedMethod> <AllowedOrigin>origin</AllowedOrigin> <AllowedHeader>header</AllowedHeader> <MaxAgeSeconds>seconds</MaxAgeSeconds> <ExposeHeader>header</ExposeHeader> </CORSRule> </CORSConfiguration> Table 4-3 describes each parameter in a CORS rule. Issue 03 ( ) 18

22 Table 4-3 CORS rule Parameter Description Mandatory or Not CORSConfiguration CORSRule ID AllowedMethod AllowedOrigin AllowedHeader The root node of CORSRule and its capacity cannot exceed 64 KB. Type: Container Ancestor: None The CORS rule. CORSConfiguration can contain a maximum of 100 CORS rules. Type: Container Ancestor: CORSConfiguration Indicates the unique identifier of a rule. The value can contain a maximum of 255 characters. Type: String Ancestor: CORSRule Specifies the cross-origin request methods allowed by a CORS rule. If the other parameters are the same, each CORS rule can contain multiple allowed methods. Each method occupies one line. Type: String Valid values: GET, PUT, HEAD, POST, and DELETE Ancestor: CORSRule The origin of the cross-origin requests allowed by a CORS rule. Requests from this origin can access the bucket. Multiple matching rules are allowed. One rule occupies one line, and allows one wildcard character (*) at most. Type: String Ancestor: CORSRule Specifies the allowed header of cross-origin requests. Only CORS requests matching the allowed header are valid. You can enter multiple allowed headers (one per line). Each line can contain one wildcard character (*) at most. Spaces and special characters including &:< are not allowed. Type: String Ancestor: CORSRule Mandatory Mandatory Optional Mandatory Mandatory Optional Issue 03 ( ) 19

23 Parameter Description Mandatory or Not ExposeHeader MaxAgeSeconds Specifies the exposed header in CORS responses, providing additional information for clients. You can enter multiple exposed headers (one per line). Spaces and special characters including &:< are not allowed. Type: String Ancestor: CORSRule Specifies the duration that the client can cache CORS responses, expressed in seconds. The default value is 100. Each CORS rule can contain one MaxAgeSeconds at most. Type: Integer Ancestor: CORSRule Optional Optional Step 4 Step 5 Press Esc, input :wq!, and press Enter to save and exit the vi editor. Run the following command to configure this CORS rule for a specified bucket: s3cmd setcors cors-file-name s3://my-bucket cors-file-name is the name of the file that stores the CORS rule and my-bucket is the name of the bucket for which you want to configure this CORS rule. Step 6 (Optional) Run the following command to view the basic information about the bucket and check the CORS rule configuration: s3cmd info s3://my-bucket Bucket owners or users who have all the following five permissions can perform this operation. s3:getbucketlocation s3:getlifecycleconfiguration s3:getbucketpolicy s3:getbucketcors s3:getbucketacl You can obtain these permissions by configuring a bucket policy. For details about how to configure a bucket policy, see section Configuring a Bucket Policy. Example You can configure a CORS rule for a bucket to specify the origin, supported request methods, and other specific operations allowed to access a bucket. The following uses an example to introduce how to configure a CORS rule. In the example, a CORS rule is configured to allow the OBS static website to respond to cross-origin requests from another website. Issue 03 ( ) 20

24 Follow-up Usually, the scripts and content of a website cannot interact with those of another website because of the Same Origin Policy (SOP) for requests on the web page. In the following example rule, the bucket that is allowed to be configured as the static website can respond to the PUT, POST, and DELETE requests from website <CORSConfiguration> <CORSRule> <AllowedOrigin> <AllowedMethod>PUT</AllowedMethod> <AllowedMethod>POST</AllowedMethod> <AllowedMethod>DELETE</AllowedMethod> <AllowedHeader>*</AllowedHeader> <MaxAgeSeconds>3000</MaxAgeSeconds> <ExposeHeader>x-server-side-encryption</ExposeHeader> </CORSRule> </CORSConfiguration> In this example, the optional parameters and their descriptions are as follows: AllowedHeader: In this example, its value is a wildcard character (*) which indicates that all specific headers are allowed to access the bucket. MaxAgeSeconds: In this example, its value is 3000 seconds which indicates that the duration that your browser can cache CORS responses is 3000 seconds. The browser caches the CORS responses so that if repeated requests are sent, it can respond to these requests without sending them to OBS. ExposeHeader: In this example, its value is x-server-side-encryption. Each ExposeHeader element identifies a header in the response that you want customers to be able to access from their applications. If you need to modify a CORS rule for a bucket, you can modify the CORS configuration file or create a CORS file, and then run the CORS rule configuration command. By doing so, the bucket's CORS rule is updated. You can delete a CORS rule as follows if necessary: Run the following command to delete a specific CORS rule: s3cmd delcors s3://my-bucket If the following information is displayed in the command output, it indicates that the CORS rule of the bucket is successfully deleted. s3://my-bucket/: CORS deleted (Optional) Run the following command to view the basic information about the bucket and check that the CORS rule is deleted: s3cmd info s3://my-bucket Issue 03 ( ) 21

25 Bucket owners or users who have all the following five permissions can perform this operation. s3:getbucketlocation s3:getlifecycleconfiguration s3:getbucketpolicy s3:getbucketcors s3:getbucketacl Configuring Logging You can obtain these permissions by configuring a bucket policy. For details about how to configure a bucket policy, see section Configuring a Bucket Policy. By default, creating a bucket does not generate operation logs. If you want to record a bucket's operation logs, enable the logging function for this bucket. The bucket owner can configure logging. Run the following command to enable logging for a specific bucket: s3cmd accesslog --access-logging-target-prefix=log-target-prefix s3://my-bucket log-target-prefix is the path with a specified prefix that stores logs and uses the S3 URI format. For example, s3://examplebucket/logs indicates that logs are stored in bucket examplebucket and log file names are prefixed with logs. my-bucket specifies the bucket for which logging is enabled. If the following information is displayed in the command output, it indicates that logging is successfully enabled: Access logging for: s3://my-bucket/ Logging Enabled: True Target prefix: log-target-prefix Follow-up You can disable logging for the bucket as follows if necessary: Run the following command to disable logging for the bucket: s3cmd accesslog --no-access-logging s3://my-bucket If the following information is displayed in the command output, it indicates that logging is successfully enabled: Access logging for: s3://my-bucket/ Logging Enabled: False Issue 03 ( ) 22

26 4.1.9 Configuring a Lifecycle Rule Bucket owners or users who have the following permissions on the buckets can configure a lifecycle rule. Specify a lifecycle rule to automatically delete objects that match the rule once they expire, saving your storage space and costs. s3:putlifecycleconfiguration s3:getlifecycleconfiguration You can obtain these permissions by configuring a bucket policy. For details about how to configure a bucket policy, see section Configuring a Bucket Policy. Run the following command to create a file that stores a lifecycle rule: vi lifecycle-file-name lifecycle-file-name is the name of the file that stored a lifecycle rule and can be user-defined. Press i to go to the edit mode and configure a lifecycle rule in the following format: <LifecycleConfiguration> <Rule> <ID>id</ID> <Prefix>prefix</Prefix> <Status>status</Status> <Expiration> <Days>days</Days> </Expiration> <NoncurrentVersionExpiration> <NoncurrentDays>1</NoncurrentDays> </NoncurrentVersionExpiration> </Rule> </LifecycleConfiguration> Table 4-4 describes parameters in a lifecycle rule. Table 4-4 Parameters of a lifecycle rule Parameter Description Mandatory or Optional LifecycleConfiguration Indicates the container for lifecycle rules. You can add multiple rules. The total size of the rules cannot exceed 20 KB. Type: XML Children: Rule Ancestor: None Mandatory Issue 03 ( ) 23

27 Parameter Description Mandatory or Optional Rule ID Prefix Status Expiration Days Indicates the container for a specific lifecycle rule. Type: Container Ancestor: LifecycleConfiguration Indicates the unique identifier of a rule. The value can contain a maximum of 255 characters. Type: String Ancestor: Rule Indicates the object name prefix identifying one or more objects to which the rule applies. Type: String Ancestor: Rule Indicates whether the rule is enabled. Type: String Ancestor: Rule Valid values: Enabled and Disabled, which indicate that the rule is enabled or disabled respectively. Indicates the container for the object expiration rule (only applicable to the latest versions of objects). Type: XML Children: Days Ancestor: Rule Indicates the number of days when the rule takes effect after the object creation (only applicable to latest versions of objects). Type: Integer Ancestor: Expiration Mandatory Optional Mandatory Mandatory Mandatory Mandatory Issue 03 ( ) 24

28 Parameter Description Mandatory or Optional NoncurrentVersionExpiration NoncurrentDays Indicates the container for the expiration time of objects' historical versions. If versioning of a bucket is enabled or suspended, you can set NoncurrentVersionExpiration to delete historical versions of objects that match the lifecycle rule (only applicable to the historical versions of objects). Type: XML Children: NoncurrentDays Ancestor: Rule Indicates the number of days when the specified rule takes effect after the object becomes a historical version (only applicable to an object's historical version). Type: Integer Ancestor: NoncurrentVersionExpiration Optional Mandatory if NoncurrentVe rsionexpiratio n exists Step 4 Step 5 Press Esc, input :wq!, and press Enter to save and exit the vi editor. Run the following command to configure a lifecycle rule for a specific bucket: s3cmd setlifecycle lifecycle-file-name s3://my-bucket lifecycle-file-name is the name of the file that stores a lifecycle rule and my-bucket is the name of the bucket for which you want to configure this lifecycle rule. If the following information is displayed in the command output, it indicates that this lifecycle rule is successfully configured: s3://my-bucket/: Lifecycle Policy updated Step 6 (Optional) Run the following command to view the basic information about the bucket and check that the lifecycle rule is configured: s3cmd info s3://my-bucket Bucket owners or users who have all the following five permissions can perform this operation. s3:getbucketlocation s3:getlifecycleconfiguration s3:getbucketpolicy s3:getbucketcors s3:getbucketacl You can obtain these permissions by configuring a bucket policy. For details about how to configure a bucket policy, see section Configuring a Bucket Policy. Issue 03 ( ) 25

29 Example The following provides a configuration example. This example introduces how to use a lifecycle rule to manage log files that are periodically uploaded. Some periodically uploaded files need only to be retained for one week or one month, and can be deleted once they expire. In the following configuration example, log files whose name prefixes are logs/ will be automatically deleted after they are retained on OBS for 7 days. <LifecycleConfiguration> <Rule> <ID>delete-7-days</ID> <Prefix>logs/</Prefix> <Status>Enabled</Status> <Expiration> <Days>7</Days> </Expiration> </Rule> </LifecycleConfiguration> Follow-up You can delete the lifecycle rule of a specific bucket if you do not need automatic object deletion for this bucket. The deletion procedure is as follows: Run the following command to delete the lifecycle rule for a specific bucket: s3cmd dellifecycle s3://my-bucket Step 4 If the following information is displayed in the command output, it indicates that this lifecycle rule is successfully deleted: s3://my-bucket/: Lifecycle Policy deleted (Optional) Run the following command to view the basic information about the bucket and check that the lifecycle rule is successfully deleted: s3cmd info s3://my-bucket Bucket owners or users who have all the following five permissions can perform this operation. s3:getbucketlocation s3:getlifecycleconfiguration s3:getbucketpolicy s3:getbucketcors s3:getbucketacl Deleting a Bucket You can obtain these permissions by configuring a bucket policy. For details about how to configure a bucket policy, see section Configuring a Bucket Policy. You can delete a specific empty bucket and a bucket created by yourself. Issue 03 ( ) 26

30 NOTICE Before deleting a bucket, ensure that all objects in this bucket have been deleted and all multipart upload tasks are completed. Run the following command to delete a specific bucket: s3cmd rb s3://my-bucket my-bucket is the name of the bucket that you want to delete. For example, s3cmd rb s3:// examplebucket indicates that the bucket to be deleted is examplebucket. If the following information is displayed in the command output, it indicates that this bucket is successfully deleted: Bucket 's3://my-bucket/' removed (Optional) Run the following command to check that the deleted bucket does not exist in the bucket list: s3cmd ls 4.2 Object Management Uploading an Object Bucket owners or users who have the s3:putobject permission on buckets can upload local objects to OBS. You can obtain the permission by configuring a bucket policy. For details about how to configure a bucket policy, see section Configuring a Bucket Policy. Prerequisites Currently, S3cmd does not provide versioning. Therefore, an object name must be unique in a bucket. If you upload a new object with the same name as the existing object in the bucket, the new object will overwrite the existing one. The target bucket already exists. Run the following command to upload an object to a specific location on OBS: Issue 03 ( ) 27

31 s3cmd put params object s3://my-bucket object is the name of the object that you want to upload, which can be a file or folder. If the object to be uploaded is a folder, parameter --recursive is mandatory. The object name complies with the following rules: The folder name cannot contain spaces and special characters. The folder name cannot exceed 1023 bytes. The length of a folder name is the sum of the length of its own and the length of its upperlevel directories, and cannot exceed 1023 bytes. Levels of directories are automatically separated by slashes (/). For example, if the upper-level directory of file01 is folder01, file name length is calculated based on folder01/file01. my-bucket varies with the actual bucket to which the object is uploaded. For example, s3cmd put test s3://examplebucket indicates that object test will be uploaded to bucket examplebucket. params is an optional variable. It can be specified in the command to determine whether to encrypt objects for upload and the encryption mode. Multiple parameters can be specified for this variable and they are separated by spaces. Table 4-5 describes the parameters. If parameters are specified in the command, it indicates that these settings are only applicable to the current command. For global application, modify related parameters in the configuration file. For details, see section Modifying the Configuration File. Table 4-5 Parameters of uploading an object Parameter --ssl --encrypt --skip-existing --delete-removed --recursive --disable-multipart --multipart-chunk-sizemb=value --continue-put Description Uploads objects over HTTPS. If you do not want to use HTTPS, use parameter --no-ssl. Uses client-side encryption to upload objects. This encryption function requires installing the GPG encryption tool on the client and configuring client-side encryption through S3cmd. You need to configure the client-side encryption password and path of the encryption tool. For details about how to configure client-side encryption, see 3.3 Interconnection with OBS. Skips existing objects that do not need to be uploaded. Deletes objects that have already been deleted at the source on OBS. Recursively uploads objects. If the object to be uploaded is a folder, you can upload the parent-level object (this folder) first and then the child-level objects (files in this folder) in a recursive manner. Disables the function of uploading an object in multiparts. Specifies the part size. The part size configured through S3cmd ranges from 5 MB to 5120 MB. The number of parts ranges from 1 to 10,000. The default part size is 15 MB. Continues uploading an object's remaining content. Issue 03 ( ) 28

32 Parameter --server-sideencryption-kmsid=kms-key Description Uploads objects using the server-side encryption. KMS-KEY indicates the KMS key used for the server-side encryption. For details about how to create and view a KMS key, see sections Creating a CMK and Querying a CMK in the Key Management Service User Guide. If you use server-side encryption to upload objects, the request must be sent over HTTPS. Therefore, parameters -s or --ssl must be used in the command. --limit-rate=value Restricts the upload speed. If the upload speed is restricted to 10 KB/s, the configuration is: --limit-rate=10k If the unit is not specified, the default unit MB is used. --stop-on-error Stops the upload once an error occurs. In the command output, if done is displayed at the end of all objects uploaded, and the cursor appears on the command-line interface (CLI) again, it indicates that all objects have been uploaded. During the upload, if you want to stop the upload task, press Ctrl+C. OBS retains data that has been uploaded. You can add parameter --continue-put in the upload command to continue the upload next time. (Optional) Run the following command to check whether the object is uploaded to the specified location: s3cmd ls s3://my-bucket Only the bucket owner can run this command Obtaining the Object List You can view the information about an object in your bucket. The information includes the object's creation time, size, and storage path. Run the following command to view the object list in a specific bucket: s3cmd ls s3://my-bucket Issue 03 ( ) 29

33 Follow-up my-bucket is the name of the bucket whose object information you want to view. Operations on S3cmd prevail globally. Thus, you can view all objects in all buckets. The operation method is as follows: Run the following method to list all objects in a bucket: s3cmd la Synchronizing an Object When local data or data stored on OBS changes, you can perform this operation to synchronize the changed data from your local client to OBS or from OBS to your local client. By doing so, local data and data stored on OBS are consistent. Run a command specific to the scenario: Synchronizing local data to a bucket on OBS s3cmd sync params local-dir s3://my-bucket/prefix Synchronizing data in a bucket on OBS to a local client s3cmd sync params s3://my-bucket/prefix local-dir local-dir is the path of local data. my-bucket/prefix is the object to be synchronized. prefix is optional and specifies the prefix of objects in this bucket. If you do not specify objects, it indicates that all objects will be synchronized. params is an optional variable that indicates the command parameter. You can configure a specific parameter to set properties of the object to be uploaded. Table 4-6 describes the optional parameters. Table 4-6 Parameters of synchronizing an object Parameter --no-check-md5 --skip-existing Description Configure this parameter when the MD5 check is not required. After configuring this parameter, consistency between source objects and target objects is checked by object names and size. Skips existing objects that do not need to be synchronized. Issue 03 ( ) 30

34 Parameter --delete-removed --recursive --continue-put --multipart-chunk-sizemb=value --server-sideencryption-kms-id Description my-bucket/prefix is the object to be synchronized. prefix is optional and specifies the prefix of objects in this bucket. Recursively uploads objects. Specifies the part size. The part size configured through S3cmd ranges from 5 MB to 5120 MB. The number of parts ranges from 1 to 10,000. The default part size is 15 MB. Continues uploading an object's remaining content. Specifies the server-side KMS encryption password. In the command output, if done is displayed at the end of all objects synchronized, and the cursor appears on the CLI again, it indicates that all objects have been synchronized. (Optional) Run a command specific to the scenario to check whether objects have been synchronized. Check whether the local object has been synchronized with the target object on OBS. s3cmd ls s3://my-bucket Check whether the object on OBS has been synchronized to the local client. ls local-dir Copying an Object S3cmd provides the object copy command to create a copy for an existing object on OBS. Run a command that varies depending on whether a target object name is specified. Without the target object name specified s3cmd cp s3://my-bucket-1/object s3://my-bucket-2 It is a good practice to copy an object to different buckets or folders. An object name must be unique in a bucket. If the source object's path is the same as the copy's path, the object's copy fails to be created. With the target object name specified s3cmd cp s3://my-bucket-1/object s3://my-bucket-2/object-copy (Optional) Run the following command to check that the object's copy has been created in a specific bucket. s3cmd ls s3://my-bucket-2 Issue 03 ( ) 31

35 4.2.5 Moving an Object You can move data in a bucket to a specific folder within this bucket or between buckets. Run the following command to move an object to a specific location on OBS: s3cmd mv s3://my-bucket-1/object-1 s3://my-bucket-2/object-2 In the command output, the source path and target path are displayed. When the cursor appears on the CLI again, the object is successfully moved. (Optional) Run the following command both in the source path and target path to check that the object has been moved: s3cmd ls s3://my-bucket Viewing Basic Information About an Object Object owners or users who have the read permission on the objects can view the basic information about objects, their size, last modification time, storage class, and so on. Permissions on objects can be obtained by configuring the ACL permissions. For details, see section Configuring ACL Permissions for an Object. Run the following command to view the basic information about a specific object: s3cmd info s3://my-bucket/object In the command output, the object information is displayed in the following format: s3://my-bucket/object(object): File size: Last Mod: MIME type: Storage: MD5 sum: SSE: policy: cors: ACL: x-amz-meta-s3cmd-attrs: Table 4-7 describes metadata parameters. Table 4-7 Object metadata parameters Metadata File size Description The object size. Issue 03 ( ) 32