Third-Party Client (s3fs) User Guide

Size: px

Start display at page:

Download "Third-Party Client (s3fs) User Guide"

Kellie Kennedy
6 years ago
Views:

1 Issue 02 Date HUAWEI TECHNOLOGIES CO., LTD.

2 2018. All rights reserved. No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of Huawei Technologies Co., Ltd. Trademarks and Permissions and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd. All other trademarks and trade names mentioned in this document are the property of their respective holders. Notice The purchased products, services and features are stipulated by the contract made between Huawei and the customer. All or part of the products, services and features described in this document may not be within the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements, information, and recommendations in this document are provided "AS IS" without warranties, guarantees or representations of any kind, either express or implied. The information in this document is subject to change without notice. Every effort has been made in the preparation of this document to ensure accuracy of the contents, but all statements, information, and recommendations in this document do not constitute a warranty of any kind, express or implied. Huawei Technologies Co., Ltd. Address: Website: Huawei Industrial Base Bantian, Longgang Shenzhen People's Republic of China support@huawei.com i

3 Contents Contents 1 Introduction Set-Up Preparing Tools Registering a Public Cloud Service Account Enabling OBS Logging In to OBS Console Creating Access Keys (AKs and SKs) Creating a Bucket Configuring Key Files Operation Guide Mounting a Bucket Unmounting a Bucket Troubleshooting Error: CURLE_COULDNT_RESOLVE_HOST Error: CURLE_SSL_CACERT Related Resources...14 A Document Change History ii

4 1 Introduction 1 Introduction S3 File System (s3fs) is a solution based on the Filesystem in Userspace (FUSE) interface. It allows users to use Object Storage Service (OBS) as a file folder in their local operating system (OS). Specifically, s3fs allows user to mount a bucket to a local directory and then copy files/folders to and from the directory, or delete files/folders from the directory. These operations are actually performed on the bucket. s3fs is applicable to Linux and Mac OS X operating systems. This document uses the s3fs 1.79 version as an example to describe how to use s3fs in the Linux Red Hat operating system. If you find any software operation inconsistency with this document due to an s3fs software upgrade (for example), please refer to the operation guide on the s3fs official website 1

5 2 Set-Up 2 Set-Up 2.1 Preparing Tools s3fs is a FUSE-based open-source tool, which is applicable to the Linux and Mac OS X operating systems. Its required installation environment and method differ according to the operating system. For details about how to install s3fs and its required installation environment, go to its official website Installation-Notes. After installation, check whether s3fs is installed successfully: Step 1 Step 2 Open the command terminal. Run the following commands: s3fs --version If s3fs version information is displayed, the installation is successful. ----End 2.2 Registering a Public Cloud Service Account You must register a public cloud service account before using OBS. Step 1 Step 2 Step 3 Step 4 Open a browser. Log in to the HUAWEI CLOUD website at Click Register. Complete the registration as instructed. After the registration is complete, you are navigated to your account information page. ----End 2

6 2 Set-Up 2.3 Enabling OBS You must recharge the account before using the OBS service. Step 1 Step 2 Step 3 Step 4 Step 5 Log in to Console. Click Fees. The Billing Center window is displayed. Recharge the account as prompted. After the recharging, close the window. On the homepage, click Object Storage Service to enable the OBS service and log in to OBS Console. ----End 2.4 Logging In to OBS Console This section describes how to log in to OBS Console using a web browser. Step 1 Step 2 Step 3 Log in to the HUAWEI CLOUD website at In the upper left corner, click Console. Log in using your account. If you have not registered with the public cloud, click Register and register a public cloud service account as prompted. Step 4 On the homepage of the management Console, choose All Services > Storage > Object Storage Service. ----End 2.5 Creating Access Keys (AKs and SKs) Context This section describes how to create access keys (AKs and SKs) in OBS Console. A pair of AK and SK is used to encrypt the signature of a request, ensuring that the request is secure and integral, and that identities of the request sender and receiver are correct. AKs and SKs support the authentication mechanism of Identity and Access Management (IAM). Detailed explanations are as follows: An Access Key ID (AK) defines a user that accesses the OBS system. An AK belongs to only one user, but one user can have multiple AKs. The OBS system recognizes the users that access the system by their AKs. 3

7 2 Set-Up A Secret Access Key (SK) is the key used by users to access OBS. It is the authentication information generated based on the AK and the request header. SKs and AKs are in one-to-one mapping. Restrictions and Limitations Prerequisites Each user can create up to two valid AK/SK pairs. You have registered and activated a cloud account. Step 1 Step 2 Step 3 Step 4 Step 5 Log in to OBS Console. In the upper right corner of the page, click the username and select Basic Information from the drop-down list. On the displayed page, click Manage my credentials. On the My Credential page, click the Access Keys tab, and then click Add Access Key below the displayed access key list. In the Add Access Key dialog box that is displayed, enter the password and its verification code. NOTE If you have not bound an address or mobile number, you need to enter only the password. If you have bound an address and a mobile number, you can select the verification either by address or mobile number. Step 6 Step 7 Click OK. In the Download Access Key dialog box that is displayed, click OK to save the access keys to your browser's default download path. NOTE To prevent the access keys from being leaked, keep it secure. If you click Cancel in the dialog box, the access keys will not be downloaded, and you cannot download them later. You can re-create an access key if you need to use it. Step 8 Open the downloaded credentials.csv file to obtain the access keys (AK and SK). ----End Follow-up If the AK and SK are used abnormally (including loss and leakage) or they are no longer needed, click Delete on the My Credential page to delete them. When deleting the AK and SK, you need to enter the login password and either an or mobile verification code. After the AK and SK are deleted, they will be permanently deleted and cannot be restored. 4

8 2 Set-Up 2.6 Creating a Bucket A bucket is a container used to store objects on OBS. This section describes how to create buckets. Step 1 Step 2 Log in to OBS Console. In the upper right corner of the page, click + Create Bucket. A dialog box is displayed, as shown in Figure 2-1. Figure 2-1 Creating a bucket Step 3 Set Region and Storage Class, and then enter a name for the bucket. NOTE Once you create a bucket, you cannot change the name of it. Make sure that the bucket name you set is appropriate. When a URL is used to access a bucket, the bucket name will become a part of the URL. According to the DNS rule, URLs do not support uppercase letters and cannot be used to access a bucket whose name contains uppercase letters. Therefore, a bucket name can contain only lowercase letters, digits, hyphens (-), and periods (.). For example, if you attempt to access bucket MyBucket using the URL, bucket mybucket will be accessed instead, causing an access error. Step 4 Click OK. NOTE In a multi-region scenario, if you create a bucket in a region, delete it from the region later, and immediately create a bucket with the same name in another region, the system will show a message stating that the bucket failed to be created. In this case, wait one hour and create a bucket with the same name again. Then, the system will show a message stating that the bucket has been created successfully. ----End 5

9 2 Set-Up 2.7 Configuring Key Files To access OBS, an account must use a pair of keys (AK and SK). Before using s3fs, you need to write the key information into the key file, enabling account authentication before mounting a bucket and performing other bucket operations. Prerequisites The s3fs tool is ready in place. For details about how to prepare the s3fs tool, see 2.1 Preparing Tools. You have obtained the access keys (AK and SK). For details about how to obtain access keys, see 2.5 Creating Access Keys (AKs and SKs). Step 1 Step 2 Open the command line interface (CLI). Run the following command to write the OBS account's access key into the global default file /etc/passwd-s3fs or a user-created file: echo AK:SK > Key file name Command example: echo XVSM6UAMWTPEC8G6IYBY:NxiNK1IfQVtZ4dSPNMw9QdU6EYLmXJJgvz8pB1G q > /etc/passwd-s3fs Step 3 Run the following command to check whether the key information is successfully configured: cat Key file name Step 4 Run the following command to set the key file to be used only by the current user: chmod 600 Key file name ----End 6

10 3 Operation Guide 3 Operation Guide 3.1 Mounting a Bucket Limitations and Constraints To access OBS using s3fs, you need to mount an OBS bucket to the local operating system on your host. After mounting an OBS bucket, you can operate the bucket as a local directory. For example, you can upload an object to OBS by saving a file to the mount directory. The local mount directory can either be empty or contain contents. But after an OBS bucket is mounted to the directory, the contents in the directory will be unavailable until the OBS bucket is unmounted. Therefore, it is recommended that you use an empty directory as the mount directory. A mounted OBS bucket cannot provide exact performance and functions as a local file system. For details about how to use the mounted bucket, see limitations on the official website of s3fs. Prerequisites You have configured the access keys. For details, see 2.7 Configuring Key Files. The bucket to be mounted is available on OBS. For details about how to create a bucket, see 2.6 Creating a Bucket. The local mount directory already exists. NOTE You can run the mkdir command to create a directory for mounting a bucket. For example, the command for creating a directory named mountpoint is as follows: [root@host01 ~]mkdir mountpoint Step 1 Step 2 Open the CLI. Run the mount command in the following format: s3fs Bucket name Local mounting directory -o url=endpoint address of the region -o endpoint=region Mounting parameter 7

11 3 Operation Guide You can create a bucket in a specified region on OBS. Each region has separate resources. To shorten the response time, each region provides endpoints to process OBS requests. Therefore, obtain the owning region and endpoint of the bucket to be mounted. For details, see Regions and Endpoints.. The mount parameter is optional and its format is -o Parameter name=parameter value. If you do not need to assign a parameter value, use only -o Parameter name. Table 3-1 lists OBS mount parameters and their descriptions. NOTE For details about s3fs mount parameter settings, run the s3fs --help command. Table 3-1 s3fs mount parameters Parameter Description Configuration passwd_file default_acl retries use_cache connect_timeout readwrite_timeout Indicates the key file path where the AK and SK are obtained for account authentication when using s3fs. The default path is /etc/passwds3fs. Indicates the default access control list (ACL) policy. The default value is private, indicating that the owner of the bucket has the full control permission for the bucket. Other users have no permissions to access the bucket. Indicates the maximum allowed number of attempts. The default value is 2. Indicates the local file cache used by the local directory. This parameter is not mandatory. If you set a value for this parameter, the local directory cache is used. If you do not set any value, the local directory cache is disabled. Indicates the connection timeout interval (in seconds). The default value is 10. Indicates the read/write timeout interval (in seconds). The default value is 30. Set the parameter to the path configured in 2.7 Configuring Key Files. Configure this parameter based on the preset ACL policies. The parameter value must be a positive integer. you need to enable the local directory cache. you need to change the connection timeout interval. you need to change the read/ write timeout interval. 8

12 3 Operation Guide Parameter Description Configuration max_stat_cache_size stat_cache_expire enable_noobj_cache Indicates the maximum number of items that can be saved in the local cache. This parameter is available only after use_cache is specified. The default value is 1000, approximately 4 MB in size. Indicates the cached items' expiration time (in seconds). If you use this parameter, cached items will expire after the specified time. This parameter is available only after use_cache is specified. If you do not use this parameter, cached items will not expire. Indicates whether to use the object in the cache when a specific object does not exist. By default, this function is disabled. The parameter value must be a positive integer. you need to configure an expiration time for cached items. you need to enable this function. sigv2 Indicates using for OBS requests. no_check_certificate nodnscache nosscache multireq_max parallel_count Indicates not to authenticate the server's license when the server connects to OBS using Hypertext Transfer Protocol Secure (HTTPS). Indicates not to use the domain name server (DNS) cache. Indicates not to use the Secure Sockets Layer (SSL) session cache. Indicates the maximum allowed number of concurrent requests for listing objects. The default value is 20. Indicates the maximum allowed number of requests for uploading objects. The default value is 5. you do not need to authenticate certificates. you do not need to enable DNS cache. you do not need to enable SSL session cache. you need to change the maximum allowed number of concurrent requests for listing objects. you need to change the maximum number of allowed requests for uploading objects. 9

13 3 Operation Guide Parameter Description Configuration multipart_size fd_page_size nomultipart enable_content_md5 use_path_request_styl e Indicates the size of each part in a multipart upload. The setting takes effect only when the multipart upload function is enabled. The default value is 10 MB. Indicates the size of the file description's internal management page. The default value is 50 MB. Indicates not to use a multipart upload. If you do not configure this parameter, the multipart upload method will be used. Use this parameter together with the multipart_size parameter. Indicates using content_md5 as the HTTP header when uploading an object. Do not configure this parameter, if you do not need to use content_md5 as the HTTP header. Indicates the path-based access. When using a path to access OBS, the endpoint domain name of a specific region is used to access a bucket. you need to change the size of each part in a multipart upload. you need to change the size. you do not need to use the multipart upload. Configure this parameter, if you need to use content_md5 as the HTTP header when uploading an object. - NOTE If a bucket mount fails, you can add the following parameter following up to the mount command, to display the mount process and detailed debug logs in the command return: -d -d -d -o f2 -o curldbg Then, locate the fault according to the detailed error information. For common troubleshooting information, see 4 Troubleshooting and 5 Related Resources. Step 3 Run the following command to view the mount result: df -h If the information similar to the following is displayed, the OBS bucket is successfully mounted: 10

14 3 Operation Guide ~]df -h Filesystem Size Used Avail Use% Mounted on s3fs 256T 0 256T 0% /path/to/mountpoint ----End Follow-up Operations Before adding, modifying, or deleting a parameter setting for a mounted bucket, you need to unmount the bucket and then mount it again. For details about how to unmount a bucket, see 3.2 Unmounting a Bucket. 3.2 Unmounting a Bucket Prerequisites If you do not need to access OBS by mounting a bucket, you can unmount the bucket. Unmounting the bucket does not affect data that has been stored to OBS. The to-be-unmounted bucket is in the mounted state. Step 1 Step 2 Open the CLI. Run the following command to unmount the bucket: umount Local mount directory Step 3 Run the following command to check whether the bucket has been unmounted: df -h ----End 11

15 4 Troubleshooting 4 Troubleshooting 4.1 Error: CURLE_COULDNT_RESOLVE_HOST Symptom The CURLE_COULDNT_RESOLVE_HOST error is reported when a user tries to mount a bucket by using an IP address, as shown in Figure 4-1. Figure 4-1 Mounting exception when using an IP address to mount Possible Causes Handling Method s3fs does not support IP address-based connection to OBS. None Mount the bucket using a domain name. 4.2 Error: CURLE_SSL_CACERT Symptom The CURLE_SSL_CACERT error is reported when a user tries to mount a bucket through HTTPS, as shown in Figure

16 4 Troubleshooting Figure 4-2 Mounting exception when using HTTPS to mount Possible Causes Handling Method The OBS domain name address is not authenticated by any well-known Certificate Authority (CA) authentication center. None. In the bucket mount command, set the parameter no_check_certificate. 13

17 5 Related Resources 5 Related Resources As an open source project, s3fs is highly active on GitHub and keeps irregular updates on the wiki page, helping users better understand and use s3fs. For specific page links, see Table 5-1. Table 5-1 Related resources Page s3fs overview s3fs FAQs URL

18 A Document Change History A Document Change History Release Date What's New This is the second official release. The modification is as follows: Added section "2 Set-Up", describing the basic configurations required before using s3fs. In section "3.1 Mounting a Bucket", added information about HUAWEI CLOUD regions and endpoints This issue is the first official release. 15

Third-Party Client (s3fs) User Guide

Third-Party Client (s3fs) User Guide Issue 02 Date 2017-09-28 HUAWEI TECHNOLOGIES CO., LTD. 2017. All rights reserved. No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of