Third-Party Client (s3fs) User Guide

Transcription

1 Issue 02 Date HUAWEI TECHNOLOGIES CO., LTD.

2 2017. All rights reserved. No part of this document may be reproduced or transmitted in any form or by any means without prior written consent of Huawei Technologies Co., Ltd. Trademarks and Permissions and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd. All other trademarks and trade names mentioned in this document are the property of their respective holders. Notice The purchased products, services and features are stipulated by the contract made between Huawei and the customer. All or part of the products, services and features described in this document may not be within the purchase scope or the usage scope. Unless otherwise specified in the contract, all statements, information, and recommendations in this document are provided "AS IS" without warranties, guarantees or representations of any kind, either express or implied. The information in this document is subject to change without notice. Every effort has been made in the preparation of this document to ensure accuracy of the contents, but all statements, information, and recommendations in this document do not constitute a warranty of any kind, express or implied. Huawei Technologies Co., Ltd. Address: Website: Huawei Industrial Base Bantian, Longgang Shenzhen People's Republic of China support@huawei.com i

3 Contents Contents 1 Introduction Set-Up Preparing Tools Registering a Public Cloud Service Account Enabling OBS Logging In to OBS Console Creating Access Keys (AKs and SKs) Creating a Bucket Configuring Access Key Files Operation Guide Mounting a Bucket Unmounting a Bucket Troubleshooting An Error: CURLE_COULDNT_RESOLVE_HOST An Error: CURLE_SSL_CACERT Related Resources...17 A Document Change History ii

4 1 Introduction 1 Introduction S3 File System (s3fs) is a File System in User Space (FUSE)-based solution. It allows users to use Object Storage Service (OBS) like using a normal file folder in their local operating system (OS). Specifically, s3fs allows user to mount a bucket to a local directory and then copy files/folders to, copy files/folders from, or delete files/folders in this directory. These operations are actually file/folder upload, download, and deletion operations within the bucket. s3fs is applicable to Linux and Mac Os X operating systems. This document uses the s3fs 1.79 version as an example to describe how to use s3fs in the Linux RedHat operating system. If you find any software operation inconsistency with this document due to an s3fs software upgrade (for example), please refer to the operation guide on the s3fs official website 1

5 2 Set-Up 2 Set-Up 2.1 Preparing Tools s3fs is a FUSE-based open-source tool, which is applicable to the Linux and Mac OS X operating systems. Its required installation environment and method differ according to the operating system. For details about how to install s3fs and its required installation environment, go to its official website Installation-Notes. After installation, check whether s3fs is installed successfully: Step 1 Step 2 Open the command terminal. Run the following commands: s3fs --version If s3fs version information is displayed, the installation is successful. ----End 2.2 Registering a Public Cloud Service Account Procedure You must register a public cloud service account before using OBS. Step 1 Step 2 Step 3 Step 4 Open a browser. Log in to the HUAWEI CLOUD website at Click Register. Complete the registration as instructed. After the registration is complete, you are navigated to your account information page. ----End 2

6 2 Set-Up 2.3 Enabling OBS Procedure You must recharge the account before using the OBS service. Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Log in to OBS Console. In the upper right corner of the page, click the username and select User Center. Click Recharge. The recharging window is displayed. Recharge the account as prompted. After the recharging, close the window. On the homepage, click Object Storage Service to enable the OBS service and log in to OBS Console. ----End 2.4 Logging In to OBS Console Procedure This section describes how to log in to OBS Console using a web browser. Step 1 Step 2 Step 3 Log in to the HUAWEI CLOUD website at In the upper right corner, click Management Console. Log in using the account. If you have not registered with the public cloud, click Register and register a public cloud service account as prompted. Step 4 On the homepage of the management Console, choose All Services > Storage > Object Storage Service. ----End 2.5 Creating Access Keys (AKs and SKs) Context This section describes how to create access keys (AKs and SKs) on OBS Console. A pair of AK and SK is used to encrypt the signature of a request, ensuring that the request is secure and integral, and that identities of the request sender and receiver are correct. AKs and SKs support the authentication mechanism of Identity and Access Management (IAM). Detailed explanations are as follows: 3

7 2 Set-Up An Access Key ID (AK) defines a user that accesses the OBS system. An AK belongs to only one user, but one user can have multiple AKs. The OBS system recognizes the users that access the system by their AKs. Restrictions and Limitations Prerequisites Procedure A Secret Access Key (SK) is the key used by users to access OBS. It is the authentication information generated based on the AK and the request header. SKs and AKs are in one-to-one match. Each user can create up to two valid AK/SK pairs. You have registered and activated a cloud account. Step 1 Step 2 Step 3 Step 4 Log in to OBS Console. In the upper right corner of the page, click the username and select My Credential. On the My Credential page, click the Access Keys tab, and then click Add Access Key below the displayed access key list. In the Add Access Key dialog box that is displayed, enter the password and its verification code. NOTE If you have not bound an address or mobile number, you need to enter only the password. If you have bound an address and a mobile number, you can select the verification either by address or mobile number. Step 5 Step 6 Click OK. In the Download Access Key dialog box that is displayed, click OK to save the access keys to your browser's default download path. NOTE To prevent the access keys from being leaked, keep it secure. If you click Cancel in the dialog box, the access keys will not be downloaded, and you cannot download them later. You can recreate an access key if you need to use it. Step 7 Open the downloaded credentials.csv file to obtain the access keys (AK and SK). ----End Follow-up Procedure If the access key is used abnormally (including loss and leakage) or it is no longer needed, click Delete to delete it from the access key list. When deleting the access keys, you need to enter the login password and either an or mobile verification code. After the access keys are deleted, it will be permanently deleted and cannot be restored. 4

8 2 Set-Up 2.6 Creating a Bucket This section describes how to create a bucket in OBS Console. A bucket is a container used to store objects on OBS. Before you store data onto OBS, you need to create buckets. Procedure NOTE A user can create a maximum of 100 buckets on OBS. Step 1 Step 2 Log in to OBS Console. In the upper right corner of the page, click Create Bucket. The Create Bucket dialog box is displayed, as shown in Figure 2-1. Figure 2-1 Create Bucket Step 3 Select a region and a storage class, and enter a name for the bucket. Table 2-1 Parameters for creating a bucket Parameter Region Description Region where the bucket to be created is located. 5

9 2 Set-Up Parameter Storage Class Bucket Name Description OBS has the following storage classes: Standard, Infrequent Access, and Archive. In such a manner, OBS comprehensively meets customers' requirements on storage performance and costs. When creating a bucket, you can specify the storage class for the bucket. On OBS, each object has a storage class that is the same as its owning bucket. OBS Standard: features low access latency and high throughput. OBS Infrequent Access: applicable to storing semi-frequently accessed (less than 12 times a year) data requiring quick response. OBS Archive: applicable to archiving rarely-accessed (once a year) data. Name of the bucket to be created. The bucket name must be globally unique and comply with the following rules: Contains 3 to 63 characters, including lowercase letters, digits, hyphens (-), and periods (.) Cannot be an IP address. Cannot start or end with a hyphen (-) or period (.) Cannot contain two consecutive periods (.), for example, my..bucket. Cannot contain periods (.) and hyphens (-) adjacent to each other, for example, my-.bucket or my.-bucket. NOTE When a URL is used to access a bucket, the bucket name will become a part of the URL. According to DNS standards, URLs do not support uppercase letters and cannot be used to access a bucket whose name contains uppercase letters. Therefore, a bucket name can contain only lowercase letters, digits, hyphens (-), and periods (.). For example, if you attempt to access bucket MyBucket using the URL, it will be resolved into mybucket, causing an access error. Once you create a bucket, you cannot change the name of it. Make sure the bucket name you set is appropriate. Step 4 Click Advanced, as shown in Figure 2-2, and set the following bucket properties. Alternatively, you can set these properties after creating the bucket. 6

10 2 Set-Up Figure 2-2 Creating a bucket Versioning: OBS can store multiple versions of an object. You can quickly search for and restore different versions or restore data in the event of misoperations or application faults. By default, versioning is disabled for new buckets on OBS. Log management: Access logs enable a bucket owner to analyze the property, type, or trend of requests to the bucket in depth. By default, OBS does not log bucket access. Step 5 Click Create Bucket. NOTE In a multi-region scenario, if you create a bucket in a region, delete it from the region later, and immediately create a bucket with the same name in another region, the system will show a message stating that the bucket failed to be created. In this case, wait one hour and create a bucket with the same name again. Then, the system will show a message stating that the bucket has been created successfully. ----End 2.7 Configuring Access Key Files An account can access OBS only when using a pair of keys (namely, AK and SK). Before using s3fs, you need to write key information into the key file, enabling account authentication before mounting a bucket and other bucket operations. Prerequisites The s3fs tool is ready in place. For details about how to prepare the s3fs tool, see 2.1 Preparing Tools. You have obtained the access keys (a pair of AK and SK). For details about how to obtain access keys, see 2.5 Creating Access Keys (AKs and SKs). 7

11 2 Set-Up Procedure Step 1 Step 2 Open the command terminal. Run the following command to write the OBS account's access key into the global default file /etc/passwd-s3fs or a user-created file: echo AK:SK > Key file name Command example: echo XVSM6UAMWTPEC8G6IYBY:NxiNK1IfQVtZ4dSPNMw9QdU6EYLmXJJgvz8pB1G q > /etc/passwd-s3fs Step 3 Run the following command to check whether the key information is successfully configured: cat Key file name Step 4 Run the following command to set the key file allowing it to be used only by the current user: chmod 600 Key file name ----End 8

12 3 Operation Guide 3 Operation Guide 3.1 Mounting a Bucket Limitations and Constraints To access OBS by s3fs, you need to mount an OBS bucket to the local operating system of your host. After mounting the OBS bucket, you can operate an OBS bucket like a local directory. For example, you can upload an object to OBS by saving a file to the mount directory. The local mount directory can either be empty or contain contents. But after the OBS bucket is mounted, the contents in the directory will be unavailable until the OBS bucket is unmounted. You are advised to use an empty directory as the mount directory. A mounted OBS bucket cannot provide exact performance and functions as the local file system. For more information about use details, see limitations on the official website of s3fs. Prerequisites You have configured the access keys. For details, see 2.7 Configuring Access Key Files. The bucket to be mounted is available on OBS. For details about how to create a bucket, see 2.6 Creating a Bucket. The local mount directory already exists. NOTE You can run the mkdir command to create a directory for mounting a bucket. For example, the command for creating a directory named mountpoint is as follows: [root@host01 ~]mkdir mountpoint Procedure Step 1 Step 2 Open the command terminal. Run the mount command with the following format: s3fs Bucket name Local mounting directory -o url=endpoint address of the region -o endpoint=region Mounting parameter 9

13 3 Operation Guide You can create a bucket in a specified region on OBS. Each region has separate resources. To shorten the response time, each region provides endpoints to process OBS requests. Therefore, obtain the owning region of the bucket to be mounted and its corresponding endpoint address. Table 3-1 lists regions where OBS is available. For details, see Regions and Endpoints Table 3-1 OBS regions Region Name Region Endpoint North China 1 cn-north-1 obs.cn-north-1.myhwclouds.com East China 1 cn-east-2 obs.cn-east-2.myhwclouds.com South China 1 cn-south-1 obs.cn-south-1.myhwclouds.com The mount parameter is optional and its format is -o Parameter name=parameter value. If you do not need to assign a parameter value, use only -o Parameter name. Table 3-2 lists OBS mount parameters and their descriptions. NOTE For details about s3fs mount parameter settings, run the s3fs --help command. Table 3-2 s3fs mount parameters Parameter Description Configuration passwd_file default_acl retries Indicates the key file path where the AK and SK are obtained for account authentication when using s3fs. The default path is /etc/passwds3fs. Indicates the default access control list (ACL) policy. The default value is private, indicating that the owner of the bucket has the full control permission for the bucket. Other users have no permissions to access the bucket. Indicates the maximum allowed number of attempts. The default value is 2. Set the parameter to the path configured in 2.7 Configuring Access Key Files. Configure this parameter based on the preset ACL policies. For details about the preset ACL policies, see ACL in Object Storage Service API Reference. The parameter value must be a positive integer. 10

14 3 Operation Guide Parameter Description Configuration use_cache connect_timeout readwrite_timeout max_stat_cache_size stat_cache_expire enable_noobj_cache sigv2 no_check_certificate Indicates the local file cache used by the local directory. This parameter is not mandatory. If you set a value for this parameter, the local directory cache is used. Otherwise, the local directory cache is disabled. Indicates the connection timeout interval (in seconds). The default value is 10. Indicates the read/write timeout interval (in seconds). The default value is 30. Indicates the maximum number of items that can be saved in the local cache. This parameter is available only after use_cache is specified. The default value is 1000, approximately 4 MB in size. Indicates the cached items' expiration time (in seconds). If you use this parameter, cached items will expire after the specified time. This parameter is available only after use_cache is specified. If you do not use this parameter, cached items will not expire. Indicates whether to use the object in the cache when a specific object does not exist. By default, this function is disabled. Indicates using V2 authentication for OBS requests. If you do not set this parameter, it indicates using V4 authentication. Indicates not to authenticate the server's license when the server connects to OBS using Hypertext Transfer Protocol Secure (HTTPS). you need to enable the local directory cache. you need to change the connection timeout interval. you need to change the read/ write timeout interval. The parameter value must be a positive integer. you need to configure an expiration time for cached items. you need to enable this function. For details about request authentication, see Authenticating a Request in the Object Storage Service API Reference. you do not need to authenticate certificates. 11

15 3 Operation Guide Parameter Description Configuration nodnscache nosscache multireq_max parallel_count multipart_size fd_page_size nomultipart enable_content_md5 Indicates not to use the domain name server (DNS) cache. Indicates not to use the Secure Sockets Layer (SSL) session cache. Indicates the maximum allowed number of concurrent requests for listing objects. The default value is 20. Indicates the maximum allowed number of requests for uploading objects. The default value is 5. Indicates the size of each part in a multipart upload. The setting takes effect only when the multipart upload function is enabled. The default value is 10 MB. Indicates the size of the file description's internal management page. The default value is 50 MB. Indicates not to use a multipart upload. If you do not configure this parameter, the multipart upload method will be used. Use this parameter together with the multipart_size parameter. Indicates using content_md5 as the HTTP header when uploading an object. Do not configure this parameter, if you do not need to use content_md5 as the HTTP header. you do not need to enable DNS cache. you do not need to enable SSL session cache. you need to change the maximum allowed number of concurrent requests for listing objects. you need to change the maximum number of allowed request requests for uploading objects. you need to change the size of each part in a multipart upload. you need to change the size. you do not need to use the multipart upload. Configure this parameter, if you need to use content_md5 as the HTTP header when uploading an object. 12

16 3 Operation Guide Parameter Description Configuration use_path_request_styl e Indicates the path-based access. When using a path to access OBS, the endpoint domain name of a specific region is used to access a bucket. - NOTE If a bucket mount fails, you can add the following parameter after the mount command, to display the mount process and detailed debug logs in the command return: -d -d -d -o f2 -o curldbg Then, locate the fault according to the detailed error information. For common troubleshooting information, see 4 Troubleshooting and 5 Related Resources. Step 3 Run the following command to view the mount result: df -h If the information similar to the following is displayed, the OBS bucket is successfully mounted: [root@host01 ~]df -h Filesystem Size Used Avail Use% Mounted on s3fs 256T 0 256T 0% /path/to/mountpoint ----End Command Example OBS bucket-targeted s3fs' functions, such as authentication and multipart upload are managed based on relevant parameter settings during a bucket mount. In addition, s3fs supports timeout and cache settings. The following describes how to manage buckets by setting parameters, with the prerequisites that the key file path is /etc/passwd-s3fs, and that an OBS bucket is mounted from the North China 1 region. Mount the bucket mybucket to the /path/to/mountpoint directory: s3fs mybucket /path/to/mountpoint -o url= -o endpoint=cn-north-1 -o passwd_file=/etc/passwd-s3fs Mount the bucket mybucket to the /path/to/mountpoint directory, set the retry times to 3, set the connection timeout interval to 60s, and set the read/write timeout interval to 30s: s3fs mybucket /path/to/mountpoint -o url= -o endpoint=cn-north-1 -o passwd_file=/etc/passwd-s3fs -o retries=3 -o connect_timeout=60 -o readwrite_timeout=30 Mount the bucket mybucket to the /path/to/mountpoint directory, use v2 authentication to access OBS, and set a 50 MB per-part upload size for a multipart upload: s3fs mybucket /path/to/mountpoint -o url= -o endpoint=cn-north-1 -o passwd_file=/etc/passwd-s3fs -o sigv2 -o multipart_size=50mb 13

17 3 Operation Guide Follow-up Operations Before adding, modifying, or deleting a parameter setting for a mounted bucket, you need to unmount the bucket and then mount it again. For details about how to unmount a bucket, see 3.2 Unmounting a Bucket. 3.2 Unmounting a Bucket Prerequisites Procedure If you do not need to access OBS by mounting a bucket, you can unmount the bucket. Unmounting the bucket does not affect data that has been stored to the OBS. The to-be-unmounted bucket is in mounted state. Step 1 Step 2 Open the command terminal. Run the following command to unmount the bucket: umount Local mount directory Step 3 Run the following command to check whether the bucket has been unmounted: df -h ----End 14

18 4 Troubleshooting 4 Troubleshooting 4.1 An Error: CURLE_COULDNT_RESOLVE_HOST Symptom The CURLE_COULDNT_RESOLVE_HOST error is reported when a user tries to mount a bucket by using an IP address, as shown in Figure 4-1. Figure 4-1 Mounting exception when using an IP address to mount Possible Causes Handling Method Procedure s3fs does not support IP address-based connection to OBS. None Mount the bucket using a domain name. 4.2 An Error: CURLE_SSL_CACERT Symptom The CURLE_SSL_CACERT error is reported when a user tries to mount a bucket through HTTPS, as shown in Figure

19 4 Troubleshooting Figure 4-2 Mounting exception when using HTTPS to mount Possible Causes Handling Method Procedure The OBS domain name address is not authenticated by any well-known Certificate Authority (CA) authentication center. None. In the bucket mount command, set the parameter no_check_certificate. 16

20 5 Related Resources 5 Related Resources As an open source project, s3fs is highly active on GitHub and keeps irregular updates on the wiki page, helping users better understand and use s3fs. For specific page links, see Table 5-1. Table 5-1 Related resources Page s3fs overview s3fs FAQs URL

21 A Document Change History A Document Change History Release Date What's New This is the second official release. The modification is as follows: Added section "2 Set-Up", describing the basic configurations required before using s3fs. In section "3.1 Mounting a Bucket", added information about HUAWEI CLOUD regions and endpoints, and described how to mount an OBS bucket from the North China 1 region as an example This issue is the first official release. 18