Hands-On Lab: NetApp StorageGRID Webscale March 2017 SL10303 Version 3.1

Transcription

1 Hands-On Lab: NetApp StorageGRID Webscale 10.3 March 2017 SL10303 Version 3.1

2 TABLE OF CONTENTS 1 Introduction to Object Storage Introduction to this Lab Lab Objectives Prerequisites Lab Environment Lab Activities Lab Tools Overview Connect to Admin GUI S3 Account Management Configure S3 Client Create S3 Buckets Examine ILM Rules Create New ILM rules Activate ILM rules Ingest data and Validate Data Placement Validate ILM policy has been applied ILM Re-evaluation S3 Versioning Configure a Swift Client Configure the NAS Bridge Exploring S3 and Swift APIs Access to S3 via the AWS CLI Access to S3 through Python Access to Swift through the CLI Access to Swift Using Python Version History

3 1 Introduction to Object Storage Object storage is a little different than the familiar block and file storage. It organizes data into flexibly sized data containers called objects. Objects are stored in a flat namespace that may span multiple locations. Each object has both data (an un-interpreted sequence of bytes), and metadata (a unique ID plus an extensible set of attributes that describe the object). Data stored using object storage is typically accessed by a client application using an object protocol or API, rather than at the user level (accessing a file). A simple way to think about this is that object storage is like valet parking you give the valet your ticket, and you get your car back without ever needing to know anything about where the car was parked. The advantage of this approach is that data can be referenced and retrieved just by providing the unique ID. Figure 1-1: Object Storage Concepts Three trends are contributing to increasing interest in object storage: Continued growth in the amount of unstructured data necessitates a new approach to storing and protecting data. Object-enabled data management facilitates intelligent data placement to meet a wide range of performance, durability, availability, location, and longevity requirements. Enterprises are increasingly coping with decentralized data creation and consumption. The "primary data center plus DR data center" model is being replaced by a multisite approach in which users, workloads, and data are brought closer together. The hybrid cloud is providing new options to balance cost and performance, and IT teams are looking at how they can best leverage both on-premises and cloud-based storage. The Opportunity: Storage growth to billions of objects. Decentralized data production and global distribution is driving new data management approaches to reduce operation costs and complexity. Due to data sovereignity, performance, cost, or a combination of all three, a hybrid cloud with both onpremises and clooud-based storage is a requirement. Use Cases StorageGRID Webscale is extremely well suited for web data repositories, data archives, and media repositories. Each of these use cases has its own distinct set of requirements, but StorageGRID Webscale adapts to accommodate the wide variety of needs encompassed in this set of use cases. 3

4 Figure 1-2: Use Cases for Object Storage Web Data Repositories Web data repositories are characterized by small object size, high object count, and high transactions. Because it can handle up to 100 billion objects in a single repository distributed across many locations and with S3, Swift and CDMI API support, StorageGRID Webscale is well suited for this use case. Data Archives Increasingly, enterprises are storing massive amounts of data for extended periods to satisfy both corporate governance and legal requirements. With data archives of this type, cost and management are typically the most important concerns. Long access latency is tolerated in exchange for reduced cost. StorageGRID Webscale satisfies this use case with tape integration, proven data durability, and flexible, policy based management. Media Repositories Media repositories are characterized by large object sizes, with the need for geographical distribution, a need for data integrity, and a low time-to-first byte latency. The geographically distributed, durable design of StorageGRID Webscale satisfies these requirements. It also supports ranged reads, so, for example, a video can be streamed from any point without having to download the entire object. Introduction to StorageGRID Webscale The StorageGRID Webscale system is a distributed object storage system that stores, protects, and preserves fixed-content data over long periods of time. By employing a grid architecture that distributes copies of object data throughout the system, a highly reliable system is created where data is continuously available. If one part of the system goes down, another immediately takes over, which results in objects always being available for retrieval. To implement this architecture, the StorageGRID Webscale system employs a system of network connected servers hosting grid nodes. These grid nodes host a collection of one or more services, each providing a set of capabilities to the StorageGRID Webscale system. To manage objects ingested into the system, the StorageGRID Webscale system employs metadata based information lifecycle management (ILM) rules. These ILM rules determine what happens to an object s data after it is ingested where it is stored, how it is protected from loss, and how long it is stored. The StorageGRID Webscale system operates over wide area network (WAN) links, providing the system with off-site loss protection. Copies are made and distributed throughout the system so that objects are continuously available. In systems with multiple sites, this distribution of copies means that if a site is lost, data is not lost, and clients can seamlessly retrieve it from other sites. Advantages of the StorageGRID Webscale system include the following: 4 Software-defined storage management for hardware independence Support for standard object storage protocols and APIs:

5 Simple Storage Service (S3) OpenStack Swift Cloud Data Management Interface (CDMI) Support for standard file system storage protocols: Network File System (NFS) Server Message Block (SMB) Configurable data protection using replicated copies, erasure coding or cloud tiering. Ability to address hardware obsolescence through the transparent migration of data from previous to current generations of hardware. Policy-driven tiered storage that reduces overall storage costs: Data can be migrated to lower cost tier based on last access time or other value. An object stored on a low cost tier can be copied to faster tier once accessed. Consider a policy that stores data on tape or public cloud a policy to copy frequently accessed data to a replica provides user driven cache effect. Virtualization of storage across sites and tiers, insulating applications from changes to underlying storage infrastructure. Ability to monitor and verify data integrity proactively as data is stored, updated, retrieved, and deleted. Non-disruptive upgrade, expansion, and maintenance that maintains access to content StorageGRID Webscale Partners StorageGRID Webscale can work with client applications that support storing to S3 or Swift compatible storage. Figure 1-3: Sample Partners and Client Applications in the StorageGRID Webscale ecosystem 5

6 2 Introduction to this Lab This lab provides a basic introduction to StorageGRID Webscale (SGWS). The intent is to provide an overview of functionality and terminology for self-education purposes. In this lab, you will learn the basic functionality of StorageGRID Webscale by ingesting and retrieving data using the S3 and Swift object RESTful protocols, and CIFS and NFS NAS protocols. You will be introduced to the Information Lifecycle Management rules and policies, and learn how to configure and protect objects stored in the StorageGRID Webscale system. And finally, you learn how to access the S3 and Swift protocols using 3rd party command line and scripting tools, and how application developers can integrate and use StorageGRID Webscale with their client applications. 2.1 Lab Objectives This lab has the following objectives: Configure StorageGRID Webscale for access by an API client. Create an S3 account and configure the S3 Browser tool. Create a Swift account and configure the Cloudberry Explorer tool. Create custom ILM rules based on S3 buckets. Ingest data using S3 Browser and Swift Cloudberry tools and validate ILM policies have been applied. Enable and test S3 Versioning. Integrate the NAS Bridge, and ingest data using NFS and SMB. Ingest data using command line and scripting tools with the S3 and Swift APIs 2.2 Prerequisites The following prerequisites are recommended for this lab: 6 Basic Linux command line (shell) knowledge. Basic Windows skills. Complete NetAppU WBT course StorageGRID Webscale Administration course, or equivalent. Review the StorageGRID Webscale Primer document, SGWS Administration Guide, and the NAS Bridge Administration Guide.

7 3 Lab Environment StorageGRID Webscale is a software defined object storage solution. All grid nodes are virtual appliances. The following illustration depicts the components of the Lab environment. Figure 3-1: Table 1: Lab Host Information Server \ Resource 7 Purpose IP Address Username Password Jumphost Lab access Administrator Netapp1! NTP Time server, linux shell root Netapp1! DC1-ADM1 StorageGRID admin node root bycast

8 Server \ Resource 8 Purpose IP Address Username Password DC1-S1 StorageGRID storage node root bycast DC1-S2 StorageGRID storage node root bycast DC1-S3 StorageGRID storage node root bycast DC1-G1 StorageGRID API gateway node root bycast DC1-ARC1 StorageGRID archive node root bycast LINUX1 Linux host root Netapp1! NAS Bridge GUI NAS Bridge administrative console admin@netapp.com Netapp1! Admin GUI \ GMI StorageGRID Grid Management Interface (GMI) web GUI hosted from the admin node root netapp01 Audit log export Read only access to audit log Administrator Netapp1! \\ \auditexport

9 4 Lab Activities In this lab you configure the StorageGRID Webscale object storage system for access using the S3 and Swift protocols. You will create Integrated Lifecycle Management (ILM) rules to manage objects ingested through S3 or Swift, and apply policy based on bucket/container name. Webscale s support for the widely adopted S3 protocol ensures interoperability with many existing cloud based applications. Customers and Service Providers can create cloud solutions with on and off premises storage, and leverage Webscale s granular ILM rules to ensure data is stored in the most cost effective manner, while still honoring locality and performance requirements. To accomplish these objectives, the lab provides the following lab activities: Lab Tools Overview on page 9 Connect to Admin GUI on page 10 S3 Account Management on page 14 Configure S3 Client on page 18 Create S3 Buckets on page 21 Examine ILM Rules on page 24 Create New ILM rules on page 25 Activate ILM rules on page 36 Ingest data and Validate Data Placement on page 39 Discover StorageGRID Object ID for sample files Validate ILM policy has been applied on page 42 ILM Re-evaluation on page 46 S3 Versioning on page 52 Configure a Swift Client on page 63 Configure the NAS Bridge on page 68 Exploring S3 and Swift APIs on page 82 Access to S3 through Python on page 86 Access to S3 via the AWS CLI on page 83 Access to Swift Using Python on page Lab Tools Overview Note: At several points in this lab you will use Copy and Paste operations. Please use browser Copy and Paste (right click, select) operations for this purpose. Ctrl+C and Ctrl+V may not work as expected. To facilitate quick access to lab resources the following desktop short cuts have been created. 1. S3 Browser: Client used to perform storage actions (create buckets, ingest, retrieve) using the S3 protocol. 2. Cloudberry Explorer: Client used to perform Swift storage actions (create containers, ingest and retrieve). 3. NetApp StorageGRID Webscale: shortcut to Admin Web GUI. 4. NetApp StorageGRID Webscale NAS Bridge: shortcut to NAS Bridge web interface. 5. PuTTY: SSH client used to access a Linux client for command line, API, and NFS exercises 6. Sample Files: to be used for ingest and ILM tests. 7. Notepad++: use for copy and paste of text strings required for the demo. 8. Webscale Audit Logs: Read only mapped drive to Webscale Audit logs. 9. Downloads: short cut to download folder. Note: The position of these icons may be different on your screen. 9

10 Figure 4-1: 4.2 Connect to Admin GUI The administrative interface to StorageGRID Webscale is called the Grid Management Interface, or GMI. It was formerly called the Network Management System (NMS). Both names are still used, however going forward GMI is preferred. It provides a web- based interface. 1. Double click the shortcut for NetApp StorageGRID Webscale. 10

11 1 Figure 4-2: 2. Your browser will launch, and open the StorageGRID Webscale Grid Management Interface (GMI) 3. Login to the account with Username root, and password netapp01. 11

12 3 Figure 4-3: 4. Review the NMS/GMI Dashboard. 5. Click the Grid button to display the grid topology. 12

13 5 Figure 4-4: 6. Click the + next to Data Center 1 (under LOD Grid) to expand it. The LOD environment consists of a single site SGWS deployment: 3 Storage Nodes, 1 Admin Node, 1 Gateway Node and 1 Archive Node. 13

14 6 Figure 4-5: 4.3 S3 Account Management In this activity you will create an S3 account. Webscales native support for the S3 protocol allows interoperability with Amazon S3 compatible applications. The Account Access Key Identifier and Account Secret Access Key will be familiar to users of AWS S3. 1. Click the Tenants button 14

15 1 Figure 4-6: 2. Click the + Create button to create a new account. 2 Figure 4-7: 3. The Add Tenant Account popup window will launch. 4. Select the S3 protocol. 15

16 4 Figure 4-8: 5. In the Tenant Details, Name, type LOD. 6. Click Save. 5 6 Figure 4-9: 7. The Save Keys dialog appears 8. Click Download. Figure 4-10: 9. When the dialog box appears, select Save File. 10. Click OK. 16

17 9 10 Figure 4-11: 11. A file with a name like Secret Access Key - LOD - Y6DSCN7IV5TOZ5SSQM1B.csv will be downloaded to the Download folder (see shortcut on desktop). You can also copy and paste the Account Access Key Identifier and Account Secret Access Key to a notepad. You will need this information to configure your S3 client Figure 4-12: 12. After the dialog box disappears, (back in the Save Keys dialog) click Finish. 17

18 9 Figure 4-13: 13. The new tenant account will appear on the Tenant Accounts screen. Figure 4-14: 4.4 Configure S3 Client This lab activity uses the S3 Browser. This application provides an easy to use interface for the S3 lab exercises. 1. You will need the IP address of the API Gateway node: (or optionally, the hostname: dc1g1.demo.netapp.com). 2. You can also find this information by clicking on the Grid tab. 18

19 2 Figure 4-15: 3. In the Grid Topology pane, select Data Center 1 > DC1-G1 > SSM > Resources, and scroll down to the Network Addresses section. 3 Figure 4-16: 19

20 You will also need your S3 Account Access Key Identifier, and Account Secret Access key that were downloaded in the previous section. Open the Downloads folder used by the browser. Look for the Secret Access Key*.csv file. Open the credentials file with Notepad. Notice the Access Key ID and Secret Access Key (the longer value is the Secret Access Key). 5 Figure 4-17: 8. Double click the S3 Browser on the desktop. 8 Figure 4-18: 20

21 9. The Add New Account window will launch. 10. Enter LOD for the Account Name. 11. Select S3 compatible storage. 12. Enter the IP address of the Gateway node with port In this lab, this will be :8082, or dc1-g1.demo.netapp.com: Fill in the Access Key ID and Secret Access Key. 14. Click Add new account Figure 4-19: 4.5 Create S3 Buckets In this lab activity you create three S3 buckets using the S3 Browser tool. Later, you will create ILM policies based on these bucket names. 1. Click the New Bucket Icon. 21

22 1 Figure 4-20: 2. Name the bucket 3-copy. 3. Click Create new bucket. 22

23 2 3 Figure 4-21: 4. Repeat the process twice more, to create a bucket named 1-copy, and another named ec-copy. 5. You should now have three buckets: one named 1-copy, one named 3-copy, and one named eccopy. 23

24 5 Figure 4-22: 4.6 Examine ILM Rules Advanced Information Lifecylcle Management policy is a key feature of StorageGRID Webscale. It allows granular evaluation of each object, and then applies storage actions to configure the following: Data protection how many replicated copies or erasure coded objects the grid will create and maintain. Location where these copies will be maintained. In production grid environments, customers create Storage Pools to determine location. In this example you manage objects based on the S3 bucket they are stored in, but administrators can create rules based on other metadata such as object size, or user specified metadata, and last access time. 1. Return to the StorageGRID NMS/GMI user interface. 2. Click on the ILM drop-down button and click on Rules. 24

25 2 Figure 4-23: 3. Examine the default Make 2 Copies rule. This default rule is created with a new Webscale installation, and protects objects that do not meet the criteria for any other rule. Figure 4-24: 4.7 Create New ILM rules In this activity, you create a set of Information Lifecycle Management (ILM) rules. These ILM rules define how objects stored to the StorageGRID Webscale system are protected. You will create rules that apply data protection based on the S3 bucket the object is stored into. You will also create a 3 copy replicate rule for the 3copy bucket, a 1 copy replicate rule for the 1-copy bucket, and two rules for the ec-copy bucket: a erasure coded rule for objects larger than 1 MB, and a replicated rule for smaller objects. 25

26 First, create a 3 copy rule. 1. Click the ILM drop-down button and select Rules (if you are not there already). 2. Click the + Create button. 2 Figure 4-25: 3. Configure the new rule in the Create ILM Rule dialog box. 4. Enter 3-copy for the Rule Name. 5. Enter 3 copy rule as the Description. 6. Leave Ignore as the Tenant Account. 7. For Bucket Name, select Equals. 8. Enter 3-copy. 9. For Object Type select S3/Swift. 10. Click Next. 26

27 Figure 4-26: 11. Set Reference Time to Ingest Time. 12. Set Placements From day 0 store forever. Create 1 copies as replicated in All Storage Nodes. Click +, and add two more Create 1 copies as replicated in All Storage Nodes lines to create 3 copies. Click Refresh on Retention Diagram. Verify 3 copies are created. Click Save.

28 Figure 4-27: Now, create the 1 copy rule. 1. Click the ILM drop-down button and select Rules. 2. Click the + Create button. 3. Configure the new rule in the Create ILM Rule dialog box. 4. Enter 1-copy for the Rule Name. 5. Enter 1 copy rule as the Description. 6. Leave Ignore as the Tenant Account. 7. For Bucket Name, select Equals. 8. Enter 1-copy. 9. For Object Type select S3/Swift. 10. Click Next. 28

29 Figure 4-28: 11. Set Reference Time to Ingest Time. 12. Placements From day 0 store forever. Create 1 copies as replicated in All Storage Nodes 13. Click Refresh on Retention Diagram. 14. Verify 1 copy is created. 15. Click Save. 29

30 Figure 4-29: Now, create an erasure coding (EC) rule. First create the Erasure Coding Profile: 1. Click the ILM drop-down and select Erasure Coding. 1 Figure 4-30: Under Profile Name, enter LOD 2 plus 1. For the Storage Pool, select All Storage Nodes. The EC Scheme is 2+1, as there are only three Storage Nodes in this deployment. Click Apply Changes.

31 Figure 4-31: Perform this activity to create a new EC Copy rule. 1. Click the ILM drop-down button, and select Rules. 2. Click the + Create Button. 3. Configure the new rule in the Create ILM Rule dialog box. 4. Enter EC-copy for the Rule Name. 5. Enter EC copy rule as the Description. 6. Leave Ignore as the Tenant Account. 7. For Bucket Name, select Equals. 8. Enter ec-copy. 9. For Object Type select S3/Swift. 10. Click on Advanced filtering... 31

32 Figure 4-32: 11. In the Advanced Filtering dialog box, select Object Size (MB), greater than and enter 1. See Figure for example. 12. Click Save Figure 4-33: 13. Back in the Create ILM Rule dialog box, click Next. 32

33 13 Figure 4-34: 14. Set reference time to Ingest Time. 15. Placements From day 0 store forever. Create 1 copies as erasure coded in All Storage Nodes (LOD 2 plus 1) 16. Click Refresh on Retention Diagram. 17. Verify Erasure Coded copy is made (use of EC icon) using LOD 2 plus Click Save 33

34 Figure 4-35: Perform this activity to create a new ec-copy-small rule. 1. Click the ILM drop-down button and select Rules. 2. Click the + Create Button. 3. Configure the new rule in the Create ILM Rule dialog box. 4. Enter EC-copy-small for the Rule Name. 5. Enter EC copy small as the Description. 6. Leave Ignore as the Tenant Account. 7. For Bucket Name, select Equals. 8. Enter ec-copy. 9. For Object Type select S3/Swift. 10. Click on Advanced filtering... 34

35 Figure 4-36: 11. In the Advanced Filtering dialog box, select Object Size (MB), less than or equals and enter 1. See Figure for example. 12. Click Save Figure 4-37: 13. Back in the Create ILM Rule dialog box, click Next. 14. Set reference time to Ingest Time. 15. Set Placements. From day 0 store forever. Create 1 copies as replicated in All Storage Nodes. 16. Click Refresh on Retention Diagram. 35

36 Verify 1 copy is created. Add 1 more replicated copy (start by clicking +). Click Refresh on Retention Diagram again. Verify 2 copies will be kept. Click Save Figure 4-38: 4.8 Activate ILM rules Once the ILM rules are created, you need to add the rules to the ILM policy, and activate it. 1. From the ILM drop-down, select Policies. 36

37 1 Figure 4-39: 2. Click Configure from the menu on the left. 3. Check boxes to add the rules to the policy: 1-copy, 3 copy, EC-copy and ec-copy-small. 4. ILM rules are evaluated top down. When a rule s filter is met, it is executed and ILM is satisfied. Only one rule is executed per object. 5. Leave the 2 copy rule as default and at the bottom this will be applied to any objects that do not meet the criteria for other rules. 6. Give the Proposed Policy a name: Webscale lab. 7. Click Save, then click OK in the popup to verify. It may take up to a minute before the Activate button becomes clickable. 8. Click Activate, then click OK in the popup to verify. 37

38 Figure 4-40: Note: Notice the order in which the rules are applied. Rule are evaluated in a top down order to create a data management policy. Only 1 rule is executed per object. Also take note of the Reevaluate Content button. This will re-evaluate data in the system, and apply the new rules. 9. Click Active from the menu on the left. 10. Expand each of the + symbols. 11. Validate that the new rules are active. 38

39 10 9 Figure 4-41: 12. Minimize the StorageGRID window for now. 4.9 Ingest data and Validate Data Placement On the desktop, open the Sample Files folder. Open the S3 Browser. Select the 1-copy bucket. Drag and drop the 1-copy.txt file into the upper right box of S3 Browser, as shown.

40 4 3 Figure 4-42: 5. Repeat for the 3-copy rule. 6. Make sure you select the 3-copy bucket in the S3 browser. 7. Drag and drop 3-copy.txt to S3 Browser like you did with 1-copy.txt. 40

41 7 6 Figure 4-43: 8. Repeat for the EC-copy rule. 9. Make sure you select the ec-copy bucket in the S3 browser. 10. Drag and drop the ec-copy.txt and ec-copy-small.txt files into the EC-copy bucket in the S3 Browser. 11. Minimize the Sample Files and S3 Browser windows. 41

42 10 9 Figure 4-44: 4.10 Validate ILM policy has been applied This activity uses the Object Lookup tool to validate that the ILM rules are applied correctly to an object Return to the StorageGRID Webscale GMI. Select Grid from the top menu bar. In the Grid Topology section, select Data Center 1, then DC1-ADM1. Select CMN > Object Lookup. Select the Configuration tab. If you know the Object Identifier you can enter it here. In this case, you do not yet know the Object Identifier so enter the bucket/key identifier. The tool will look up the Object Identifier for you. Enter 1copy/1-copy.txt into the Object Identifier box Important: Object Identifier lookup by bucket/key values is case sensitve! 7. Click Apply Changes. 42

43 8. Click the Overview tab to see the results Figure 4-45: 9. Scroll down to see all of the information. Note that the object is stored as 1 copy. 9 Figure 4-46: 10. Repeat for 3-copy/3-copy.txt. Notice that the object is stored 3 times. 43

44 10 Figure 4-47: 11. Repeat for ec-copy/ec-copy.txt. Notice that the object is stored with erasure coding in accordance with the ILM rules you set. 44

45 11 Figure 4-48: 12. Repeat for ec-copy/ec-copy-small.txt. Notice that the object is stored twice, in accordance with the ILM rules you set. 45

46 12 Figure 4-49: 4.11 ILM Re-evaluation Another strength of StorageGRID Webscale is the ability to change ILM rules and apply them, not only to new objects, but also to re-evaluate existing objects. If business or regulatory requirements change, customers may need to adjust how their objects are stored. The re-evaluation process takes place in the background to ensure that performance is not impacted. In this activity, you disable the existing 3-copy rule, and create a new 3-copy rule that only makes 2 replicated copies. After you modify the ILM rules and activate the new ILM policy, run the ILM re-evaluation. Once the ILM re-evaluation is complete, you can verify that the object stored in the 3-copy bucket now has only 2 copies. 1. From the ILM drop-down, select Rules. 46

47 1 Figure 4-50: 2. Press the + Create button. 3. In the Create ILM Rule dialog window, enter the following: Name : new-3-copy. Description : New rule: 3-copy bucket - 2 copies only. Tenant Account : Ignore. Bucket Name : Equals 3-copy. For Object Type : S3/Swift. 4. Click Next. 3 4 Figure 4-51: 5. In the new window, enter the following: 47 Set Reference Time : Ingest Time. For Placements: From day 0 store forever. Create 1 copies as replicated in All Storage Nodes.

48 Press the + button, to create a second copy: Create 1 copies as replicated in All Storage Nodes (same as the last one). 6. Refresh the Retention Diagram to confirm Placement. 7. Click Save Figure 4-52: 8. Verify that the new-3-copy rule is created and listed in the ILM Rules list. It should currently not be used in the Active Policy (no checkmark). 9. From the ILM drop-down, select Policies. 48

49 9 8 Figure 4-53: From the left menu, select Configure. Give the Proposed Policy a new name. Enter Webscale Lab-v2 in the Policy field. Under Saved Rules, click on new-3-copy v1.0 under Add to Policy. Unselect the old rule: 3-copy v1.0. If needed, under Proposed Policy, sort the rules into the appropriate order by dragging the hand icon under Actions. The Make 2 Copies v1.0 default rule should always be at the bottom. Rules are evaluated top down. The proposed policy should now contain the new-3-copy rule instead of the old 3-copy rule. Click Save on the proposed policy. Click OK to confirm. It takes some time to save. Once saved, activate the new policy: Press Activate. Click OK to confirm the activation. After about 1 minute, the Re-evaluate Content button will become clickable. Click Re-evaluate Content.

50 Figure 4-54: 21. You will receive a verification pop up after clicking Re-evaluate Content. 22. Click OK. 21 Figure 4-55: 23. Select Grid from the top menu. 24. Navigate to LOD Grid > Data Center 1 > DC1-ADM1 > CMN > Grid Tasks. 25. Verify the ILM Re-evaluation task is running. The time the task takes depends roughly on the amount of content residing in the system. The re-evaluation of existing objects runs as a background task on the grid. 26. The task will move to Historical once completed. Wait for this re-evaluation task to complete before moving to the next step. 50

51 Figure 4-56: After you modify the ILM rules and activate the new ILM policy, run the ILM re-evaluation. Once the ILM reevaluation is complete, you can verify that the object stored in the 3-copy bucket now only has 2 copies Select Grid from the top menu. Navigate to Grid Topology > LOD Grid > Data Center 1 > DC1-ADM1. Expand DC1-ADM1, and select CMN > Object Lookup. Select the Configuration tab. Enter 3-copy/3-copy.txt into the Object identifier box. Click Apply Changes Figure 4-57: 7. Switch to the Overview tab. 51

52 8. Verify that there are now only 2 copies of this object. 7 8 Figure 4-58: 4.12 S3 Versioning In StorageGRID Webscale 10.3, S3 Versioning is supported, this enables managing and storing multiple versions of an object. In this lab, you will exercise the S3 versioning capabilities in StorageGRID Webscale. This lab activity uses the S3 Browser to enable and try S3 versioning. 1. Click and open the S3 Browser from the desktop. 1 Figure 4-59: 2. Create a new bucket called versioning. 3. Leave Bucket region as Default. 4. Click Create New Bucket. 52

53 2 3 4 Figure 4-60: 5. Right click on the versioning bucket and select Edit Versioning Settings. Versioning is enabled on a per bucket basis. 53

54 5 Figure 4-61: 6. Click the Enable versioning for bucket check box. 7. Click OK. Versioning is now enabled for the bucket. 54

55 6 7 Figure 4-62: 8. Now test S3 versioning. Create a new file called version.txt, and store it on the desktop. Inside the file type version 1. Figure 4-63: 9. Drop the version.txt file into the versioning bucket. 55

56 9 Figure 4-64: 10. Click on the Versions tab in the S3 browser. 11. Review the version information under the Versions tab Figure 4-65: 56

57 12. Edit the version.text file located on the Desktop, and change the text to version 2. Drop the updated file into the bucket. 12 Figure 4-66: 13. The S3 browser now displays 2 versions of version.txt residing in the versioning bucket. 13 Figure 4-67: 14. Download the version.txt file. It will return you to version 2. 57

58 14 Figure 4-68: 15. Restore version 1, and download the file: From the Versions tab, select revision # Click the Restore button. 58

59 15 16 Figure 4-69: 17. The S3 Browser may display duplicates in the Versions pane. Press Refresh to get an updated view. 59

60 17 Figure 4-70: 18. Revision 3 should now be displayed. 60

61 18 Figure 4-71: 19. Download and open the versions.txt file. Inside you will now find version 1. 61

62 19 Figure 4-72: 20. You can now also delete old versions. Since revision 1 and 3 are duplicates, you can delete version 1 by selecting it, and pressing Delete. 21. Click Yes. 62

63 20 21 Figure 4-73: 4.13 Configure a Swift Client In this exercise, you will create a container and ingest an object using the Swift protocol. The Cloudberry Explorer client application will be used for this exercise Configure a Swift Account 1. From the Grid Management Interface (GMI), select Tenants from the top menu. 2. Press the + Create button. 1 2 Figure 4-74: 63

64 Select Swift. Enter LOD Swift as the Name under Tenant Details. Record the username as swiftadmin for later use. Enter the password: Netapp1!. Click Save Figure 4-75: 8. From the Tenants Accounts page, record the ID for the LOD Swift account: 8 Figure 4-76: 4.13 Configure Cloudberry Explorer for Swift 1. On the Windows desktop, click the Cloudberry Explorer icon. 64

65 1 Figure 4-77: 2. When the Select Cloud Storage window opens, select OpenStack as the Cloud Storage. 65

66 2 Figure 4-78: 3. When the Add New OpenStack Storage Account window opens, enter LOD Swift as the Display Name. 4. Enter the Username: :swiftadmin. The format of the username is ID:swiftadmin. The ID was created and recorded during the creation of the Swift account. 5. Enter the password created for the Swift account as the API Key. The password is Netapp1!. 6. The Authentication service is the Gateway Node followed by /auth/v1.0. StorageGRID Webscale uses port 8083 for the Swift protocol.. Enter: 7. Select Do not use for the Keystone version. 8. Click Test Connection. The security certificate will be reported as not issued by a trusted certificate authority. Click Allow. 66

67 Figure 4-79: 9. Wait for Connection Success. Click Close. 10. Click OK. 11. Close the Registered Accounts window. 1 2 Figure 4-80: 4.13 To Test Cloudberry Explorer with StorageGRID Webscale On the left pane, select Source : LOD Swift. On the right pane, select Source : My Computer. In the left pane, press the Cube Icon, to create a new container. Name the Container: swifttest. In the left pane, open the swifttest container.

68 Figure 4-81: 6. Copy some files from My Computer to LOD Swift (drag and drop). A sample swifttest.txt file is provided in the Sample Files directory on the Desktop. Since there is no specific rule for the swifttest container, the object will be protected with the default 2 copy rule. Note: The objects ingested into the swifttest container will trigger the 2 copy default rule for ILM. 2 Figure 4-82: 4.14 Configure the NAS Bridge In this exercise, you will integrate the NAS Bridge with the StorageGRID Webscale system. You will configure NFS and CIFS shares and test ingesting files into the NAS shares Configuring the NAS Bridge Perform the following activity to configure the NAS bridge. 1. Connect to the NAS Bridge web interface by selecting the desktop icon NetApp SGWS NAS Bridge 2.0, or opening FireFox and entering 2. Login into the interface by entering admin@netapp.com as the username, and Netapp1! as the password. 3. Agree to the End User License. 68

69 2 Figure 4-83: Configure the DNS Select Bridge from the left menu bar. Click Add next to DNS Servers. Enter the IP address, Enter 1 as the Priority. Click Save Figure 4-84: 69

70 6. Wait until the Status reports READY. 6 Figure 4-85: Configure NTP Servers Select Bridge from the left menu bar. Click Add next to NTP Servers. Enter the host name, dc1.demo.netapp.com. Click Save Figure 4-86: 70

71 5. Wait until the Status reports READY. 5 Figure 4-87: Configure SGWS Integration Select StorageGRID from the left menu bar. Enter hostname: dc1-g1.demo.netapp.com. Enter port Enter your S3 credentials previously created, they can be re-used for the lab. Enter both the Access Key and the Secret Key. Remember: You collected this information back in 3.3 of the lab guide, and it's in a.txt file in your Downloads folder 6. Click Save. 7. Wait unil Status reports READY. 71

72 Figure 4-88: Configure the Active Directory 1. Select Active Directory from the left menu bar. 2. Click Add next to Active Directory Servers. 2 1 Figure 4-89: Enter the Domain : demo.netapp.com Enter Primary Controller : dc1.demo.netapp.com Enter NETBIOS Domain : DEMO Enter Computer name : NAS Enter Admin Username : Administrator Enter Password : Netapp1! Click Save.

73 Figure 4-90: 10. Wait until Status reports Ready. 10 Figure 4-91: Configure the Cache Devices 1. Select Cache Devices from the left menu bar. 2. Click Add next to Cache Devices. 73

74 2 1 Figure 4-92: 3. Enter Name : SMB1, 4. Select Disk : /dev/sdb. 5. Click Save Figure 4-93: 6. Wait until SMB1 status is READY. 74

75 6 Figure 4-94: 7. Repeat these steps using name NFS1, and disk /dev/sdc. 7 Figure 4-95: Configure an NFS Share 1. Select NFS Filesystems from the left menu bar. 2. Click Add next to NFS Filesystems. 75

76 2 1 Figure 4-96: For StorageGRID, select dc1-g1.demo.netapp.com from the drop down list. Enter nfs1 (must be lowercase) as the Bucket name. Select NFS1 as the Cache Device. Enter /nfs1 (remember the forward slash) as the Export path. Click Add Permission Figure 4-97: 8. Enter * (wildcard) for the Client. 9. Click Readwrite. 10. Select No Root for the Squash. 11. Leave Anon UID and Anon GID at default. 12. Click Add. 76

77 Figure 4-98: 13. Click Save. 13 Figure 4-99: 14. Wait until status is READY. 77

78 14 Figure 4-100: Configure an SMB Share 1. Select SMB Filesystems from the left menu bar. 2. Click Add next to SMB Filesystems. 2 1 Figure 4-101: 3. For StorageGRID, select dc1-g1.demo.netapp.com. 4. For Bucket, enter smb1 (all lower case). 5. For Cache Device, select SMB1. 6. For Active Directory, select demo.netapp.com. 7. For Share name, enter smb1. 8. For Owner, leave as the default Administrator. 9. For Group, leave as the default Domain Users. 10. Click Save. 78

79 Figure 4-102: 11. Wait until Status returns READY. 11 Figure 4-103: Test the NFS Filesystem From the Linux Machine, perform the following tasks: 1. Using Putty, ssh to the Linux Machine: linux1.demo.netapp.com. 2. Login: root, password Netapp1!. 3. Enter the following commands: mkdir /mnt/nfs1 mount t nfs o vers=3,rsize= ,wsize= ,hard,intr nas.demo.netapp.com:/mnt/nfs1 / mnt/nfs1 4. Test by copying some files into the mount point: /mnt/nfs1. 79

80 5. Using S3 Browser, you can view the contents of the bucket: nfs1. You may need to wait 1 minute for objects to be stored to SGWS. Test the SMB Filesystem From the Windows Jumphost, perform the following tasks: 1. Open Windows Explorer on the Windows desktop. 2. Right-click This PC, and select Map network drive... 2 Figure 4-104: 3. Select a free Drive Letter. 4. Enter folder: \\nas.demo.netpp.com\smb1 5. Click Finish. 80

81 3 4 5 Figure 4-105: 6. Test by copying some files into the mounted share. 81

82 6 Figure 4-106: 7. Using S3 Browser, you can view the contents of the bucket: smb1. Wait for a minute and refresh S3 Browser's view. 7 Figure 4-107: 4.15 Exploring S3 and Swift APIs In this section, you will learn how to access the S3 and Swift APIs using command line and scripting tools. These exercises demonstrate how client applications can be modified to store data into StorageGRID Webscale using the S3 or Swift APIs. 82

83 Access to S3 via the AWS CLI In this section you will learn how to directly ingest and retrieve objects to/from StorageGRID via the AWS S3 Command Line Interface (CLI). This will demonstrate how users or application developers can programmatically integrate with StorageGRID Webscale using the S3 protocol, to store data into the GRID, or get it back out Persisting Credentials 1. Find the PuTTY icon on the desktop (or in the task bar) and use it to run the PuTTY program. 1 Figure 4-108: 2. In the PuTTY window, choose Linux Client 3. Click Load. 4. Click Open. 83

84 3 2 4 Figure 4-109: 5. A window will open with a log in prompt. Log in with userid root, using Netapp1! as the password. 6. After you log in, configure the S3 account using the command aws configure --profile webscale. Specify the webscale profile. Specify the access key, and the secret access key. Use us-govwest-1 as the default region and json as the output format.. This setting will default to use path style access. Use the S3 Access Key and S3 Secret Access Key that you obtained back in S3 Account Management on page 14. [root@linux1 ~]# aws configure --profile webscale AWS Access Key ID [****************FCX5]: GGJNYGCPXKWSSBQYR8IC AWS Secret Access Key [****************ZgWu]: Mh4Eax22r+lx/kSmj78/yMtheUXuNbbR37dLnGYx Default region name [us-gov-west-1]: us-gov-west-1 Default output format [json]: json [root@linux1 ~]# To verify the configuration, you can view the following files: [root@linux1 ~] cat ~/.aws/credentials The credentials file contains contains S3 Access Key and Secret Access Key credentials, organized by profiles. [webscale] aws_access_key_id = GGJNYGCPXKWSSBQYR8IC aws_secret_access_key = Mh4Eax22r+lx/kSmj78/yMtheUXuNbbR37dLnGYx Using the AWS CLI In this Lab the AWS CLI is already installed. In case you want to install it on your own machine, refer to aws.amazon.com/cli/. 84

85 The SSL/TLS certificate for this StorageGRID instance has already been placed at ~/grid_lod.crt. First, list the existing buckets for our S3 profile webscale : aws s3 ls --profile webscale --endpoint-url --ca-bundle ~/ grid_lod.crt Enter this command into the PuTTY window and observe the output: [root@linux1 ~]# aws s3 ls --profile webscale --endpoint-url --cabundle ~/grid_lod.crt :37:30 1-copy :37:18 3-copy :37:35 ec-copy :50:36 nfs :51:50 smb :13:39 versioning [root@linux1 ~]# Enter this command into the PuTTY window to perform the same operation with SSL verification disabled. [root@linux1 ~]# aws s3 ls --profile webscale --endpoint-url --noverify-ssl /usr/lib/python2.7/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: InsecureRequestWarning) :37:30 1-copy :37:18 3-copy :37:35 ec-copy :50:36 nfs :51:50 smb :13:39 versioning [root@linux1 ~]# Now you will create a new bucket: [root@linux1 ~]# aws s3 mb s3://new-bucket --profile webscale --endpoint-url --ca-bundle ~/grid_lod.crt make_bucket: s3://new-bucket/ [root@linux1 ~]# Next, copy a file into this bucket: aws s3 cp ~/files/1/test.txt s3://new-bucket/ --profile webscale --endpoint-url --ca-bundle ~/grid_lod.crt A subsequent ls operation on the bucket will list the contents: aws s3 ls s3://new-bucket/ --profile webscale --endpoint-url --ca-bundle ~/grid_lod.crt You can use the cp command to copy the object back: aws s3 cp s3://new-bucket/test.txt ~/copied.txt --profile webscale --endpoint-url --ca-bundle ~/grid_lod.crt Now, if you want to sync a whole directory, use the sync command: aws s3 sync ~/files/ s3://new-bucket/ --profile webscale --endpoint-url --ca-bundle ~/grid_lod.crt As you can see, this command did not sync the empty directory 3. This is because S3 does not know about folders as we know them from POSIX based file systems. Nearly all S3 clients only interpret / as a subdirectory for making it more human readable. A full documentation of the supported commands for the AWS S3 CLI can be found at docs.aws.amazon.com/cli/latest/userguide/using-s3-commands.html. Among other interesting commands are: 85

86 Remove an empty bucket: aws s3 rb s3://bucket-name Remove a non-empty bucket: aws s3 rb s3://bucket-name --force Copy a file to an S3 bucket: aws s3 cp file.txt s3://my-bucket/ Sync a whole directory recursively to/from S3 (will re-copy all files with a different size or modified time): aws s3 sync <source path> s3://my-bucket/<target-path> Delete a file in S3: aws s3 rm s3://my-bucket/<path to the file> For the cp, mv or rm command, you can use the --recursive option to walk through all directories, including subdirectories. Another useful tool is S3cmd ( which supports more features than the official AWS CLI Access to S3 through Python In many cases, developers will integrate with StorageGRID Webscale using the S3 API via program code. AWS provides many SDKs for accessing S3 programmatically. StorageGRID Webscale is 100% compatible with the S3 API, thus supports the various S3 SDKs. Some of the supported languages are Java, Ruby, Python, PHP, Go, etc. In this exercise, you will briefly perform basic S3 operations in Python against StorageGRID Webscale. First, configure the Python library ( boto3 ) to only use path-style access. While StorageGRID also supports virtual hostname access, this is not enabled in the lab environment. Edit the following file (previously created by aws): [root@linux1 ~] vi ~/.aws/config Add lines to the file to make it look like this: [profile webscale] s3= addressing_style = path output = json region = us-gov-west-1 Now, create a new file s3.py and paste the following code to access StorageGRID: import boto3 import boto3.session session = boto3.session.session(profile_name='webscale') endpoint = ' s3 = session.resource(service_name='s3', endpoint_url=endpoint, verify='/root/grid_lod.crt') client = s3.meta.client You can execute the Python script by running: [root@linux1 ~] python s3.py As this script only connects to StorageGRID, there will not be any output. 86

87 Next, you can create, list and delete buckets: # Create new bucket s3.bucket('my-bucket').create() # List all buckets for bucket in s3.buckets.all(): print(bucket.name) #Delete bucket s3.bucket('my-bucket').delete() Note that the code might throw errors if the bucket already exists. A new object can be uploaded with the following code: obj = s3.object('my-bucket', 'my-key') obj.upload_file('file/1/test.txt') The object can be downloaded to a file via: obj = s3.object('my-bucket', 'my-key') obj.download_file('target.txt') If desired, you can also add metadata or encryption to the file you upload: obj = s3.object('my-bucket', 'my-key') obj.upload_file('input.txt', ExtraArgs={'Metadata': {'customer_id': '42'}, 'ServerSideEncryption': 'AES256'}) When you look up the object in GMI using the object lookup, you can see that the metadata has persisted. Encryption has also been applied. You can also directly return the metadata, size, and last modification time for an object: obj = print print print s3.object('my-bucket', 'my-key') obj.metadata obj.content_length obj.last_modified And you can also copy objects directly within StorageGRID. This copy process happens within StorageGRID and does not flow through the client: copied_obj = s3.object('<source bucket>','<target key>') copied_obj.copy_from(copysource='/<source bucket>/<source key>') Access to Swift through the CLI In this section you learn how to directly ingest and retrieve objects to/from StorageGRID Webscale through the Python Swift client from the Command Line Interface (CLI). This allows users and application developers to programmatically integrate with StorageGRID Webscale, to put data into the StorageGRID Webscale, or get it back out Using Python Swift Client on the CLI First, log into the Linux Client host with PuTTy. In this Lab setup, the Python Swift client is already installed. In case you want to install it on your own machine, refer to The SSL/TLS certificate for this StorageGRID instance has already been placed at ~/grid_lob.crt. 87

88 Check the version of the Python Swift client: ~] swift version Specify the auth url using the A argument, the swift account using the U argument, and the K argument to specify the password. The SSL/TLS certificate can be specified using the os-cacert argument. The Swift account ID can be copied from the GMI, under Tenants. The stat command provides the status of the specified swift account. [root@linux1 ~]# swift -A -U :swiftadmin -K Netapp1! --os-cacert grid_lod.crt stat You can add the account information into your.bashrc profile: # Swift Connection Settings export ST_AUTH= export ST_USER= :swiftadmin export ST_KEY=Netapp1! And run the swift command with fewer arguments: [root@linux1 ~]# swift --os-cacert grid_lod.crt stat [root@linux1 ~]# swift --os-cacert grid_lod.crt list [root@linux1 ~]# swift --os-cacert grid_lod.crt upload my-container upload.txt [root@linux1 ~]# swift --os-cacert grid_lod.crt download my-container upload.txt Access to Swift Using Python In this exercise, you will perform some basic operations using the Swift API and Python using StorageGRID Webscale as the storage endpoint. 1. Create a new file called swift.py, and paste the following code to access StorageGRID Webscale: The user is the ID for the Swift account created and specified from the NMS. The Authentication URL points to the Gateway Node. The SSL/TLS certificate for this StorageGRID instance has already been placed at ~/grid_lob.crt. import swiftclient user=' :swiftadmin' key='netapp1!' authurl=' cacert='/root/grid_lod.crt' conn=swiftclient.connection( user=user, key=key, authurl=authurl, cacert=cacert ) 2. Execute the Python script by issuing: [root@linux1 ~] python swift.py As this script only connects to StorageGRID Webscale, there will not be any output. 3. Next, create and list containers: # Create a new container container_name='my-container-swift' conn.put_container(container_name) 88

89 #List containers for container_list in conn.get_account()[1]: print container_list['name'] Upload a new object can with the following code, and include specific custom metadata: # Create an object with metadata file_name='swift.txt' with open('swift.txt', 'r') as swift_file: conn.put_object(container_name, file_name, contents=swift_file.read(), content_type='type/plain', headers={'x-object-meta-customerid':'42','x-object-meta-color':'red'}) 4. List the objects in the container: for data in conn.get_container(container_name)[1]: print data['name'] 5. Retrieve the object: # Get object get_obj=conn.get_object(container_name, file_name) with open('new_swift.txt','w') as get_file: get_file.write(get_obj[1]) 6. Delete the object: conn.delete_object(container_name, file_name) 89

90 5 Version History 90 Version Date Document Version History Version 0.9 July 2014 Initial Version Version 1.0 October 2014 Insight 2014 Version Version 1.0 December 2014 Lab on Demand Version Version 2.0 October 2015 Insight 2015 Version Version 3.0 February 2016 Updated for SGWS v10.2 Version 3.1 September 2016 Updated for SGWS v10.3 (Insight 2016 Version) Version 3.1 November 2016 Fix minor typos

91 Refer to the Interoperability Matrix Tool (IMT) on the NetApp Support site to validate that the exact product and feature versions described in this document are supported for your specific environment. The NetApp IMT defines the product components and versions that can be used to construct configurations that are supported by NetApp. Specific results depend on each customer's installation in accordance with published specifications. NetApp provides no representations or warranties regarding the accuracy, reliability, or serviceability of any information or recommendations provided in this publication, or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS, and the use of this information or the implementation of any recommendations or techniques herein is a customer s responsibility and depends on the customer s ability to evaluate and integrate them into the customer s operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document. Go further, faster 2017NetApp, Inc. All rights reserved. No portions of this document may be reproduced without prior written consent of NetApp, Inc. Specifications are subject to change without notice. NetApp, the NetApp logo, Data ONTAP, ONTAP, OnCommand, SANtricity, FlexPod, SnapCenter, and SolidFire are trademarks or registered trademarks of NetApp, Inc. in the United States and/or other countries. All other brands or products are trademarks or registered trademarks of their respective holders and should be treated as such.