Hands-On Lab: NetApp StorageGRID Webscale March 2017 SL10303 Version 3.1

Similar documents
StorageGRID Webscale Installation Guide. For VMware Deployments. January _B0

StorageGRID Webscale 10.2

StorageGRID Webscale 11.0 Expansion Guide

AltaVault Backup to Cloud Setup and Administration. November 2017 SL10337 Version 1.5.3

Zadara Enterprise Storage in

StorageGRID Webscale 10.4 Administrator Guide

StorageGRID Webscale 11.0 Administrator Guide

StorageGRID Webscale 10.3 Software Installation Guide for OpenStack Deployments

Backup Solution. User Guide. Issue 01 Date

StorageGRID Webscale Installation Guide. For VMware Deployments. October _B0

StorageGRID Webscale 11.1 Expansion Guide

StorageGRID Webscale 10.0

StorageGRID Webscale 10.3 Administrator Guide

Tintri Cloud Connector

StorageGRID Webscale 10.1

Commvault Backup to Cloudian Hyperstore CONFIGURATION GUIDE TO USE HYPERSTORE AS A STORAGE LIBRARY

Elastic Cloud Storage (ECS)

StorageGRID Webscale 10.2

StorageGRID Webscale 10.0

NTP Software VFM Administration Web Site

StorageGRID Webscale 10.3 Maintenance Guide for OpenStack Deployments

VMware AirWatch Content Gateway for Linux. VMware Workspace ONE UEM 1811 Unified Access Gateway

VMware vcloud Air User's Guide

Alliance Key Manager AKM for AWS Quick Start Guide. Software version: Documentation version:

REFERENCE ARCHITECTURE Quantum StorNext and Cloudian HyperStore

OnCommand Cloud Manager 3.2 Deploying and Managing ONTAP Cloud Systems

Dell EMC Surveillance for Reveal Body- Worn Camera Systems

Installing and Configuring VMware Identity Manager Connector (Windows) OCT 2018 VMware Identity Manager VMware Identity Manager 3.

SUREedge MIGRATOR INSTALLATION GUIDE FOR VMWARE

StorageGRID Webscale Installation Guide. For Red Hat Enterprise Linux or CentOS Deployments. October _A0

SUSE Enterprise Storage Deployment Guide for Veritas NetBackup Using S3

VMware Identity Manager Administration

VMware AirWatch Content Gateway Guide for Linux For Linux

ArcGIS 10.3 Server on Amazon Web Services

Installing and Configuring vcloud Connector

MarkLogic Server. MarkLogic Server on Microsoft Azure Guide. MarkLogic 9 January, 2018

Guide to Deploying VMware Workspace ONE. VMware Identity Manager VMware AirWatch 9.1

Migrating VMs from VMware vsphere to Oracle Private Cloud Appliance O R A C L E W H I T E P A P E R O C T O B E R

StorageGRID Webscale NAS Bridge Administration Guide

Ansible Tower Quick Setup Guide

Guide to Deploying VMware Workspace ONE. DEC 2017 VMware AirWatch 9.2 VMware Identity Manager 3.1

Setting Up Resources in VMware Identity Manager (On Premises) Modified on 30 AUG 2017 VMware AirWatch 9.1.1

StorageGRID Webscale Installation Guide. For Red Hat Enterprise Linux or CentOS Deployments. October _B0

StorageGRID Webscale 10.3 Troubleshooting Guide

Cluster Management Workflows for OnCommand System Manager

NexentaStor VVOL

Data Fabric Solution for Cloud Backup (SnapCenter for NAS File Services including the Data Fabric) July 2017 SL10374 Version 1.1.0

Integrating AirWatch and VMware Identity Manager

Data Protection Guide

VMware AirWatch Content Gateway for Windows. VMware Workspace ONE UEM 1811 Unified Access Gateway

Setting Up Resources in VMware Identity Manager

SwiftStack and python-swiftclient

Setting Up Resources in VMware Identity Manager. VMware Identity Manager 2.8

NTP Software VFM. Administration Web Site for EMC Atmos User Manual. Version 6.1

Using the Horizon vrealize Orchestrator Plug-In

DCLI User's Guide. Modified on 20 SEP 2018 Data Center Command-Line Interface

ClarityNow Best Practices Guide

StorageGRID Webscale 11.0 Upgrade Guide

SUREedge MIGRATOR INSTALLATION GUIDE FOR HYPERV

StorageGRID Webscale NAS Bridge 2.1 Administration Guide

NetApp Cloud Volumes Service for AWS

Amazon Elastic Compute Cloud

Veritas NetBackup Copilot for Oracle Configuration Guide. Release 3.1 and 3.1.1

Data Protection Guide

Evaluation Guide Host Access Management and Security Server 12.4 SP1 ( )

SUREedge MIGRATOR INSTALLATION GUIDE FOR NUTANIX ACROPOLIS

Azure Marketplace Getting Started Tutorial. Community Edition

Object storage platform How it can help? Martin Lenk, Specialist Senior Systems Engineer Unstructured Data Solution, Dell EMC

scconnect v1.x ADMINISTRATION, INSTALLATION, AND USER GUIDE

Session 1: Accessing MUGrid and Command Line Basics

StorageGRID Installation Guide. For Red Hat Enterprise Linux or CentOS Deployments. February _A0

DCLI User's Guide. Data Center Command-Line Interface 2.9.1

StorageGRID Webscale 11.1 Recovery and Maintenance Guide

Immersion Day. Getting Started with Linux on Amazon EC2

StorageGRID Webscale 10.4 Troubleshooting Guide

DCLI User's Guide. Data Center Command-Line Interface

Guide to Deploying VMware Workspace ONE with VMware Identity Manager. SEP 2018 VMware Workspace ONE

VMware AirWatch Content Gateway Guide for Windows

Object Storage Service. Client Guide (OBS Browser) Issue 10 Date HUAWEI TECHNOLOGIES CO., LTD.

Object Storage Service. Product Introduction. Issue 04 Date HUAWEI TECHNOLOGIES CO., LTD.

Migrating vrealize Automation 6.2 to 7.2

VMware AirWatch Content Gateway Guide for Windows

Third-Party Client (s3fs) User Guide

HySecure Quick Start Guide. HySecure 5.0

Paperspace. Deployment Guide. Cloud VDI. 20 Jay St. Suite 312 Brooklyn, NY Technical Whitepaper

VMware Identity Manager Administration. MAY 2018 VMware Identity Manager 3.2

Dell EMC Surveillance for IndigoVision Body-Worn Cameras

Setting Up the DR Series System as an NFS Target on Amanda Enterprise 3.3.5

Azure Marketplace. Getting Started Tutorial. Community Edition

Puppet on the AWS Cloud

ForeScout CounterACT. (AWS) Plugin. Configuration Guide. Version 1.3

StorageGRID Webscale 10.0

Data Fabric Solution for Cloud Backup. February 2017 SL10316 Version 1.0.1

Single Sign-On for PCF. User's Guide

StorageGRID Webscale Installation Guide. For Ubuntu or Debian Deployments. October _B0

VMware AirWatch - Workspace ONE, Single Sign-on and VMware Identity Manager

StorageGRID Webscale 11.0 Tenant Administrator Guide

AWS Administration. Suggested Pre-requisites Basic IT Knowledge

vrealize Suite Lifecycle Manager 1.0 Installation and Management vrealize Suite 2017

EMC Solution for VIEVU Body Worn Cameras

Transcription:

Hands-On Lab: NetApp StorageGRID Webscale 10.3 March 2017 SL10303 Version 3.1

TABLE OF CONTENTS 1 Introduction to Object Storage... 3 2 Introduction to this Lab... 6 2.1 Lab Objectives... 6 2.2 Prerequisites... 6 3 Lab Environment... 7 4 Lab Activities... 9 4.1 Lab Tools Overview...9 4.2 Connect to Admin GUI...10 4.3 S3 Account Management...14 4.4 Configure S3 Client... 18 4.5 Create S3 Buckets...21 4.6 Examine ILM Rules...24 4.7 Create New ILM rules...25 4.8 Activate ILM rules...36 4.9 Ingest data and Validate Data Placement... 39 4.10 Validate ILM policy has been applied...42 4.11 ILM Re-evaluation... 46 4.12 S3 Versioning... 52 4.13 Configure a Swift Client...63 4.14 Configure the NAS Bridge...68 4.15 Exploring S3 and Swift APIs... 82 4.15.1 Access to S3 via the AWS CLI... 83 4.15.2 Access to S3 through Python...86 4.15.3 Access to Swift through the CLI...87 4.15.4 Access to Swift Using Python... 88 5 Version History... 90 2

1 Introduction to Object Storage Object storage is a little different than the familiar block and file storage. It organizes data into flexibly sized data containers called objects. Objects are stored in a flat namespace that may span multiple locations. Each object has both data (an un-interpreted sequence of bytes), and metadata (a unique ID plus an extensible set of attributes that describe the object). Data stored using object storage is typically accessed by a client application using an object protocol or API, rather than at the user level (accessing a file). A simple way to think about this is that object storage is like valet parking you give the valet your ticket, and you get your car back without ever needing to know anything about where the car was parked. The advantage of this approach is that data can be referenced and retrieved just by providing the unique ID. Figure 1-1: Object Storage Concepts Three trends are contributing to increasing interest in object storage: Continued growth in the amount of unstructured data necessitates a new approach to storing and protecting data. Object-enabled data management facilitates intelligent data placement to meet a wide range of performance, durability, availability, location, and longevity requirements. Enterprises are increasingly coping with decentralized data creation and consumption. The "primary data center plus DR data center" model is being replaced by a multisite approach in which users, workloads, and data are brought closer together. The hybrid cloud is providing new options to balance cost and performance, and IT teams are looking at how they can best leverage both on-premises and cloud-based storage. The Opportunity: Storage growth to billions of objects. Decentralized data production and global distribution is driving new data management approaches to reduce operation costs and complexity. Due to data sovereignity, performance, cost, or a combination of all three, a hybrid cloud with both onpremises and clooud-based storage is a requirement. Use Cases StorageGRID Webscale is extremely well suited for web data repositories, data archives, and media repositories. Each of these use cases has its own distinct set of requirements, but StorageGRID Webscale adapts to accommodate the wide variety of needs encompassed in this set of use cases. 3

Figure 1-2: Use Cases for Object Storage Web Data Repositories Web data repositories are characterized by small object size, high object count, and high transactions. Because it can handle up to 100 billion objects in a single repository distributed across many locations and with S3, Swift and CDMI API support, StorageGRID Webscale is well suited for this use case. Data Archives Increasingly, enterprises are storing massive amounts of data for extended periods to satisfy both corporate governance and legal requirements. With data archives of this type, cost and management are typically the most important concerns. Long access latency is tolerated in exchange for reduced cost. StorageGRID Webscale satisfies this use case with tape integration, proven data durability, and flexible, policy based management. Media Repositories Media repositories are characterized by large object sizes, with the need for geographical distribution, a need for data integrity, and a low time-to-first byte latency. The geographically distributed, durable design of StorageGRID Webscale satisfies these requirements. It also supports ranged reads, so, for example, a video can be streamed from any point without having to download the entire object. Introduction to StorageGRID Webscale The StorageGRID Webscale system is a distributed object storage system that stores, protects, and preserves fixed-content data over long periods of time. By employing a grid architecture that distributes copies of object data throughout the system, a highly reliable system is created where data is continuously available. If one part of the system goes down, another immediately takes over, which results in objects always being available for retrieval. To implement this architecture, the StorageGRID Webscale system employs a system of network connected servers hosting grid nodes. These grid nodes host a collection of one or more services, each providing a set of capabilities to the StorageGRID Webscale system. To manage objects ingested into the system, the StorageGRID Webscale system employs metadata based information lifecycle management (ILM) rules. These ILM rules determine what happens to an object s data after it is ingested where it is stored, how it is protected from loss, and how long it is stored. The StorageGRID Webscale system operates over wide area network (WAN) links, providing the system with off-site loss protection. Copies are made and distributed throughout the system so that objects are continuously available. In systems with multiple sites, this distribution of copies means that if a site is lost, data is not lost, and clients can seamlessly retrieve it from other sites. Advantages of the StorageGRID Webscale system include the following: 4 Software-defined storage management for hardware independence Support for standard object storage protocols and APIs:

Simple Storage Service (S3) OpenStack Swift Cloud Data Management Interface (CDMI) Support for standard file system storage protocols: Network File System (NFS) Server Message Block (SMB) Configurable data protection using replicated copies, erasure coding or cloud tiering. Ability to address hardware obsolescence through the transparent migration of data from previous to current generations of hardware. Policy-driven tiered storage that reduces overall storage costs: Data can be migrated to lower cost tier based on last access time or other value. An object stored on a low cost tier can be copied to faster tier once accessed. Consider a policy that stores data on tape or public cloud a policy to copy frequently accessed data to a replica provides user driven cache effect. Virtualization of storage across sites and tiers, insulating applications from changes to underlying storage infrastructure. Ability to monitor and verify data integrity proactively as data is stored, updated, retrieved, and deleted. Non-disruptive upgrade, expansion, and maintenance that maintains access to content StorageGRID Webscale Partners StorageGRID Webscale can work with client applications that support storing to S3 or Swift compatible storage. Figure 1-3: Sample Partners and Client Applications in the StorageGRID Webscale ecosystem 5

2 Introduction to this Lab This lab provides a basic introduction to StorageGRID Webscale (SGWS). The intent is to provide an overview of functionality and terminology for self-education purposes. In this lab, you will learn the basic functionality of StorageGRID Webscale by ingesting and retrieving data using the S3 and Swift object RESTful protocols, and CIFS and NFS NAS protocols. You will be introduced to the Information Lifecycle Management rules and policies, and learn how to configure and protect objects stored in the StorageGRID Webscale system. And finally, you learn how to access the S3 and Swift protocols using 3rd party command line and scripting tools, and how application developers can integrate and use StorageGRID Webscale with their client applications. 2.1 Lab Objectives This lab has the following objectives: Configure StorageGRID Webscale for access by an API client. Create an S3 account and configure the S3 Browser tool. Create a Swift account and configure the Cloudberry Explorer tool. Create custom ILM rules based on S3 buckets. Ingest data using S3 Browser and Swift Cloudberry tools and validate ILM policies have been applied. Enable and test S3 Versioning. Integrate the NAS Bridge, and ingest data using NFS and SMB. Ingest data using command line and scripting tools with the S3 and Swift APIs 2.2 Prerequisites The following prerequisites are recommended for this lab: 6 Basic Linux command line (shell) knowledge. Basic Windows skills. Complete NetAppU WBT course StorageGRID Webscale Administration course, or equivalent. Review the StorageGRID Webscale Primer document, SGWS Administration Guide, and the NAS Bridge Administration Guide.

3 Lab Environment StorageGRID Webscale is a software defined object storage solution. All grid nodes are virtual appliances. The following illustration depicts the components of the Lab environment. Figure 3-1: Table 1: Lab Host Information Server \ Resource 7 Purpose IP Address Username Password Jumphost Lab access 192.168.0.5 Administrator Netapp1! NTP Time server, linux shell 192.168.0.9 root Netapp1! DC1-ADM1 StorageGRID admin node 192.168.0.10 root bycast

Server \ Resource 8 Purpose IP Address Username Password DC1-S1 StorageGRID storage node 192.168.0.6 root bycast DC1-S2 StorageGRID storage node 192.168.0.7 root bycast DC1-S3 StorageGRID storage node 192.168.0.8 root bycast DC1-G1 StorageGRID API gateway node 192.168.0.11 root bycast DC1-ARC1 StorageGRID archive node 192.168.0.12 root bycast LINUX1 Linux host 192.168.0.13 root Netapp1! NAS Bridge GUI NAS Bridge administrative console 192.168.0.14 admin@netapp.com Netapp1! Admin GUI \ GMI StorageGRID Grid Management Interface (GMI) web GUI hosted from the admin node 192.168.0.10 root netapp01 Audit log export Read only access to audit log Administrator Netapp1! \\192.168.0.10\auditexport

4 Lab Activities In this lab you configure the StorageGRID Webscale object storage system for access using the S3 and Swift protocols. You will create Integrated Lifecycle Management (ILM) rules to manage objects ingested through S3 or Swift, and apply policy based on bucket/container name. Webscale s support for the widely adopted S3 protocol ensures interoperability with many existing cloud based applications. Customers and Service Providers can create cloud solutions with on and off premises storage, and leverage Webscale s granular ILM rules to ensure data is stored in the most cost effective manner, while still honoring locality and performance requirements. To accomplish these objectives, the lab provides the following lab activities: Lab Tools Overview on page 9 Connect to Admin GUI on page 10 S3 Account Management on page 14 Configure S3 Client on page 18 Create S3 Buckets on page 21 Examine ILM Rules on page 24 Create New ILM rules on page 25 Activate ILM rules on page 36 Ingest data and Validate Data Placement on page 39 Discover StorageGRID Object ID for sample files Validate ILM policy has been applied on page 42 ILM Re-evaluation on page 46 S3 Versioning on page 52 Configure a Swift Client on page 63 Configure the NAS Bridge on page 68 Exploring S3 and Swift APIs on page 82 Access to S3 through Python on page 86 Access to S3 via the AWS CLI on page 83 Access to Swift Using Python on page 88 4.1 Lab Tools Overview Note: At several points in this lab you will use Copy and Paste operations. Please use browser Copy and Paste (right click, select) operations for this purpose. Ctrl+C and Ctrl+V may not work as expected. To facilitate quick access to lab resources the following desktop short cuts have been created. 1. S3 Browser: Client used to perform storage actions (create buckets, ingest, retrieve) using the S3 protocol. 2. Cloudberry Explorer: Client used to perform Swift storage actions (create containers, ingest and retrieve). 3. NetApp StorageGRID Webscale: shortcut to Admin Web GUI. 4. NetApp StorageGRID Webscale NAS Bridge: shortcut to NAS Bridge web interface. 5. PuTTY: SSH client used to access a Linux client for command line, API, and NFS exercises 6. Sample Files: to be used for ingest and ILM tests. 7. Notepad++: use for copy and paste of text strings required for the demo. 8. Webscale Audit Logs: Read only mapped drive to Webscale Audit logs. 9. Downloads: short cut to download folder. Note: The position of these icons may be different on your screen. 9

2 1 4 3 5 6 8 7 9 Figure 4-1: 4.2 Connect to Admin GUI The administrative interface to StorageGRID Webscale is called the Grid Management Interface, or GMI. It was formerly called the Network Management System (NMS). Both names are still used, however going forward GMI is preferred. It provides a web- based interface. 1. Double click the shortcut for NetApp StorageGRID Webscale. 10

1 Figure 4-2: 2. Your browser will launch, and open the StorageGRID Webscale Grid Management Interface (GMI) 3. Login to the account with Username root, and password netapp01. 11

3 Figure 4-3: 4. Review the NMS/GMI Dashboard. 5. Click the Grid button to display the grid topology. 12

5 Figure 4-4: 6. Click the + next to Data Center 1 (under LOD Grid) to expand it. The LOD environment consists of a single site SGWS deployment: 3 Storage Nodes, 1 Admin Node, 1 Gateway Node and 1 Archive Node. 13

6 Figure 4-5: 4.3 S3 Account Management In this activity you will create an S3 account. Webscales native support for the S3 protocol allows interoperability with Amazon S3 compatible applications. The Account Access Key Identifier and Account Secret Access Key will be familiar to users of AWS S3. 1. Click the Tenants button 14

1 Figure 4-6: 2. Click the + Create button to create a new account. 2 Figure 4-7: 3. The Add Tenant Account popup window will launch. 4. Select the S3 protocol. 15

4 Figure 4-8: 5. In the Tenant Details, Name, type LOD. 6. Click Save. 5 6 Figure 4-9: 7. The Save Keys dialog appears 8. Click Download. Figure 4-10: 9. When the dialog box appears, select Save File. 10. Click OK. 16

9 10 Figure 4-11: 11. A file with a name like Secret Access Key - LOD - Y6DSCN7IV5TOZ5SSQM1B.csv will be downloaded to the Download folder (see shortcut on desktop). You can also copy and paste the Account Access Key Identifier and Account Secret Access Key to a notepad. You will need this information to configure your S3 client. 9 10 Figure 4-12: 12. After the dialog box disappears, (back in the Save Keys dialog) click Finish. 17

9 Figure 4-13: 13. The new tenant account will appear on the Tenant Accounts screen. Figure 4-14: 4.4 Configure S3 Client This lab activity uses the S3 Browser. This application provides an easy to use interface for the S3 lab exercises. 1. You will need the IP address of the API Gateway node: 192.168.0.11 (or optionally, the hostname: dc1g1.demo.netapp.com). 2. You can also find this information by clicking on the Grid tab. 18

2 Figure 4-15: 3. In the Grid Topology pane, select Data Center 1 > DC1-G1 > SSM > Resources, and scroll down to the Network Addresses section. 3 Figure 4-16: 19

4. 5. 6. 7. You will also need your S3 Account Access Key Identifier, and Account Secret Access key that were downloaded in the previous section. Open the Downloads folder used by the browser. Look for the Secret Access Key*.csv file. Open the credentials file with Notepad. Notice the Access Key ID and Secret Access Key (the longer value is the Secret Access Key). 5 Figure 4-17: 8. Double click the S3 Browser on the desktop. 8 Figure 4-18: 20

9. The Add New Account window will launch. 10. Enter LOD for the Account Name. 11. Select S3 compatible storage. 12. Enter the IP address of the Gateway node with port 8082. In this lab, this will be 192.168.0.11:8082, or dc1-g1.demo.netapp.com:8082. 13. Fill in the Access Key ID and Secret Access Key. 14. Click Add new account. 10 11 12 13 14 Figure 4-19: 4.5 Create S3 Buckets In this lab activity you create three S3 buckets using the S3 Browser tool. Later, you will create ILM policies based on these bucket names. 1. Click the New Bucket Icon. 21

1 Figure 4-20: 2. Name the bucket 3-copy. 3. Click Create new bucket. 22

2 3 Figure 4-21: 4. Repeat the process twice more, to create a bucket named 1-copy, and another named ec-copy. 5. You should now have three buckets: one named 1-copy, one named 3-copy, and one named eccopy. 23

5 Figure 4-22: 4.6 Examine ILM Rules Advanced Information Lifecylcle Management policy is a key feature of StorageGRID Webscale. It allows granular evaluation of each object, and then applies storage actions to configure the following: Data protection how many replicated copies or erasure coded objects the grid will create and maintain. Location where these copies will be maintained. In production grid environments, customers create Storage Pools to determine location. In this example you manage objects based on the S3 bucket they are stored in, but administrators can create rules based on other metadata such as object size, or user specified metadata, and last access time. 1. Return to the StorageGRID NMS/GMI user interface. 2. Click on the ILM drop-down button and click on Rules. 24

2 Figure 4-23: 3. Examine the default Make 2 Copies rule. This default rule is created with a new Webscale installation, and protects objects that do not meet the criteria for any other rule. Figure 4-24: 4.7 Create New ILM rules In this activity, you create a set of Information Lifecycle Management (ILM) rules. These ILM rules define how objects stored to the StorageGRID Webscale system are protected. You will create rules that apply data protection based on the S3 bucket the object is stored into. You will also create a 3 copy replicate rule for the 3copy bucket, a 1 copy replicate rule for the 1-copy bucket, and two rules for the ec-copy bucket: a erasure coded rule for objects larger than 1 MB, and a replicated rule for smaller objects. 25

First, create a 3 copy rule. 1. Click the ILM drop-down button and select Rules (if you are not there already). 2. Click the + Create button. 2 Figure 4-25: 3. Configure the new rule in the Create ILM Rule dialog box. 4. Enter 3-copy for the Rule Name. 5. Enter 3 copy rule as the Description. 6. Leave Ignore as the Tenant Account. 7. For Bucket Name, select Equals. 8. Enter 3-copy. 9. For Object Type select S3/Swift. 10. Click Next. 26

4 5 6 7 8 9 10 Figure 4-26: 11. Set Reference Time to Ingest Time. 12. Set Placements 13. 14. 15. 16. 27 From day 0 store forever. Create 1 copies as replicated in All Storage Nodes. Click +, and add two more Create 1 copies as replicated in All Storage Nodes lines to create 3 copies. Click Refresh on Retention Diagram. Verify 3 copies are created. Click Save.

11 12 13 14 16 15 Figure 4-27: Now, create the 1 copy rule. 1. Click the ILM drop-down button and select Rules. 2. Click the + Create button. 3. Configure the new rule in the Create ILM Rule dialog box. 4. Enter 1-copy for the Rule Name. 5. Enter 1 copy rule as the Description. 6. Leave Ignore as the Tenant Account. 7. For Bucket Name, select Equals. 8. Enter 1-copy. 9. For Object Type select S3/Swift. 10. Click Next. 28

4 5 6 8 7 9 10 Figure 4-28: 11. Set Reference Time to Ingest Time. 12. Placements From day 0 store forever. Create 1 copies as replicated in All Storage Nodes 13. Click Refresh on Retention Diagram. 14. Verify 1 copy is created. 15. Click Save. 29

11 12 13 15 14 Figure 4-29: Now, create an erasure coding (EC) rule. First create the Erasure Coding Profile: 1. Click the ILM drop-down and select Erasure Coding. 1 Figure 4-30: 2. 3. 4. 5. 30 Under Profile Name, enter LOD 2 plus 1. For the Storage Pool, select All Storage Nodes. The EC Scheme is 2+1, as there are only three Storage Nodes in this deployment. Click Apply Changes.

2 3 4 5 Figure 4-31: Perform this activity to create a new EC Copy rule. 1. Click the ILM drop-down button, and select Rules. 2. Click the + Create Button. 3. Configure the new rule in the Create ILM Rule dialog box. 4. Enter EC-copy for the Rule Name. 5. Enter EC copy rule as the Description. 6. Leave Ignore as the Tenant Account. 7. For Bucket Name, select Equals. 8. Enter ec-copy. 9. For Object Type select S3/Swift. 10. Click on Advanced filtering... 31

4 5 6 8 7 9 10 Figure 4-32: 11. In the Advanced Filtering dialog box, select Object Size (MB), greater than and enter 1. See Figure for example. 12. Click Save. 11 12 Figure 4-33: 13. Back in the Create ILM Rule dialog box, click Next. 32

13 Figure 4-34: 14. Set reference time to Ingest Time. 15. Placements From day 0 store forever. Create 1 copies as erasure coded in All Storage Nodes (LOD 2 plus 1) 16. Click Refresh on Retention Diagram. 17. Verify Erasure Coded copy is made (use of EC icon) using LOD 2 plus 1. 18. Click Save 33

14 15 16 18 17 Figure 4-35: Perform this activity to create a new ec-copy-small rule. 1. Click the ILM drop-down button and select Rules. 2. Click the + Create Button. 3. Configure the new rule in the Create ILM Rule dialog box. 4. Enter EC-copy-small for the Rule Name. 5. Enter EC copy small as the Description. 6. Leave Ignore as the Tenant Account. 7. For Bucket Name, select Equals. 8. Enter ec-copy. 9. For Object Type select S3/Swift. 10. Click on Advanced filtering... 34

4 5 6 7 8 9 10 Figure 4-36: 11. In the Advanced Filtering dialog box, select Object Size (MB), less than or equals and enter 1. See Figure for example. 12. Click Save. 11 12 Figure 4-37: 13. Back in the Create ILM Rule dialog box, click Next. 14. Set reference time to Ingest Time. 15. Set Placements. From day 0 store forever. Create 1 copies as replicated in All Storage Nodes. 16. Click Refresh on Retention Diagram. 35

17. 18. 19. 20. 21. Verify 1 copy is created. Add 1 more replicated copy (start by clicking +). Click Refresh on Retention Diagram again. Verify 2 copies will be kept. Click Save. 14 15 18 19 16 21 Figure 4-38: 4.8 Activate ILM rules Once the ILM rules are created, you need to add the rules to the ILM policy, and activate it. 1. From the ILM drop-down, select Policies. 36

1 Figure 4-39: 2. Click Configure from the menu on the left. 3. Check boxes to add the rules to the policy: 1-copy, 3 copy, EC-copy and ec-copy-small. 4. ILM rules are evaluated top down. When a rule s filter is met, it is executed and ILM is satisfied. Only one rule is executed per object. 5. Leave the 2 copy rule as default and at the bottom this will be applied to any objects that do not meet the criteria for other rules. 6. Give the Proposed Policy a name: Webscale lab. 7. Click Save, then click OK in the popup to verify. It may take up to a minute before the Activate button becomes clickable. 8. Click Activate, then click OK in the popup to verify. 37

2 3 6 5 7 8 Figure 4-40: Note: Notice the order in which the rules are applied. Rule are evaluated in a top down order to create a data management policy. Only 1 rule is executed per object. Also take note of the Reevaluate Content button. This will re-evaluate data in the system, and apply the new rules. 9. Click Active from the menu on the left. 10. Expand each of the + symbols. 11. Validate that the new rules are active. 38

10 9 Figure 4-41: 12. Minimize the StorageGRID window for now. 4.9 Ingest data and Validate Data Placement 1. 2. 3. 4. 39 On the desktop, open the Sample Files folder. Open the S3 Browser. Select the 1-copy bucket. Drag and drop the 1-copy.txt file into the upper right box of S3 Browser, as shown.

4 3 Figure 4-42: 5. Repeat for the 3-copy rule. 6. Make sure you select the 3-copy bucket in the S3 browser. 7. Drag and drop 3-copy.txt to S3 Browser like you did with 1-copy.txt. 40

7 6 Figure 4-43: 8. Repeat for the EC-copy rule. 9. Make sure you select the ec-copy bucket in the S3 browser. 10. Drag and drop the ec-copy.txt and ec-copy-small.txt files into the EC-copy bucket in the S3 Browser. 11. Minimize the Sample Files and S3 Browser windows. 41

10 9 Figure 4-44: 4.10 Validate ILM policy has been applied This activity uses the Object Lookup tool to validate that the ILM rules are applied correctly to an object. 1. 2. 3. 4. 5. 6. Return to the StorageGRID Webscale GMI. Select Grid from the top menu bar. In the Grid Topology section, select Data Center 1, then DC1-ADM1. Select CMN > Object Lookup. Select the Configuration tab. If you know the Object Identifier you can enter it here. In this case, you do not yet know the Object Identifier so enter the bucket/key identifier. The tool will look up the Object Identifier for you. Enter 1copy/1-copy.txt into the Object Identifier box Important: Object Identifier lookup by bucket/key values is case sensitve! 7. Click Apply Changes. 42

8. Click the Overview tab to see the results. 8 2 5 3 6 4 7 Figure 4-45: 9. Scroll down to see all of the information. Note that the object is stored as 1 copy. 9 Figure 4-46: 10. Repeat for 3-copy/3-copy.txt. Notice that the object is stored 3 times. 43

10 Figure 4-47: 11. Repeat for ec-copy/ec-copy.txt. Notice that the object is stored with erasure coding in accordance with the ILM rules you set. 44

11 Figure 4-48: 12. Repeat for ec-copy/ec-copy-small.txt. Notice that the object is stored twice, in accordance with the ILM rules you set. 45

12 Figure 4-49: 4.11 ILM Re-evaluation Another strength of StorageGRID Webscale is the ability to change ILM rules and apply them, not only to new objects, but also to re-evaluate existing objects. If business or regulatory requirements change, customers may need to adjust how their objects are stored. The re-evaluation process takes place in the background to ensure that performance is not impacted. In this activity, you disable the existing 3-copy rule, and create a new 3-copy rule that only makes 2 replicated copies. After you modify the ILM rules and activate the new ILM policy, run the ILM re-evaluation. Once the ILM re-evaluation is complete, you can verify that the object stored in the 3-copy bucket now has only 2 copies. 1. From the ILM drop-down, select Rules. 46

1 Figure 4-50: 2. Press the + Create button. 3. In the Create ILM Rule dialog window, enter the following: Name : new-3-copy. Description : New rule: 3-copy bucket - 2 copies only. Tenant Account : Ignore. Bucket Name : Equals 3-copy. For Object Type : S3/Swift. 4. Click Next. 3 4 Figure 4-51: 5. In the new window, enter the following: 47 Set Reference Time : Ingest Time. For Placements: From day 0 store forever. Create 1 copies as replicated in All Storage Nodes.

Press the + button, to create a second copy: Create 1 copies as replicated in All Storage Nodes (same as the last one). 6. Refresh the Retention Diagram to confirm Placement. 7. Click Save. 5 6 7 Figure 4-52: 8. Verify that the new-3-copy rule is created and listed in the ILM Rules list. It should currently not be used in the Active Policy (no checkmark). 9. From the ILM drop-down, select Policies. 48

9 8 Figure 4-53: 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 49 From the left menu, select Configure. Give the Proposed Policy a new name. Enter Webscale Lab-v2 in the Policy field. Under Saved Rules, click on new-3-copy v1.0 under Add to Policy. Unselect the old rule: 3-copy v1.0. If needed, under Proposed Policy, sort the rules into the appropriate order by dragging the hand icon under Actions. The Make 2 Copies v1.0 default rule should always be at the bottom. Rules are evaluated top down. The proposed policy should now contain the new-3-copy rule instead of the old 3-copy rule. Click Save on the proposed policy. Click OK to confirm. It takes some time to save. Once saved, activate the new policy: Press Activate. Click OK to confirm the activation. After about 1 minute, the Re-evaluate Content button will become clickable. Click Re-evaluate Content.

10 13 12 11 14 15 17 20 Figure 4-54: 21. You will receive a verification pop up after clicking Re-evaluate Content. 22. Click OK. 21 Figure 4-55: 23. Select Grid from the top menu. 24. Navigate to LOD Grid > Data Center 1 > DC1-ADM1 > CMN > Grid Tasks. 25. Verify the ILM Re-evaluation task is running. The time the task takes depends roughly on the amount of content residing in the system. The re-evaluation of existing objects runs as a background task on the grid. 26. The task will move to Historical once completed. Wait for this re-evaluation task to complete before moving to the next step. 50

23 24 25 Figure 4-56: After you modify the ILM rules and activate the new ILM policy, run the ILM re-evaluation. Once the ILM reevaluation is complete, you can verify that the object stored in the 3-copy bucket now only has 2 copies. 1. 2. 3. 4. 5. 6. Select Grid from the top menu. Navigate to Grid Topology > LOD Grid > Data Center 1 > DC1-ADM1. Expand DC1-ADM1, and select CMN > Object Lookup. Select the Configuration tab. Enter 3-copy/3-copy.txt into the Object identifier box. Click Apply Changes. 1 4 2 3 5 6 Figure 4-57: 7. Switch to the Overview tab. 51

8. Verify that there are now only 2 copies of this object. 7 8 Figure 4-58: 4.12 S3 Versioning In StorageGRID Webscale 10.3, S3 Versioning is supported, this enables managing and storing multiple versions of an object. In this lab, you will exercise the S3 versioning capabilities in StorageGRID Webscale. This lab activity uses the S3 Browser to enable and try S3 versioning. 1. Click and open the S3 Browser from the desktop. 1 Figure 4-59: 2. Create a new bucket called versioning. 3. Leave Bucket region as Default. 4. Click Create New Bucket. 52

2 3 4 Figure 4-60: 5. Right click on the versioning bucket and select Edit Versioning Settings. Versioning is enabled on a per bucket basis. 53

5 Figure 4-61: 6. Click the Enable versioning for bucket check box. 7. Click OK. Versioning is now enabled for the bucket. 54

6 7 Figure 4-62: 8. Now test S3 versioning. Create a new file called version.txt, and store it on the desktop. Inside the file type version 1. Figure 4-63: 9. Drop the version.txt file into the versioning bucket. 55

9 Figure 4-64: 10. Click on the Versions tab in the S3 browser. 11. Review the version information under the Versions tab. 10 11 Figure 4-65: 56

12. Edit the version.text file located on the Desktop, and change the text to version 2. Drop the updated file into the bucket. 12 Figure 4-66: 13. The S3 browser now displays 2 versions of version.txt residing in the versioning bucket. 13 Figure 4-67: 14. Download the version.txt file. It will return you to version 2. 57

14 Figure 4-68: 15. Restore version 1, and download the file: From the Versions tab, select revision #1. 16. Click the Restore button. 58

15 16 Figure 4-69: 17. The S3 Browser may display duplicates in the Versions pane. Press Refresh to get an updated view. 59

17 Figure 4-70: 18. Revision 3 should now be displayed. 60

18 Figure 4-71: 19. Download and open the versions.txt file. Inside you will now find version 1. 61

19 Figure 4-72: 20. You can now also delete old versions. Since revision 1 and 3 are duplicates, you can delete version 1 by selecting it, and pressing Delete. 21. Click Yes. 62

20 21 Figure 4-73: 4.13 Configure a Swift Client In this exercise, you will create a container and ingest an object using the Swift protocol. The Cloudberry Explorer client application will be used for this exercise. 4.13 Configure a Swift Account 1. From the Grid Management Interface (GMI), select Tenants from the top menu. 2. Press the + Create button. 1 2 Figure 4-74: 63

3. 4. 5. 6. 7. Select Swift. Enter LOD Swift as the Name under Tenant Details. Record the username as swiftadmin for later use. Enter the password: Netapp1!. Click Save. 3 4 6 7 Figure 4-75: 8. From the Tenants Accounts page, record the ID for the LOD Swift account: 8 Figure 4-76: 4.13 Configure Cloudberry Explorer for Swift 1. On the Windows desktop, click the Cloudberry Explorer icon. 64

1 Figure 4-77: 2. When the Select Cloud Storage window opens, select OpenStack as the Cloud Storage. 65

2 Figure 4-78: 3. When the Add New OpenStack Storage Account window opens, enter LOD Swift as the Display Name. 4. Enter the Username:20157648367352802448:swiftadmin. The format of the username is ID:swiftadmin. The ID was created and recorded during the creation of the Swift account. 5. Enter the password created for the Swift account as the API Key. The password is Netapp1!. 6. The Authentication service is the Gateway Node followed by /auth/v1.0. StorageGRID Webscale uses port 8083 for the Swift protocol.. Enter: https://dc1-g1.demo.netapp.com:8083/auth/v1.0 7. Select Do not use for the Keystone version. 8. Click Test Connection. The security certificate will be reported as not issued by a trusted certificate authority. Click Allow. 66

3 4 5 7 6 8 Figure 4-79: 9. Wait for Connection Success. Click Close. 10. Click OK. 11. Close the Registered Accounts window. 1 2 Figure 4-80: 4.13 To Test Cloudberry Explorer with StorageGRID Webscale 1. 2. 3. 4. 5. 67 On the left pane, select Source : LOD Swift. On the right pane, select Source : My Computer. In the left pane, press the Cube Icon, to create a new container. Name the Container: swifttest. In the left pane, open the swifttest container.

1 2 3 5 Figure 4-81: 6. Copy some files from My Computer to LOD Swift (drag and drop). A sample swifttest.txt file is provided in the Sample Files directory on the Desktop. Since there is no specific rule for the swifttest container, the object will be protected with the default 2 copy rule. Note: The objects ingested into the swifttest container will trigger the 2 copy default rule for ILM. 2 Figure 4-82: 4.14 Configure the NAS Bridge In this exercise, you will integrate the NAS Bridge with the StorageGRID Webscale system. You will configure NFS and CIFS shares and test ingesting files into the NAS shares. 4.14 Configuring the NAS Bridge Perform the following activity to configure the NAS bridge. 1. Connect to the NAS Bridge web interface by selecting the desktop icon NetApp SGWS NAS Bridge 2.0, or opening FireFox and entering http://nas.demo.netapp.com. 2. Login into the interface by entering admin@netapp.com as the username, and Netapp1! as the password. 3. Agree to the End User License. 68

2 Figure 4-83: Configure the DNS 1. 2. 3. 4. 5. Select Bridge from the left menu bar. Click Add next to DNS Servers. Enter the IP address, 192.168.0.253 Enter 1 as the Priority. Click Save. 1 4 3 5 Figure 4-84: 69

6. Wait until the Status reports READY. 6 Figure 4-85: Configure NTP Servers 1. 2. 3. 4. Select Bridge from the left menu bar. Click Add next to NTP Servers. Enter the host name, dc1.demo.netapp.com. Click Save. 1 3 4 Figure 4-86: 70

5. Wait until the Status reports READY. 5 Figure 4-87: Configure SGWS Integration 1. 2. 3. 4. 5. Select StorageGRID from the left menu bar. Enter hostname: dc1-g1.demo.netapp.com. Enter port 8082. Enter your S3 credentials previously created, they can be re-used for the lab. Enter both the Access Key and the Secret Key. Remember: You collected this information back in 3.3 of the lab guide, and it's in a.txt file in your Downloads folder 6. Click Save. 7. Wait unil Status reports READY. 71

1 7 2 5 6 Figure 4-88: Configure the Active Directory 1. Select Active Directory from the left menu bar. 2. Click Add next to Active Directory Servers. 2 1 Figure 4-89: 3. 4. 5. 6. 7. 8. 9. 72 Enter the Domain : demo.netapp.com Enter Primary Controller : dc1.demo.netapp.com Enter NETBIOS Domain : DEMO Enter Computer name : NAS Enter Admin Username : Administrator Enter Password : Netapp1! Click Save.

3 5 6 8 4 7 9 Figure 4-90: 10. Wait until Status reports Ready. 10 Figure 4-91: Configure the Cache Devices 1. Select Cache Devices from the left menu bar. 2. Click Add next to Cache Devices. 73

2 1 Figure 4-92: 3. Enter Name : SMB1, 4. Select Disk : /dev/sdb. 5. Click Save. 3 4 5 Figure 4-93: 6. Wait until SMB1 status is READY. 74

6 Figure 4-94: 7. Repeat these steps using name NFS1, and disk /dev/sdc. 7 Figure 4-95: Configure an NFS Share 1. Select NFS Filesystems from the left menu bar. 2. Click Add next to NFS Filesystems. 75

2 1 Figure 4-96: 3. 4. 5. 6. 7. For StorageGRID, select dc1-g1.demo.netapp.com from the drop down list. Enter nfs1 (must be lowercase) as the Bucket name. Select NFS1 as the Cache Device. Enter /nfs1 (remember the forward slash) as the Export path. Click Add Permission. 4 3 5 6 7 Figure 4-97: 8. Enter * (wildcard) for the Client. 9. Click Readwrite. 10. Select No Root for the Squash. 11. Leave Anon UID and Anon GID at default. 12. Click Add. 76

8 9 10 12 Figure 4-98: 13. Click Save. 13 Figure 4-99: 14. Wait until status is READY. 77

14 Figure 4-100: Configure an SMB Share 1. Select SMB Filesystems from the left menu bar. 2. Click Add next to SMB Filesystems. 2 1 Figure 4-101: 3. For StorageGRID, select dc1-g1.demo.netapp.com. 4. For Bucket, enter smb1 (all lower case). 5. For Cache Device, select SMB1. 6. For Active Directory, select demo.netapp.com. 7. For Share name, enter smb1. 8. For Owner, leave as the default Administrator. 9. For Group, leave as the default Domain Users. 10. Click Save. 78

3 4 5 6 7 8 9 10 Figure 4-102: 11. Wait until Status returns READY. 11 Figure 4-103: Test the NFS Filesystem From the Linux Machine, perform the following tasks: 1. Using Putty, ssh to the Linux Machine: linux1.demo.netapp.com. 2. Login: root, password Netapp1!. 3. Enter the following commands: mkdir /mnt/nfs1 mount t nfs o vers=3,rsize=1048576,wsize=1048576,hard,intr nas.demo.netapp.com:/mnt/nfs1 / mnt/nfs1 4. Test by copying some files into the mount point: /mnt/nfs1. 79

5. Using S3 Browser, you can view the contents of the bucket: nfs1. You may need to wait 1 minute for objects to be stored to SGWS. Test the SMB Filesystem From the Windows Jumphost, perform the following tasks: 1. Open Windows Explorer on the Windows desktop. 2. Right-click This PC, and select Map network drive... 2 Figure 4-104: 3. Select a free Drive Letter. 4. Enter folder: \\nas.demo.netpp.com\smb1 5. Click Finish. 80

3 4 5 Figure 4-105: 6. Test by copying some files into the mounted share. 81

6 Figure 4-106: 7. Using S3 Browser, you can view the contents of the bucket: smb1. Wait for a minute and refresh S3 Browser's view. 7 Figure 4-107: 4.15 Exploring S3 and Swift APIs In this section, you will learn how to access the S3 and Swift APIs using command line and scripting tools. These exercises demonstrate how client applications can be modified to store data into StorageGRID Webscale using the S3 or Swift APIs. 82

4.15.1 Access to S3 via the AWS CLI In this section you will learn how to directly ingest and retrieve objects to/from StorageGRID via the AWS S3 Command Line Interface (CLI). This will demonstrate how users or application developers can programmatically integrate with StorageGRID Webscale using the S3 protocol, to store data into the GRID, or get it back out. 4.15.1.1 Persisting Credentials 1. Find the PuTTY icon on the desktop (or in the task bar) and use it to run the PuTTY program. 1 Figure 4-108: 2. In the PuTTY window, choose Linux Client 3. Click Load. 4. Click Open. 83

3 2 4 Figure 4-109: 5. A window will open with a log in prompt. Log in with userid root, using Netapp1! as the password. 6. After you log in, configure the S3 account using the command aws configure --profile webscale. Specify the webscale profile. Specify the access key, and the secret access key. Use us-govwest-1 as the default region and json as the output format.. This setting will default to use path style access. Use the S3 Access Key and S3 Secret Access Key that you obtained back in S3 Account Management on page 14. [root@linux1 ~]# aws configure --profile webscale AWS Access Key ID [****************FCX5]: GGJNYGCPXKWSSBQYR8IC AWS Secret Access Key [****************ZgWu]: Mh4Eax22r+lx/kSmj78/yMtheUXuNbbR37dLnGYx Default region name [us-gov-west-1]: us-gov-west-1 Default output format [json]: json [root@linux1 ~]# To verify the configuration, you can view the following files: [root@linux1 ~] cat ~/.aws/credentials The credentials file contains contains S3 Access Key and Secret Access Key credentials, organized by profiles. [webscale] aws_access_key_id = GGJNYGCPXKWSSBQYR8IC aws_secret_access_key = Mh4Eax22r+lx/kSmj78/yMtheUXuNbbR37dLnGYx 4.15.1.2 Using the AWS CLI In this Lab the AWS CLI is already installed. In case you want to install it on your own machine, refer to https:// aws.amazon.com/cli/. 84

The SSL/TLS certificate for this StorageGRID instance has already been placed at ~/grid_lod.crt. First, list the existing buckets for our S3 profile webscale : aws s3 ls --profile webscale --endpoint-url https://dc1-g1.demo.netapp.com:8082 --ca-bundle ~/ grid_lod.crt Enter this command into the PuTTY window and observe the output: [root@linux1 ~]# aws s3 ls --profile webscale --endpoint-url https://dc1-g1.demo.netapp.com:8082 --cabundle ~/grid_lod.crt 2016-09-22 11:37:30 1-copy 2016-09-22 11:37:18 3-copy 2016-09-22 11:37:35 ec-copy 2016-09-22 13:50:36 nfs1 2016-09-22 13:51:50 smb1 2016-09-22 12:13:39 versioning [root@linux1 ~]# Enter this command into the PuTTY window to perform the same operation with SSL verification disabled. [root@linux1 ~]# aws s3 ls --profile webscale --endpoint-url https://dc1-g1.demo.netapp.com:8082 --noverify-ssl /usr/lib/python2.7/site-packages/botocore/vendored/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html InsecureRequestWarning) 2016-09-22 11:37:30 1-copy 2016-09-22 11:37:18 3-copy 2016-09-22 11:37:35 ec-copy 2016-09-22 13:50:36 nfs1 2016-09-22 13:51:50 smb1 2016-09-22 12:13:39 versioning [root@linux1 ~]# Now you will create a new bucket: [root@linux1 ~]# aws s3 mb s3://new-bucket --profile webscale --endpoint-url https://dc1g1.demo.netapp.com:8082 --ca-bundle ~/grid_lod.crt make_bucket: s3://new-bucket/ [root@linux1 ~]# Next, copy a file into this bucket: aws s3 cp ~/files/1/test.txt s3://new-bucket/ --profile webscale --endpoint-url https://dc1-g1.demo.netapp.com:8082 --ca-bundle ~/grid_lod.crt A subsequent ls operation on the bucket will list the contents: aws s3 ls s3://new-bucket/ --profile webscale --endpoint-url https://dc1-g1.demo.netapp.com:8082 --ca-bundle ~/grid_lod.crt You can use the cp command to copy the object back: aws s3 cp s3://new-bucket/test.txt ~/copied.txt --profile webscale --endpoint-url https://dc1-g1.demo.netapp.com:8082 --ca-bundle ~/grid_lod.crt Now, if you want to sync a whole directory, use the sync command: aws s3 sync ~/files/ s3://new-bucket/ --profile webscale --endpoint-url https://dc1-g1.demo.netapp.com:8082 --ca-bundle ~/grid_lod.crt As you can see, this command did not sync the empty directory 3. This is because S3 does not know about folders as we know them from POSIX based file systems. Nearly all S3 clients only interpret / as a subdirectory for making it more human readable. A full documentation of the supported commands for the AWS S3 CLI can be found at http:// docs.aws.amazon.com/cli/latest/userguide/using-s3-commands.html. Among other interesting commands are: 85

Remove an empty bucket: aws s3 rb s3://bucket-name Remove a non-empty bucket: aws s3 rb s3://bucket-name --force Copy a file to an S3 bucket: aws s3 cp file.txt s3://my-bucket/ Sync a whole directory recursively to/from S3 (will re-copy all files with a different size or modified time): aws s3 sync <source path> s3://my-bucket/<target-path> Delete a file in S3: aws s3 rm s3://my-bucket/<path to the file> For the cp, mv or rm command, you can use the --recursive option to walk through all directories, including subdirectories. Another useful tool is S3cmd (http://s3tools.org/s3cmd), which supports more features than the official AWS CLI. 4.15.2 Access to S3 through Python In many cases, developers will integrate with StorageGRID Webscale using the S3 API via program code. AWS provides many SDKs for accessing S3 programmatically. StorageGRID Webscale is 100% compatible with the S3 API, thus supports the various S3 SDKs. Some of the supported languages are Java, Ruby, Python, PHP, Go, etc. In this exercise, you will briefly perform basic S3 operations in Python against StorageGRID Webscale. First, configure the Python library ( boto3 ) to only use path-style access. While StorageGRID also supports virtual hostname access, this is not enabled in the lab environment. Edit the following file (previously created by aws): [root@linux1 ~] vi ~/.aws/config Add lines to the file to make it look like this: [profile webscale] s3= addressing_style = path output = json region = us-gov-west-1 Now, create a new file s3.py and paste the following code to access StorageGRID: import boto3 import boto3.session session = boto3.session.session(profile_name='webscale') endpoint = 'https://dc1-g1.demo.netapp.com:8082' s3 = session.resource(service_name='s3', endpoint_url=endpoint, verify='/root/grid_lod.crt') client = s3.meta.client You can execute the Python script by running: [root@linux1 ~] python s3.py As this script only connects to StorageGRID, there will not be any output. 86

Next, you can create, list and delete buckets: # Create new bucket s3.bucket('my-bucket').create() # List all buckets for bucket in s3.buckets.all(): print(bucket.name) #Delete bucket s3.bucket('my-bucket').delete() Note that the code might throw errors if the bucket already exists. A new object can be uploaded with the following code: obj = s3.object('my-bucket', 'my-key') obj.upload_file('file/1/test.txt') The object can be downloaded to a file via: obj = s3.object('my-bucket', 'my-key') obj.download_file('target.txt') If desired, you can also add metadata or encryption to the file you upload: obj = s3.object('my-bucket', 'my-key') obj.upload_file('input.txt', ExtraArgs={'Metadata': {'customer_id': '42'}, 'ServerSideEncryption': 'AES256'}) When you look up the object in GMI using the object lookup, you can see that the metadata has persisted. Encryption has also been applied. You can also directly return the metadata, size, and last modification time for an object: obj = print print print s3.object('my-bucket', 'my-key') obj.metadata obj.content_length obj.last_modified And you can also copy objects directly within StorageGRID. This copy process happens within StorageGRID and does not flow through the client: copied_obj = s3.object('<source bucket>','<target key>') copied_obj.copy_from(copysource='/<source bucket>/<source key>') 4.15.3 Access to Swift through the CLI In this section you learn how to directly ingest and retrieve objects to/from StorageGRID Webscale through the Python Swift client from the Command Line Interface (CLI). This allows users and application developers to programmatically integrate with StorageGRID Webscale, to put data into the StorageGRID Webscale, or get it back out. 4.15.3.1 Using Python Swift Client on the CLI First, log into the Linux Client host with PuTTy. In this Lab setup, the Python Swift client is already installed. In case you want to install it on your own machine, refer to https://github.com/openstack/python-swiftclient. The SSL/TLS certificate for this StorageGRID instance has already been placed at ~/grid_lob.crt. 87

Check the version of the Python Swift client: [root@linux1 ~] swift version Specify the auth url using the A argument, the swift account using the U argument, and the K argument to specify the password. The SSL/TLS certificate can be specified using the os-cacert argument. The Swift account ID can be copied from the GMI, under Tenants. The stat command provides the status of the specified swift account. [root@linux1 ~]# swift -A https://dc1-g1.demo.netapp.com:8083/auth/v1.0 -U 20157648367352802448:swiftadmin -K Netapp1! --os-cacert grid_lod.crt stat You can add the account information into your.bashrc profile: # Swift Connection Settings export ST_AUTH=https://dc1-g1.demo.netapp.com:8083/auth/v1.0 export ST_USER=20157648367352802448:swiftadmin export ST_KEY=Netapp1! And run the swift command with fewer arguments: [root@linux1 ~]# swift --os-cacert grid_lod.crt stat [root@linux1 ~]# swift --os-cacert grid_lod.crt list [root@linux1 ~]# swift --os-cacert grid_lod.crt upload my-container upload.txt [root@linux1 ~]# swift --os-cacert grid_lod.crt download my-container upload.txt 4.15.4 Access to Swift Using Python In this exercise, you will perform some basic operations using the Swift API and Python using StorageGRID Webscale as the storage endpoint. 1. Create a new file called swift.py, and paste the following code to access StorageGRID Webscale: The user is the ID for the Swift account created and specified from the NMS. The Authentication URL points to the Gateway Node. The SSL/TLS certificate for this StorageGRID instance has already been placed at ~/grid_lob.crt. import swiftclient user='20157648367352802448:swiftadmin' key='netapp1!' authurl='https://dc1-g1.demo.netapp.com:8083/auth/v1.0' cacert='/root/grid_lod.crt' conn=swiftclient.connection( user=user, key=key, authurl=authurl, cacert=cacert ) 2. Execute the Python script by issuing: [root@linux1 ~] python swift.py As this script only connects to StorageGRID Webscale, there will not be any output. 3. Next, create and list containers: # Create a new container container_name='my-container-swift' conn.put_container(container_name) 88

#List containers for container_list in conn.get_account()[1]: print container_list['name'] Upload a new object can with the following code, and include specific custom metadata: # Create an object with metadata file_name='swift.txt' with open('swift.txt', 'r') as swift_file: conn.put_object(container_name, file_name, contents=swift_file.read(), content_type='type/plain', headers={'x-object-meta-customerid':'42','x-object-meta-color':'red'}) 4. List the objects in the container: for data in conn.get_container(container_name)[1]: print data['name'] 5. Retrieve the object: # Get object get_obj=conn.get_object(container_name, file_name) with open('new_swift.txt','w') as get_file: get_file.write(get_obj[1]) 6. Delete the object: conn.delete_object(container_name, file_name) 89

5 Version History 90 Version Date Document Version History Version 0.9 July 2014 Initial Version Version 1.0 October 2014 Insight 2014 Version Version 1.0 December 2014 Lab on Demand Version Version 2.0 October 2015 Insight 2015 Version Version 3.0 February 2016 Updated for SGWS v10.2 Version 3.1 September 2016 Updated for SGWS v10.3 (Insight 2016 Version) Version 3.1 November 2016 Fix minor typos

Refer to the Interoperability Matrix Tool (IMT) on the NetApp Support site to validate that the exact product and feature versions described in this document are supported for your specific environment. The NetApp IMT defines the product components and versions that can be used to construct configurations that are supported by NetApp. Specific results depend on each customer's installation in accordance with published specifications. NetApp provides no representations or warranties regarding the accuracy, reliability, or serviceability of any information or recommendations provided in this publication, or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS, and the use of this information or the implementation of any recommendations or techniques herein is a customer s responsibility and depends on the customer s ability to evaluate and integrate them into the customer s operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document. Go further, faster 2017NetApp, Inc. All rights reserved. No portions of this document may be reproduced without prior written consent of NetApp, Inc. Specifications are subject to change without notice. NetApp, the NetApp logo, Data ONTAP, ONTAP, OnCommand, SANtricity, FlexPod, SnapCenter, and SolidFire are trademarks or registered trademarks of NetApp, Inc. in the United States and/or other countries. All other brands or products are trademarks or registered trademarks of their respective holders and should be treated as such.