Monitor-Alert-Manage at Scale with OnCommand Unified Manager. November 2016 SL10297 Version 2.1.0

Transcription

1 Monitor-Alert-Manage at Scale with OnCommand Unified Manager November 2016 SL10297 Version 2.1.0

2 TABLE OF CONTENTS 1 Introduction Lab Environment Storage Volume and LUN Layout Review Lab Activities Dashboard Overview Data Protection Relationships Annotations and Groups Favorites Page Capacity and Utilization Alerts Reporting Summary Version History

3 1 Introduction The NetApp OnCommand suite of tools is available, free of charge, to all NetApp customers. While you can configure NetApp storage with the onboard OnCommand System Manager software, management of large numbers of storage objects becomes tedious without help from OnCommand Unified Manager. The goal of this lab is to familiarize you with the benefits of OnCommand Unified Manager. Grow your storage environment without growing your administrative staff by leveraging OnCommand Unified Manager to monitor, alert, and manage at scale! The lab activities explore how to manage availability, capacity, and protection using OnCommand Unified Manager (OCUM). Specific use cases highlight different ways of accessing and using the information provided in the OCUM dashboard. Workflow Automation, OnCommand Insight, and OnCommand Performance Manager are covered in detail in other labs. 3

4 2 Lab Environment This lab environment has been configured with a storage layout typical of a multi-datacenter company. While the two NetApp storage clusters below reside on the same layer 2 network, the storage within each cluster is configured as if it were at different geographical locations. There is a Windows 2012 R2 management station called the Jumphost that you will run the lab activities from. Notice that the environment includes this management station, the OnCommand services, and two NetApp clustered Data ONTAP nodes operating as two single-node clusters. This lab does not require application and database servers, so they are not included the configuration. Figure 2-1: 2 Table of Systems 4 Host Name Operating System Role/Function IP Address jumphost Windows Server 2012 R2 primary desktop entry point for lab dc1 Windows Server 2012 R2 Active Directory / DNS

5 Host Name Operating System Role/Function IP Address ocum OnCommand Unified Monitoring Server Manager Virtual Appliance opm OnCommand Performance Manager Virtual Appliance Performance Monitoring wfa Windows Server 2012 R2 OnCommand Workflow Automation cluster1 clustered Data ONTAP 8.3 cluster for LA datacenter storage cluster2 clustered Data ONTAP 8.3 cluster for NYC datacenter storage User IDs and Passwords 5 Host Name User ID Password Comments jumphost DEMO\Administrator Netapp1! Domain Administrator cluster1 admin Netapp1! Same for individual cluster nodes cluster2 admin Netapp1! Same for individual cluster nodes svm[x] vsadmin Netapp1! SVM administrator ocum admin Netapp1! OCUM administrator opm admin Netapp1! OPM administrator

6 3 Storage Volume and LUN Layout Review Attention: Please take your time to orient yourself with the current lab configuration. In order to demonstrate the capabilities of OCUM, you need many objects to monitor and manage. Having an understanding of the geographical location, business units, and applications for this hypothetical company environment is crucial to understanding why the Annotations and Groups features of the product are used in the examples that follow. The Finance and Sales business units operate out of Los Angeles, CA, while the Support, Engineering, and a second sales office operate out of New York, NY. The main applications for the company are Oracle, Exchange, vsphere virtualization, and basic file shares. Note: This lab environment does not include the application servers. Instead, it focuses on the data layout for a company that would need to support those applications. Figure 3-1: 6

7 4 Lab Activities This lab has the following activities for you to perform: Dashboard Overview on page 7 Data Protection Relationships Annotations and Groups Favorites Page on page 38 Capacity and Utilization Alerts Reporting 4.1 Dashboard Overview OnCommand Unified Manager (OCUM) provides a consolidated view of your storage resources over a number of different storage clusters and sites. OCUM not only collects availability statistics, but it also collects and analyzes capacity statistics that alert you on capacity trends that need attention. Data protection relationships are monitored for compliance with replication lag time thresholds. And through the integration with OnCommand Workflow Automation, protection relationships can be created and repaired without the use of OnCommand System Manager. For performance alerts and trending, OnCommand Performance Manager is also integrated into the OCUM dashboard. This allows performance events to be consolidated into OCUM, and allows the storage administrator to observe more details of a performance event. OnCommand Performance Manager is tightly integrated with OCUM so you can now access Unified Manager and Performance Manager via a single URL and a single sign on (SSO). This enables you to accomplish the following: View cluster health and performance attributes through central dashboard Go to a single location to configure clusters, users and authentication attributes NetApp OnCommand tools are now available under a single pane of glass: Seamless navigation between products through common navigation bar Navigation between Unified Manager and Performance Manager within a cluster scope These steps help provide a view into the details available on the OCUM Dashboard. 1. Open Chrome. The default home page is set to 2. Login to OCUM as admin 3. Enter Netapp1! for the password. 7

8 4. Click Sign In Figure 4-1: Choose cluster1 from the available clusters on the left-hand side. 8

9 6. The right side shows details on capacity, health, and performance for the selected cluster. You can use the Health and Performance links to navigate to more details. Figure 4-2: 5 6 9

10 7. Click the drop down menu and notice that you can also navigate to Health and Performance metrics there. Select Health to continue. Figure 4-3: 7 The main Unified Manager dashboard includes four major sections under Quick Takes : Availability Any events related to storage availability will show up in this section. Events included here are down network interfaces, down cluster interconnects, or missing lun paths. A cluster node that no longer has spare disks available would also be reported here. Capacity Aggregate, volume, and LUN capacity issues are reported in this section. Also included are capacity trending events that alert you if a particular storage object is anticipated to fill up in the near future (The time until full thresholds are configurable). Performance While OCUM collects data at 15 minute intervals, performance events are collected by the OnCommand Performance Manager virtual appliance at 5 minute intervals, and reported up to OCUM. OPM collects baseline performance data on storage objects, and can also do predictive analysis when the access patterns vary from the norm. This lab does not go into the details of the other OPM capabilities except to explain that the events OPM generates will show up in this section of the OCUM dashboard. Protection Data protection relationships with SnapMirror or SnapVault are monitored to insure compliance with the desired update frequencies for each relationship. If the data in the mirror or vault is delayed by more than the allowed lag time, OCUM will show an event in this section. The Workflow Automator services are integrated with OCUM to enable you to correct data protection events without having to leave the OCUM interface. While the Quick Takes section shows a view of how many objects are in a green, yellow, or red state of health, the Unresolved Incidents and Risks section shows more details about the latest and most severe events. If there are many events happening at once, the events tab at the top of the OCUM interface will give a view of them all. Note: The Performance section under "Unresolved Incidents and Risks" is not populated because no workloads are generating IO operations on the controllers in this lab environment. In 10

11 a normal installation, you would see information about the most active storage resources appear in this section. Figure 4-4: Now drill down into the specifics for all Volumes. You may want to take a moment to explore the clusters, SVMs, and aggregates as well. 11

12 8. Select Storage > Volumes. Figure 4-5: 8 9. Click the Storage Virtual Machine column header to sort the column. 12

13 From this view, you can see which aggregate a volume is contained in, as well as which SVM is managing the volume. Take a moment to match up what you see in this view to the Data Layout diagram shown earlier in this guide. Figure 4-6: Click on the drop down menu and select Health to return to the health status page. 13

14 In the next section you will look into the two events that are visible on this health status page. Figure 4-7: Data Protection Relationships On the OCUM Dashboard, there are two events of interest. Locate the two events now, then take them one at a time to resolve both issues. As you click down into an event to get more details, note the other information present in OCUM. Each object-specific page in OCUM contains a wealth of collateral information related to that object, including links to those related objects. 1. The yellow event is a capacity issue with one of the eight SVMs being monitored. 2. The red event is a data protection lag event on one of nine data protection relationships. 3. Click on the 9 just below the red bar graph display where it shows 9 total Protection Lag Status indicators. 14

15 2 1 3 Figure 4-8: 4. Find the entry with the error indicator in the Lag Status column. 5. The volume effected by this error is finance. 15

16 5 4 Figure 4-9: 6. Scroll to the right and find the Schedule column. 7. Notice there is no update schedule associated with the finance volume. This is the reason for the Lag Status error. The volume was created and a SnapMirror data protection relationship was initialized; however, the mirror is now lagging behind the source volume due to the missing update schedule. To correct the error, apply a schedule to the volume, and start an on-demand update of the mirror relationship. 8. Click the svm4_finance_mirror Destination Volume name. 16

17 8 7 6 Figure 4-10: 9. Click Actions > Relationship > Edit. 9 Figure 4-11: 17

18 10. Click the SnapMirror Schedule drop down menu, and select hourly. 11. Click Submit Figure 4-12: 12. Click Actions > Relationship > Initialize/Update. 18

19 12 Figure 4-13: 13. Click Submit. 13 Figure 4-14: 14. Click Events. 15. Select the line related to this SnapMirror lag error by checking the box on the left side of the line. 16. Click Acknowledge. 19

20 Acknowleding an event lets a storage administrator communicate to other OCUM users that the error has been noticed, and the issue is in the process of being resolved. Note that there are other options in this view. Another course of action would be to assign an event to another OCUM user for resolution. 17. While the same line is still selected, select Mark as Resolved. Marking an event as resolved takes it out of the Events view of OCUM until the next polling interval. If you mark an event as resolved, and have not corrected the issue that caused the event, it will re-appear after the next polling interval. (OCUM polls data polling every 15 minutes.) Figure 4-15:

21 18. Select Health from the main menu. Figure 4-16: Notice the Protection status section of the Dashboard now shows all green indicators. 19 Figure 4-17: 21

22 Note: The remaining yellow capacity event noted in the display above will be handled in the Capacity and Utilization section later on in this guide. 4.3 Annotations and Groups Even in small IT environments, managing storage objects in a meaningful way is a challenge. Scaling the number and size of these objects out over many nodes, clusters, and geographically separate datacenters only adds complexity to storage management. OCUM s Annotations and Groups allow a storage architect to better organize these storage objects. When setup properly, Annotations allow you to instantly understand the business impacts of an event, or alert in OCUM. Administrators can associate any storage object to an annotation without the need to apply rule based annotation starting in Unified Manager 6.3. Groups allow an action to be applied to all storage objects that are a member of that group, thus eliminating repetitive tasks on similar objects. These features also allow for quick tailoring of reports generated from OCUM. 4.3 Annotations In this lab environment, Annotations are used to specify where a storage object resides, which application the storage object is supporting, and which business units rely on the storage object for daily activities. Annotataions can be manually applied to a storage object; however, there are often some simple rules you can use to automatically annotate an object depending on its name, which Storage Virtual Machine is managing it, or which cluster it is in. The following lab activity explores the defined annotations as well as the rules that are in place to associate the annotations with the related storage objects. 1. Click Administration > Manage Annotations. 1 Figure 4-18: 2. Click on theapplication annotation. 22

23 Under Values you see that this environment has been configured with four possible values for an application. These values are configured by the OCUM administrator, and can literally be anything that makes sense for organizing storage objects. 3. Click Manage Rules. 2 3 Figure 4-19: This is a list of annotation rules that define which objects will have the Application annotation values. 4. Select the Exchange annotation rule for volume objects. Note: There is also an Exchange rule that applies to SVM objects. 5. Click Edit. 23

24 5 4 Figure 4-20: While annotations can be manually applied to any object, this rule shows how annotations can be dynamically applied to all current and future storage objects that match the rule. In this example, SVM3 and SVM7 are Storage Virtual Machines that are dedicated to serving Exchange data. This being the case, all volume objects owned by either SVM3 or SVM7 will automatically have the Application annotation applied with a value of Exchange. 6. Click Cancel. 24

25 6 Figure 4-21: Take time to edit the other annotation rules to see how their conditions differ. The Exchange and File_Share values are applied to objects owned by certain SVMs, while the Oracle and vsphere values are applied by matching a part of the object name. 7. Click the Annotations tab to go back and investigate the BU (Business Unit) and Geography (Datacenter Location) annotation definitions. 25

26 7 Figure 4-22: OCUM Groups can utilize Annotations within the group rules, as you will see in the next section. 4.3 Manual Annotations Administrators can associate any storage object to an annotation without the need to apply rule based annotation. The following steps will assign a manual annotation to a volume. 26

27 1. Navigate to Storage > Volumes. Figure 4-23: 1 27

28 2. Click on the mgmt_datastore volume. Figure 4-24: 2 3. On the volume details page of mgmt_datastore, select Actions > Annotate. 28

29 Note: You can only assign a storage object to an available annotation. You cannot create a new annotation with this process. To create a new annotation, Navigate to Administration, Manage Annotations and follow the steps detailed under Annotations Figure 4-25: 3 4. On the Annotate Volumes window, select the data-priority > highannotation and value. 29

30 5. Click on Apply. Figure 4-26: Notice this new annotation is now associated with the volume. You also can see that there are two other annotations that have been automatically applied to this volume. vsphere is associated with this volume because its name contains the string "datastore" and LA is associated with this volume because the volume is owned by "cluster1". Note: Review the annotation rules using the steps in the previous section if these automatic associations are not clear. 30

31 6 Figure 4-27: 4.3 Groups The goal behind groups is to define a collection of objects that a single action can be applied to. Because the action is applied across all objects in a group, the members of that group should have very similar characteristics. They could be all file share volumes, all volumes containing LUNs, all vsphere datastore volumes, all SVM root volumes, etc. In this lab environment, the volumes have been defined in the following groups: Database_LUNs Database_NFS SVM_root_vols Unstructured_data VM_storage The activity below focuses on applying a new volume threshold to all members of the SVM_root_vols object group. 1. Click Administration > Manage Groups. 31

32 1 Figure 4-28: Notice that the groups in this environment are applied to only Volume objects. Clusters and SVMs can also be members of a group. 2. Click SVM_root_vols. 32

33 2 Figure 4-29: You can see in this view that the SVM_root_vols group has an associated rule as well as an associated action. 3. Click Manage Group Rules to discover how objects become members of this group. 33

34 3 Figure 4-30: 4. Click the SVM_root_vols name to highlight the entry. 5. Click Edit. 34

35 5 4 Figure 4-31: This group rule is very simple as it matches any volume containing _root in its name. Remember that the objects allowed in groups are clusters, SVMs, and volumes. The Target Object Type on this rule is set to Volume. The Conditions section allows for both and and or conditions to be set in order to create very specific rules. 6. Click Cancel. 35

36 6 Figure 4-32: 7. Click Group Actions. 8. Click on the name of the action to highlight the entry. 9. Click Edit. 36

37 7 9 8 Figure 4-33: This group action overrides the global capacity threshold values. Since the volumes in this group are SVM root volumes, they are very small, and are not anticipated to grow much. Note: In certain conditions, log files can build up and require clean up, or increasing the volume, so it is still a good practice to alert when they get full. 10. Click Cancel. 37

38 10 Figure 4-34: Note: Take time to investigate the other groups, and what conditions are required for objects to be in those groups. Because we are using rules to populate the groups, any new storage objects that are created with those same conditions will automatically be added to the group, and the associated actions will apply to them as well. 4.4 Favorites Page The Favorites dashboard provides faster access to critical storage objects. Cluster, aggregate and volumes can be added to the favorites dashboard. This page intelligently provides relevant information based on the storage object type. For example, an aggregate would display capacity utilization and performance statistics whereas a volume will display capacity utilization, protection and performance statistics. 38

39 4.4 Review of previously configured Favorites 1. Navigate to the drop down menu on the top left corner and click on Favorites Figure 4-35: 1 Note: Notice that for each favorite item on this page there are different details provided in this view that are closely related to the object type. Take a moment to examine the page. At a quick glance you can see that the engineering volume does not have a data protection relatioship 39

40 associated with it. This is just one example of how important details will pop up on this view of items you deem to be "favorite." Figure 4-36: 40

41 4.4 Adding Favorites 1. Now we'll navigate to a resource that we want to add to the Favorites page. Select Health from the drop down menu. Figure 4-37: 1 41

42 2. Navigate to Storage > Volumes. Figure 4-38: 2 Note: You can add Aggregates, Volumes and Cluster objects as favorites. For this example, we are configuring a volume as a favorite. 42

43 3. Click on volume vdi_datastore. Figure 4-39: 3 43

44 4. While on the Volume Details page, click the star to add this volume to the Favorites page. Figure 4-40: 4 44

45 5. Now we will go back and view the Favorites page to view the newly added object. Select Favorites from the drop down menu. Figure 4-41: 5 45

46 Note: Notice the vdi_datastore volume is now listed on the favorites page. It is that simple to populate this page with the storage objects of most importance for quick access to their associated details. Figure 4-42: 4.5 Capacity and Utilization In this section you will locate the details for the yellow capacity event and resolve the event. 1. Navigate to Health 46

47 1 Figure 4-43: 2. Now look at the next error, which is yellow under the Capacity section below, and click on the 1 above the yellow SVMs bar graph. 47

48 2 Figure 4-44: The capacity error is related to the sales_east volume being full. This time acknowledge the error before correcting it. Click the checkbox on the left side of this entry. Click Acknowledge. Notice the State changed for this event. This will communicate to other OCUM users that someone is working on the issue. 7. To resolve the event, click on the source volume name svm8:/sales_east. Figure 4-45: 8. Notice that this volume has a Full Threshold set to 40%. Since the volume has sufficient space, adjust the threshold level to a more appropriate value. 9. Click Actions > Edit Thresholds. 48

49 Figure 4-46: 49

50 10. Click Restore to Defaults to remove the custom thresholds for this volume, and set them back to the global capacity threshold 10 defaults. 11. Notice the Space Nearly Full, and Space Full thresholds that will be applied with this change. 12. Click OK. 50

51 11 12 Figure 4-47: 13. Click Close on the Success confirmation window. 13 Figure 4-48: 14. Navigate to Health. Note: The capacity issue under Unresolved Incidents and Risks and the yellow indicator under the Quick Takes section at the top of the dashboard may take a few minutes to show the 51

52 new (green) status. You can either wait for this update on the dashboard, or continue with the next activity and notice it turn green as you finish other tasks. 14 Figure 4-49: 4.6 Alerts OCUM Alerts can notify storage administrators through or SNMP trap when certain events occur. When viewing an object anywhere in OCUM, you can add an alert for that object through the Actions button under Add Alert. You can also access a list of all defined alerts under the main Administration menu at the top of all OCUM views. In addition to SNMP traps and notifications, an alert can trigger a pre-defined script. The script must be written in Perl, Shell, PowerShell, or be a batch file. The OCUM Alert will pass the following arguments to the script being called: eventid, eventsourceid, eventsourcename, eventsourcetype, eventstate, eventargs In the following activity you create an alert based on storage aggregate events. 1. Click Storage > Aggregate >. 52

53 1 Figure 4-50: 2. Click the aggr2 aggregate. 2 Figure 4-51: 53

54 3. Click Actions > Add Alert. 3 Figure 4-52: 4. Name the alert Data Aggregate Alert. 5. Click Resources. 54

55 5 4 Figure 4-53: Note: The aggr2 aggregate is already in the Selected Resources list because that was the context you were in prior to adding a new alert. From this view, you can continue to add an alert for the one object you selected, or you can modify the alert to include/exclude other objects. For this activity, remove the specific object and use a more dynamic resource selection by excluding all aggr0 aggregates (These would represent the root, non-data aggregates.). Click on the double-left arrow <<- to remove all currently selected resources. Click the << All Aggregates >> resource. Click the right arrow -> to move it under Selected Resources. Click the Exclude tab. Note: The Exclude tab will be greyed out until you put << All Aggregates >> in the selected resource list. 55

56 Figure 4-54: 10. Enter aggr0 in the Name contains field, and click Add. 11. Click Events. 56

57 10 11 Figure 4-55: 12. Select both Critical and Error. 13. Click the double-right arrow - >> to add all items to the Selected Events. 14. Click Actions. 57

58 Figure 4-56: 15. Click the Alert these users: drop down list and select admin. 16. Set the Notification Frequency to Remind every 15 minutes. 17. Click Save. Note: You can also attach a script for remedial actions on a type of event. To perform this, simply add the script in the "Select script to execute". This is just a pointer, we are not actually configuring this in this session. 58

59 Figure 4-57: 18. To view this new alert along with all other defined alerts, click Administration > Manage Alerts. Note: Notice that this is the only alert that has been created in this environment to date. 19. Select the alert, and you will see the alert details appear at the bottom of the page. 59

60 18 19 Figure 4-58: 4.7 Reporting While Alerts are an important part of OCUM communicating storage related issues to the administratior, OCUM Reports provide information that can help identify potential issues, and give the administrator time to be proactive and prevent the issue. Capacity trending reports can help predict how long a LUN, volume, or aggregate can grow at its current rate before becoming full. Other report views help communicate back to business units and executives how the current storage investment is being utilized. Because this lab environment does not add new data to the LUNs and volumes it contains, the capacity trending data will not be of much interest. The sample data that exists within the storage objects was placed there on a single day in the past, and has not been added to since then. Given this fact, OCUM does not anticipate any of the volumes in this lab to become full in the near future. The report you run in the following activity below will give you an inventory of aggregate capacity and utilization. Again, the columns of information in this report that include inventory data will be accurate. Columns containing future utilization predictions will contain near-zero values, because the data in this environment is not currently changing. 1. Click Reports. 60

61 1 Figure 4-59: 2. For the Aggregate Capacity and Utilization report, click Run Report. 2 Figure 4-60: 61

62 3. Scroll from left to right in order to see all of the columns of data available in the report. 4. Click Actions > Share. 4 3 Figure 4-61: 5. When sharing a report, you can enter a list of addresses for recipients, define the subject line, and select from five different export formats: PDF, XHTML, CSV, Excel, and Text. Click Cancel. 5 Figure 4-62: 6. By using the Manage Report Schedules button and the Actions > Schedules option, you can perform the same report export as above, but on a pre-defined schedule. This can be useful in keeping executives aware of storage utilization on a regular basis for budgeting purposes. For now, just make note of where you can set the schedule, but do not select that option. 62

63 6 Figure 4-63: 4.7 Importing Custom Reports Starting in OnCommand Unified Manager 7.0, users can import a custom report with the following process: While importing a Custom report, a user is provided the Select database user with report schema role option to key in a new (or) select from an existing DB user with report schema role and password. This will replace the user credentials and Unified Manager URL in the report file with the current URL the user is in and data that is supplied Note: Credentials are saved in base64 encoding format under the UM Database. 1. To create a custom report, click on Reports. 2. Select Import Report. 63

64 1 2 Figure 4-64: 3. Click Browse... to select a report file. 64

65 3 Figure 4-65: 4. Select Desktop. 5. Select the disk_inventory.rptdesign file. 6. Click Open. 65

66 4 5 6 Figure 4-66: 7. Name it custom1. 8. Give it a description of Custom report #1. 9. Click the drop down menu to select the reportuser database user. Note: This user must have Report Schema role associated with it. Alternatively, you can also add a new user with the option Create new user under the drop-down option available with Name. This will create a DB user in OCUM with the Report Schema role. 10. Click on Import to import this report file. 66

67 Figure 4-67: 11. Click Close. 11 Figure 4-68: 12. Click on the Report tab, and locate the new custom1 report. This is a simple report sample, but feel free to run the report and inspect the output. 67

68 12 Figure 4-69: 68

69 5 Summary With NetApp's OnCommand Unified Manager, you can easily scale your environment while keeping the associated operational tasks at a minimum. With dynamic annotations and groups, your thresholds, alerting, and reporting will remain organized as storage objects are added. With Availability, Data Protection, and Performance monitoring, you can be assured of the health of your storage infrastructure. 69

70 6 Version History 70 Version Date Document Version History September 2016 Initial Release September 2016 Initial Release November 2016 Added Favorites and Reporting sections and updated all screenshots

71 Refer to the Interoperability Matrix Tool (IMT) on the NetApp Support site to validate that the exact product and feature versions described in this document are supported for your specific environment. The NetApp IMT defines the product components and versions that can be used to construct configurations that are supported by NetApp. Specific results depend on each customer's installation in accordance with published specifications. NetApp provides no representations or warranties regarding the accuracy, reliability, or serviceability of any information or recommendations provided in this publication, or with respect to any results that may be obtained by the use of the information or observance of any recommendations provided herein. The information in this document is distributed AS IS, and the use of this information or the implementation of any recommendations or techniques herein is a customer s responsibility and depends on the customer s ability to evaluate and integrate them into the customer s operational environment. This document and the information contained herein may be used solely in connection with the NetApp products discussed in this document. Go further, faster 2016NetApp, Inc. All rights reserved. No portions of this document may be reproduced without prior written consent of NetApp, Inc. Specifications are subject to change without notice. NetApp, the NetApp logo, Data ONTAP, ONTAP, OnCommand, SANtricity, FlexPod, SnapCenter, and SolidFire are trademarks or registered trademarks of NetApp, Inc. in the United States and/or other countries. All other brands or products are trademarks or registered trademarks of their respective holders and should be treated as such.