A Detailed Review Abstract This white paper covers the use of virtual storage provisioning within the EMC Celerra storage system. It focuses on virtual provisioning functionality at several levels including file systems, iscsi LUNs, and iscsi snap copies. The information is presented to allow for more informed decisions to be made with respect to configuration and management of the Celerra storage system. May 2007
Copyright 2007 EMC Corporation. All rights reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com All other trademarks used herein are the property of their respective owners. Part Number H2572 A Detailed Review 2
Table of Contents Executive summary...4 Introduction...4 Audience... 5 Terminology... 5 Overview...5 Virtually provisioned file system... 5 iscsi LUN... 8 Virtually provisioned iscsi LUN... 9 Usage... 11 Storage pools... 11 iscsi LUN selection... 12 Virtually provisioned snaps... 12 Dynamic LUN extension... 13 Conclusion...14 References...14 A Detailed Review 3
Executive summary With ever-increasing demands for disk space, efficient allocation and management of storage are more important than ever. Storage administrators want to make sure that sufficient amounts of storage are allotted to users or applications in order to meet the needs of the business. Equally important is the desire to avoid overallocation of resources that, if not required, sit unused on the system. The virtual provisioning architecture introduced in the Celerra Network Server 5.5 release targets these two areas. With the introduction of virtual provisioned file systems, iscsi LUNs, and iscsi snaps, Celerra offers a more efficient allocation and management model for storage resource allocation and consumption. This white paper covers the virtual provisioning features introduced with the DART 5.5 release for Celerra. In some cases, such as virtually provisioned iscsi LUNs and snaps, the feature is a modification to the existing architecture, providing additional options for the way that you implement and manage iscsi storage. This paper introduces each of these new features individually and covers the interrelationships that allow you to use them in your environment to satisfy a particular set of business goals or requirements. Introduction Celerra iscsi target for IP block storage was introduced through the Celerra Network Storage system in the 5.3 version of DART. The product features set has been updated several times, providing enhancements for ease of management, configuration wizards, and VSS support for snapshot integration. In this white paper, we will describe the features introduced in the 5.5 DART release that further enhance the capabilities of the product and options for deploying Celerra IP storage for block-based applications. As the technology is more widely adopted, trends and use cases were observed that led to the need for more flexibility in the areas of LUN configuration and storage provisioning. The new features are as follows: iscsi LUN extension As the demand for storage increases it is necessary to provide a method to meet those needs. LUN extension provides an option to increase the size of an existing iscsi LUN by allocating additional space within the file system. Virtually provisioned iscsi LUNs This feature provides the option to create a LUN without allocating or reserving all of the disk space it represents. Virtually provisioned LUNs offer the ability to configure iscsi storage devices that initially consume very little storage resources. The DART file system manages the resources and allows the iscsi LUN to grow organically as clients require additional storage. Virtually provisioned snaps Snapshots or point-in-time copies that require no space when activated. These devices significantly reduce the impact of mounting and restore operations. Virtually provisioned file systems While not strictly an iscsi feature, virtually provisioned file systems offer the ability to create a file system of fixed size that is not mapped to physical disk blocks. VP is a demand-based model where consumption of file system resources will trigger block usage. File system autoextension Provides an internal method to extend a file system by either a predetermined size. That size can be either a fixed number of bytes (megabyte/gigabyte) or a percentage of the existing file system size. While not a specific iscsi feature, this functionality is a key component to supporting virtually provisioned storage. In this paper we will further introduce these features, discuss the options for their use, and highlight the dependencies that exist when they are combined to meet a specific business requirement. A Detailed Review 4
Audience The audience for this paper is implementers, technical architects, and administrators interested in the use of iscsi storage on the Celerra. The paper is largely written at a functional level and the reader will benefit from an understanding of the Celerra iscsi target architecture. If you are reading this paper for the purposes of implementing this technology, an understanding of the storage requirements and growth patterns will also be of value in consideration for the practical uses of the features. Terminology The 5.5 release introduces a new term to Celerra virtual provisioning. Its meaning in the context of iscsi LUNs and file systems is similar to that of thinly provisioned devices that are created without preallocating physical resources. The benefit of virtual provisioning is that it offers an allocation model that is demandbased. Resources are provided as they are needed, avoiding allocation of devices that may never used. Overview The virtual provisioning functionality is introduced in several components of the Celerra DART 5.5 operating environment. In storage environments, the term provisioning is very common and is used to describe the creation or allocation of storage for file system or block device use. Virtual provisioning extends this concept to mean that the device or file system being created represents a particular size but does not occupy that space on disk. While it may consume that space over time, it initially uses a fraction of the size representation. The Celerra operating system, or DART, can create virtually provisioned devices from a Celerra volume or from within a Celerra file system. The benefits of virtual provisioning are that they provide a more-efficient management model for utilization of storage resources. Disk space is not consumed until it is needed, allowing for resources to grow based upon actual usage patterns versus reserved disk allocations. The real gain then is in the flexibility offered by these devices. While it is always a good practice to understand the storage needs and plan your implementation to address them, business requirements may change, resulting in unused allocations that may be better served to address other projects. Virtual provisioning provides a storage device model that adapts to your business needs and allows for growth and allocation in the areas where it is required, avoiding the unintended results that sometimes occur due to overprovisioning of storage. In this paper, we will focus on two virtually provisioned storage objects: the virtually provisioned file system and the virtually provisioned iscsi LUN. Virtually provisioned file system Virtually provisioned file systems are created in the same manner as their densely provisioned counterparts on the Celerra storage system. At an operational level creating a virtual file system is very similar to creating a file system in previous releases. In fact the file system uses the same building blocks that are used to create a dense file system. It is created from the same storage pools using Celerra Management with configuration method leveraging Automatic Volume Manager within DART. The difference is that it merely has different on-disk usage characteristics. It is configured with an initial size that is fully allocated; however, the file system has defined attributes that determine when it should be extended and optionally what its maximum size should be. The core component of the Celerra provisioning model is the Celerra Volume Manager. It is used to create volumes and metadevices from one of a number of storage pools. The pool is an aggregate representation of disk devices presented to the Celerra from the back-end storage systems. The pool size is reflective of the number of similarly sized RAID-protected LUNs configured on the storage system. A Detailed Review 5
In Figure 1, we illustrate the mapping between the logical storage view of the virtually provisioned LUN and the physical layout of the LUN within the Celerra file system. In this example, there are three clients that are defined as Host A, Host B, and Host C. A 10 GB virtually provisioned LUN has been created and assigned to each host. The disk device manager on each host views the device size as 10 GB, however, the actual disk usage is some fraction of that as represented in the file system. These devices are all mapped to on-disk file objects within the Celerra file system. This on-disk use for each iscsi LUN only represents those blocks that have been written to. In figure 1 those sizes are 2 GB, 3 GB, and 7 GB in the file system. The file system has also reserved additional space that is represented as free space. That space will be used for ongoing LUN growth by any of the iscsi LUNs. The file system that houses the virtually provisioned LUNs is created from a storage pool on Celerra. In Figure 1 the pool is made up of several storage system RAID-protected devices. The pool allows for basic provisioning and offers available space for extension in the event that the existing file system reaches the defined usage threshold. Logical storage view Host A 10 GB Host B 10 GB Physical consumed storage Free 7 GB 3 GB 2 GB Physical allocation File System view Size = 15 GB Used = 12 Free = 3 GB Host C 10 GB Storage Pool Figure 1. Virtual provisioned storage mapping of iscsi LUNS in Celerra A Detailed Review 6
The virtual file system is created by specifying an initial size that is fully allocated within the storage pool. This space is reserved for use by the client system and functionally there is no distinction between it and a traditional file system. The file system requires a disk usage threshold value, which identifies the point at which automatic file system extension will be triggered. The disk usage threshold is based on a percentage of used space within the file system that DART monitors. The threshold is user-definable but if a threshold is not specified a default value of 90 percent of the disk space is applied. When the threshold is reached the DART operating system detects the event and triggers a request to extend the size up to a predefined maximum limit. The value can be modified up or down as usage changes. Note: Setting the threshold value is one of the most important considerations of this architecture. It is the protection method that prevents a file system full condition. File system extension that results from a high water mark condition is an asynchronous process. If the file system is subject to bursts in write activity that could result in a file system full condition, it is recommended that you reduce the threshold value to 60 percent or 70 percent, to allow time for the file system extension. Celerra Manager provides two methods to monitor physical file system usage. The File System Properties page, shown in Figure 2, within the GUI provides a snapshot view of the system usage as a percentage of the overall file system size. The server_df command in Celerra, shown in Figure 3, is a useful tool to list the number of blocks that have been written to that file system. Figure 2. File System Properties page Figure 3. server_df command A Detailed Review 7
Celerra Manager file system predictive analysis can be enabled to generate notification alerts when the file system is nearing the high water mark. Celerra predictive monitoring is a feature of the Control Station that uses file usage heuristics to generate SNMP traps or email alerts in advance of a file system reaching its capacity. The system can be defined to notify a user or distribution list at a predefined period in advance of the system reaching the full state. Celerra uses the event notification framework to trigger the automatic extension of the file system whenever the high water mark for that file system is reached. The event is also logged to the Data Mover and a message similar to the following will appear in the event log: CFS: 4: 6: FsId: 40 MaxSize: 10240 MB HWM: 70%. Forced Extension Size=1024 MB iscsi LUN A Celerra iscsi LUN is a storage object that is created within the Celerra file system. The file system is essentially a container that provides a pool of storage resources for the iscsi LUN or LUNs. It is therefore sometimes referred to as a file object within the Celerra iscsi architecture. When a LUN is created it possesses some of the same physical characteristics as a standard file within the Celerra file system. It has a fixed size and is referenced by an index node, or inode. In order to make it available to a client system, the iscsi target creates a mask that associates the target and LUN with the client Initiator Qualified Name (IQN). The client system uses an iscsi initiator to log in to the IP target, perform device discovery, and format it for use by the operating system. In the iscsi target implementation model for DART 5.4, each iscsi LUN is fully reserved or dense, which means that all of the blocks comprising the LUN s defined size are persistently reserved within the file system. This model provides a very reliable method for allocating and providing space for iscsi storage objects. In addition to the LUNs, the file system is also used for storing iscsi snap images of the LUN. The storage required for the creation of the snap is related to the number of blocks that are allocated within the iscsi LUN when the snap is created. A third component of space consideration is the temporarily writable snap (TWS). The TWS is required when restoring from a snap or promoting snaps to secondary hosts. The TWS also allows the snapshot view of the LUN to be written to. It also reserves file system storage space while the TWS is mounted. A Detailed Review 8
Virtually provisioned iscsi LUN The iscsi LUN model prior to the DART 5.5 release provides a reliable method for creating iscsi storage objects; however, there are some limitations with respect to the amount of physical disk space required. Due to the dense nature of the iscsi components there are conditions where file system space becomes limited. This could be the case when multiple snaps need to be promoted or restored at the same time, or business practices necessitate the retention of more snaps than originally planned. One method to address that need is to increase the size of the file system; however, the DART 5.5 release introduces virtually provisioned iscsi LUNs and TWS that limit the impact of those objects on the overall storage allocation. The virtual provisioning model is predicated upon a storage unit that consumes very little initial storage space. The device is assigned a fixed size limit but allocates no blocks until the client operating system requests additional space to write to. Each write request from the client file system is translated into a block allocation request by DART that will allocate additional space until it reaches the capacity of the device. The file object representing the iscsi LUN essentially grows like any other file would in a regular file system. With virtual iscsi LUNs, the file system exhibits pool-like characteristics in that any LUN can allocate any of the free blocks within the file system. At an operational level an administrator can configure many different LUNs that all have access to common disk space thus allowing for more efficient use of that space. Virtual provisioning the LUNs provides the option to establish the iscsi environment to meet future growth needs of the client systems, and reduces the need to monitor the hosts for capacity issues. It further addresses the potential need to resize the LUNs and the potential for dead storage that is allocated to a host that may never use it. This offers a deployment scenario in which provisioning LUNs does not require the detailed level of planning that is required for persistent reservation architectures. Should the need for additional space arise it can be addressed by extending the pooled resources for the LUN pool, namely the file system. The file system automatic extension feature within Celerra Manager provides the option to seamlessly extend the file system when that condition arises. Configuration decisions are less rigid in their impact on the overall storage resources. Initial configuration decisions can be made that do not require resizing of disks. A virtually provisioned iscsi LUN is created from within a file system using the command line interface for Celerra Management. The CLI server_iscsi utility has been modified to accept a new option (-vp), which defines LUNs as virtual provisioned. The virtually provisioned LUN occupies very little initial disk or file system space. That is, no disk blocks are reserved for client use. They are allocated on demand. Unlike the previous LUN provisioning model, the virtually provisioned iscsi LUN does not reserve the space that is used within the file system, it only consumes space within the Celerra file system when the client operating system performs SCSI write commands. Those writes are translated as block allocations within the Celerra file system. This includes the formatting process that will allocate a limited number of blocks for the file system structure. A virtually provisioned iscsi LUN of 1 GB in size, for example, will allocate approximately 7.5 MB of space for the file system structure for a Windows 2000 server. The remaining blocks are unused until the host operating system issues a SCSI write request to the iscsi local device. That SCSI write request will be translated into a write allocation request within DART. DART will allocate the necessary blocks to the LUN in order to satisfy the write request. A Detailed Review 9
The virtually provisioned LUNs are all stored in one or more file systems. As a result there may be other VP LUNs in the same file system. It is a very real possibility that through the overprovisioned configuration one of the LUNs may generate enough data to approach the file system s usable capacity. In that event the Celerra provides the ability to extend the file system. The trigger that initiates this is an administratively defined value associated with the file system usage. For iscsi virtually provisioned LUNs it is best to use a value such as 70 percent in order to avoid a potential race condition that could result from a runaway process or burst in file system write activity. A second recommendation when using virtually provisioned LUNs is to include sufficient space to accommodate the LUN extensions that may be required. Using a file system that is at least 50 percent of the total virtual LUN size is an adequate value to use. If you suspect a high rate of storage growth within any of the LUNs you should increase the file system size to 75 percent or greater, or consider the use of a fully allocated iscsi LUN. Monitoring the file system usage is one method to identify the number of bytes used. You may also use host utilities such as Windows Explorer and disk free or df on UNIX servers to see the amount of disk space that is used in the iscsi device. This method of provisioning will allow the client to be provided a set of storage LUNs that will not use all of the file system space unless it is required by the host. There is, however, an important characteristic that needs to be understood about block allocation and virtually provisioned LUNs. Even though the LUN does not allocate blocks initially, once allocated, they are persistent throughout the life of that device. As described previously the blocks are allocated within the Celerra file system by the client operating system. While the client may delete those blocks they are not deallocated from the Celerra file system. So a file system that performs lots of unique writes will eventually consume all of the blocks within the virtual iscsi LUN. Virtually provisioned LUNs are useful, however, in environments where the file system of the host server does not reach full capacity. In cases where the file system is written to and purged or has a static size with potential for increased capacity demand, the overall usage will remain fairly consistent. The server_df command on the Control Station will allow you to monitor the usage of the Celerra file system. Additionally file system usage is monitored by DART. There are several features in the Celerra Management architecture that will provide insight and preemptive remediation of a file system that is beginning to reach its capacity. Predictive monitoring allows for the notification in the event that a file system reaches a user-definable threshold for block storage. In that event, a notification can be provided that states the file system has reached the threshold as well as the expected time at which the file system will become full. This functionality offers a proactive method for administrators to either manually increase the file system size or perform file system maintenance such as removing unnecessary objects within the file system. Celerra can be configured to autoextend the file system to meet increased storage requirements. It is strongly recommended that this feature be configured for file systems containing virtually provisioned iscsi LUNs. A predefined threshold will trigger a file system resizing when the threshold has been encountered. This is another proactive measure that protects the file system assets as the storage requirements increase. A Detailed Review 10
Usage A virtually provisioned LUN can only be created using the command line interface. The CLI provides the ability to create the LUNs on either dense or virtually provisioned file systems. The graphical interface allows you to create dense iscsi LUNs from any densely configured file system. An overview of the usage rules for LUN configuration is listed in Table 1. Table 1. Usage rules for LUN configuration Management Virtually provisioned LUN Densely provisioned LUN CLI management Y Y Graphical management N Y Virtual file system support Y N Dense LUN support Y Y File system Extension Y N In addition to the notes above any file system can be configured for autoextension. When using virtual provisioned LUNs the allocated space is returned to the file system when: A snap has been deleted. In that case the amount of space that is recovered is minimal. A TWS with written data is demoted. The production LUN is removed. As is the case with virtually provisioned file systems, it is important to understand the iscsi LUN usage requirements when planning to use virtually provisioned LUNs. The reason for this statement is that as the underlying file system becomes full, the Control Station will detect the state and request that the file system be extended. If the change rate of the iscsi LUN surpasses the rate of LUN extension the application will encounter a file system full condition that will result in an I/O error to the iscsi client. It is therefore equally important to modify the threshold for automatic extension to be less than the default of 90 percent and to also use predictive notification to monitor the file system. The management of the iscsi virtual LUN allows for creation of the object through the command line interface only. Deleting, listing, and extending the logical size of the LUN can be accommodated through either the command line or graphical interface. In order to make this feature safe there is additional functionality that is required. That has been provided through iscsi LUN extension and file system autoextension. These features allow the Celerra to detect conditions where additional resources are required to meet growth needs. Note: When creating a virtual LUN from the Celerra Manager command line interface a warning message is posted to the standard output device. The message warns of the potential data loss in the event that the file system becomes full and cannot be extended. The message output is listed below. Warning 17716815834: server_3 : You have just created a virtually provisioned (sparse) LUN. To avoid data unavailability and potential data loss when a file system is 100 percent full, monitor file system utilization to ensure the file system contains sufficient free space for LUN growth. Storage pools In order to ensure that the system does not get to a state where the operation is disrupted, the system must provide a source to meet new storage requirements of automatic extended file systems. Celerra aggregates Symmetrix or CLARiiON LUNs into storage pools. The basic requirement for a LUN component is that A Detailed Review 11
it is of the same RAID type as other components within the pool. For Fibre Channel pools the LUNs must also be added in pairs. A pool then provides the necessary expansion space for the automated file system extension. The autofs extension uses the AVM profile of the file system to determine the necessary pool from which to allocate additional space. DART then uses the pool to extend the file system by the requested amount. The extension is limited only by the available space within the storage pool. iscsi LUN selection Since there are now two options for creating an iscsi LUN it is important to understand what the impact is in order to decide which is best for your environment. Densely allocated LUNs are very useful for hosting critical applications data and environments where the growth rate is unknown or unpredictable. Growth of the LUN can be accommodated through the iscsi LUN extension functionality in DART 5.5 It is also useful in environments where storage performance is the No. 1 criteria for iscsi storage. In the case of dense LUNs the file object is reserved and blocks are allocated within the Celerra file system as the client writes to the iscsi device. Depending on the size of the file system and the number of iscsi objects, dense LUNs could reduce the spatial locality of the on-disk blocks. This is particularly true for large iscsi LUNs. Virtually provisioned iscsi LUNs provide an option to create devices that only appear to be reserved. The virtual iscsi LUN initially uses very few 8 KB blocks to hold the metadata information for the LUN. There are additional blocks that get allocated after the LUN is presented to the host for file system creation on the iscsi device. This is still fairly negligible in regards to the total number of used blocks since most file systems are sparse as well. With this model we can create a considerable number of LUNs and allow many different users to share a single file system for iscsi LUNs. It provides a more-flexible method for configuring iscsi storage within the Celerra. It offers an option to efficiently disburse storage to your clients and allow the system to allocate the necessary storage based upon client needs. Since the LUNs are virtual devices the management concern is the iscsi storage pool or file system from which the LUNs are created. Addressing additional storage requirements can be addressed through extending the file system, thus avoiding the need to resize LUNs. Additionally with the file system autoextension feature the system can be configured to demand additional space once predefined thresholds of storage usage have been reached. It must be noted that virtually provisioned LUNs could adversely affect the host file system if the Celerra file system becomes full. The use of virtually provisioned LUNs requires sufficient amount of storage space either within the Celerra file system or in a Celerra storage pool. It also requires that you monitor the rate of growth to avoid a file system full condition. Virtually provisioned snaps Celerra also provides the ability to create a virtually provisioned device for iscsi snapshot images. Snapshot images of the iscsi LUN can be used for many purposes, including backups, restoring individual files, and restoring the entire LUN to a particular point in time. The snap images can also be used for secondarization purposes to satisfy business needs. In the former cases the snap is primarily a read-only device with very little intent to modify the contents of the LUN. Prior to DART 5.5, whenever a snap was promoted as a TWS it would reserve the full logical capacity of the primary LUN. This was to allow the secondary host to modify the entire contents of the snapshot LUN. This was also a requirement for performing restores of iscsi snap. A Detailed Review 12
With virtually provisioned TWS this is no longer the case. The snap creation will allocate roughly eight file system blocks within the Celerra file system. Promoting the snap will use another four. The total for both is roughly 88 KB of disk space. A virtual snap is enabled on a Data Mover basis by setting a DART parameter called nbs.sparse TWS. It can be modified via the Celerra Management interface. It is those allocated blocks that will be transferred to the snapshot image when a new snap is created. Unlike in the dense model, however, a snap does not require the same amount of space if does for dense LUNs. In fact it is essentially that same amount of space that was used when the LUN was first created. That is less than 1 MB of space. Depending on the client write pattern a LUN may reach a size in which, although it is actively performing write activity, the writes are updates to existing files that require no additional space. While that may affect the space requirements for snaps, it would not affect the space requirements for the LUN. Dynamic LUN extension Another method provided to accommodate increased storage requirements is Celerra Dynamic LUN extension. The Celerra DART code provides the ability to extend the size of existing LUNs using the Celerra Management interface. Through the course of standard use it may become necessary to add more space to a LUN. The LUN extension feature allows the administrator to allocate additional storage to the current LUN. This feature applies to both densely provisioned and virtually provisioned LUNs. LUN extension is a two-step process. The initial process of extending the device is accomplished on the Celerra through the management interface. However the resizing must take place on the client system accessing the iscsi LUN. It requires the client to rescan the SCSI bus to identify the device attribute changes. It also requires the use of client-side disk utilities to update the partition table with the new properties of the disk device. In the case of Windows systems this is accommodated through the diskpart utility in the Resource Kit. Solaris provides this functionality through the devfsadm command. The Linux operating system offers several tools to rescan the SCSI bus. The LUN extension feature works by reserving additional space within the file system for a densely created iscsi LUN. In that case it is reflected by a resizing of the iscsi LUN file object and is visible through the file system properties of the Celerra Management interface. The extension can also be applied to a virtually provisioned LUN in which case the client-visible attributes of the logical LUN are updated, but as described previously no Celerra storage is consumed until the client system writes to the device. A Detailed Review 13
Figure 4. Extend iscsi Lun interface Conclusion This white paper discussed the options available for virtual provisioning and how it can be employed with the Celerra file system and iscsi LUNs. The use of each of these devices offers independent benefits. When combined together they provide a solution to improved storage use as well as an option to simplify the configuration and management of the iscsi environment. Automatic file system and LUN extension were identified as important tools for addressing increasing storage needs that allow you to configure your systems to meet today s storage needs and provide organic growth to the client systems as their capacity requirements increase. Celerra Manager provides the necessary tools to establish the configuration and to monitor the state of the storage within your Celerra. Tools were presented that allow you to verify the growth patterns as well as identify the amount of free resources. The proactive analysis and event notification were identified as tools that can be used by the support staff to be alerted if the system resources become scarce. References For more information, refer to the Configuring iscsi Targets on Celerra technical module available on Powerlink, EMC s password-protected customer- and partner-only extranet. A Detailed Review 14