IBM Storwize V7000 Unified Real-time Compression (RtC) Best Practices Andreas Baer ATS System Storage Europe
Executive summary Start using RtC only after careful capacity planning and capacity contingency to prevent out of capacity condition. Important: Out of capacity (out of space) condition can lead to all file systems in the pool going offline resulting in complete access loss. This can be avoided with capacity planning and monitoring best practices. Capacity Planning Capacity planning must be done with 100% file system pool usage at planned compression ratio in mind. No planning should be done on partial usage to reduce the risk of out-of-space condition. Develop staged plan with enough lead time to add physical capacity, when needed. Set up monitoring and event notification for capacity utilization and test it regularly. Or Conservative approach: Plan 100% of capacity not considering compression. Once compression ratio is stable, start using saved capacity to store more data, but always leave capacity contingency. Important: File deletions in file system and file migrations to a different file system pool do not free up underlying physical storage when configured with RtC. 2
Table of contents 1. IBM Real-time Compression (RtC) background - page 4 2. Estimation and capacity planning - page 18 3. Configuration best practices - page 24 4. Monitoring - page 33 5. Upgrades and growth - page 39 6. Out of space situations - page 41 7. Further troubleshooting - page 45 8. References - page 47 3
IBM Real-time Compression background 4
Real-time Compression (RtC) background - 1 In 2012, IBM released RtC function for SVC/V7000 with Code Release 6.4. It introduced a new Compressed volume type. Support for RtC within V7000 Unified started with Code Release 1.4. It uses the RtC functionality in the V7000 backend and applies it to volumes used for file systems. Note: These are also called file volumes in this document and mapped to file system s NSDs. RtC is designed to work with primary, live data and not for performing the compression in a post-processing step. RtC reserves specific system resources of CPU and memory for carrying out the compression work. Hence these resources are not involved in handling I/Os when RtC is active, that is when compresed volumes are configured. Depending on your workload and I/O demands, usage of resources could cause a noticeable impact to the overall performance. There is always a trade-off between reducing capacity consumption by investing resources into RtC and coping with the I/O workload with the remaining resources. The individual performance considerations and decisions involved are not covered in this document (see References section). 5
Real-time Compression (RtC) background - 2 In cases where incoming data is compressed or encrypted already, little or no additional savings can be achieved by using RtC. It is recommended not to spend system resources trying to compress these. The V7000 Unified system already provides a list of such file types (see File system compression implementation). Common workloads suitable for compression: Databases DB2, Oracle, MS-SQL, etc. Server virtualization VMware, KVM, Hyper-V, etc. General purpose files collaboration, e-mail, home-directories Others CAD, seismic, scientific Common workloads not suitable for compression: Pre-compressed data types video, images, audio, etc. Workloads using application/network-based encryption As a rule of thumb, IBM recommends investing resources in compression for data types that achieve at least 45% compression savings. 6
Real-time Compression (RtC) background - 3 Similar to the thin provisioning function in SVC/V7000, RtC works with virtual and real (allocated) capacities (vsize vs rsize): Even though RtC uses thin provisioning concept; native thin Provisioning is still not supported with V7000 Unified. RtC functions on the same position/layer as thin Provisioning in V7000 code stack (see next page). Similar to thin provisioning, deletion of data does not automatically free up allocated physical capacity. Using RtC requires capacity planning and monitoring of capacity consumption as in thin provisioning: Requires contingency buffer for capacity Need to monitor compression efficiency Important: For every thin provisioning implementation, it is essential to allocate enough buffer capacity in order to have enough lead time for provisioning more storage capacity. 7
V7000 Real-time Compression basics Clients Front End Remote Copy Cache Flash Copy Mirroring Thin Provisioning Virtualization Back End Storage Random Access Compression Engine SVC S/W Component RACE S/W Component All copy services interoperate with compressed volumes: All copy services work with uncompressed data: No real changes in sizing and planning for FlashCopy or replication Bandwidth sizing for replication same for compressed/non-compressed volumes Compression engine resources allocated per I/O group need considered in sizing All thin provisioning properties apply to compressed volumes: Virtual capacity, real capacity, used capacity, etc. New property introduced: Uncompressed capacity: provides an indication of how much uncompressed data has been written to the volume 8
V7000 Real-time Compression: variable input fixed output 1 2 3 4 5 6 Data Compressed Data 1 2 3 4 5 6 V7000 compression takes a variable data stream size and produces fixed output units: Compressed data blocks are 32 kb Compressed volumes have a consistent layout Temporal locality: data that s accessed together is compressed together Variable sized input chunks get better compression Requires fewer disk I/Os Delivers better performance No Fragmentation Compressed Data 1 2 3 4 5 6 Consistent performance over time Consistent compression ratio over time 9 9
V7000 Real-time Compression: progressive compressed block write 64kB 1 Each write from the host is compressed independently. 4kB 2 32kB 3 20kB 1kB 7kB Compression ratio of the resulting block is nearly identical to the compression ratio of compressing the entire data in one operation. 32kB Compression dictionary is preserved between the independent writes. 10 Storwize 2010 Storwize Confidential and Proprietary 10
V7000 Real-time Compression: housekeeping Host blocks which are updated are written to new location: Old compressed block location is marked available and re-used later on. Partially freed compressed blocks are handled by garbage collection: Active data is extracted and combined/compressed with new data. 32 kb blocks freed up are available for reuse. IBM V7000 Compression 1 2 3 4 5 2 6 4 85 86 1 3 32 kb block Rewrite is Compressed Real-time Transparent Garbage Collection 11
V7000 Real-time Compression characteristics Backend I/Os are typically 32 kb in size A read miss results in staging and decompressing maximum 32 kb per block. The read decompresses the block only till the required data is fetched. Writes are aggregated and compressed into 32 kb chunks. All writes are to new locations within the space allocated to a LUN Space is reused as it is freed up but the location of data that is written will change over time. Since R1.4.1 (with R7.1 on V7000 backend) RtC is supported with Easy Tier Easy Tier is compression aware and it only counts read I/Os for compressed volumes I/Os smaller than 32 kb are possible due to partial extraction on read. 12
Configuration Scheme of space allocations in a storage pool - 1 GPFS is not thin provisioning friendly. It does not reuse allocated capacity but allocates new capacity. Block volumes and File volumes (associated with NSDs from file system pool) share the available storage in a storage pool. Block volumes (non-thin provisioned) and non-compressed file systems allocate all their extents at creation time. Remaining capacity is allocated on first come first serve basis. Legend for drawings on next page: - Not shown: mdisks, number of file volumes One extent contains many grains of a given volume, allocated over time as space is needed Extents of (various) Block volume(s) Extents of file volume(s) for non-compressed file system Extents of file volume(s) for compressed file systems 13
Configuration Scheme of space allocations in a storage pool - 2 Contingency capacity (free) Defined threshold -> notification Free capacity below defined threshold Compressed File- System with Compression ratio of 33% Extents of (various) Block volume(s) Noncompressed File- System: virtual = real capacity Compressed File- System with Compression ratio of 50% 14
File systems and compression Compression takes place on volume level: File system is not aware File system volumes (NSD) are compressed Compressed volumes are thin provisioned: Virtual size is the size foreseen for the file system. Real size is the physical capacity actually consumed. Compression can be enabled on file volumes in a file system pool: File volumes containing metadata shall not be stored on compressed volumes. Typically two file system pools are used: One without compression for Metadata and non-compressible data types. one with compression for all data types which compress well (and placement policy to place files in appropriate pool on file creation). NFS, CIFS, HTTP, FTP, Meta NSD Vol Exported file share GPFS file system V7000 Pool 1 System pool Meta NSD Vol Disk or SSD Comp. NSD Vol V7000 Pool 2 Disk V7000U System Comp. NSD Vol 15
File system compression implementation Blacklist of file types and default placement policy is implemented and it can be edited: These file types are be stored on compressed file volumes. Therefore they are typically be stored in a different file system pool. 16
Additional considerations RtC is supported for up to 200 volumes per I/O group: V7000 can have up to 4 I/O groups in the backend. Flexible assignment of mdisks to storage pools and volumes to caching I/O group. Only one I/O group, I/O group 0, can be used for file data: Max 200 volumes for hosting compressed file systems and (optional) compressed block data in I/O group 0. Manage number of volumes when approaching the limit Other I/O groups can be used for block storage access with or without RtC. Resource allocation to RtC (CPU and memory) occurs when the first compressed volume gets created: Explicitly by creating compressed volume for block Implicitly by creating file system with compression If CPU utilization in the V7000 node canisters of an I/O group is below 25% without RtC, then this I/O group is suitable for using RtC compression without impacting normal I/O performance. 17
Estimation and capacity planning 18
Estimation and capacity planning - 1 Capacity Base your capacity estimations on 100% filled file systems and file system pools. Do not count on initial thin provisioning benefit (small initial rsize while file system is empty) at all. Important: File deletions in file systems and file migrations via a remote caching migration policy to a different file system pool, do NOT free up underlying physical storage. Important: Use extra caution when using same storage pool which is associated with a compressed file system pool for either uncompressed file system pool and/or block storage as well. Since these consume available buffer capacity as well, it results in further reducing the available buffer for compressed file system pool, reducing lead time to react, hence increasing the risk to run into an out-of-space situation, if not carefully monitored and managed. 19
Estimation and capacity planning - 2 Growth Include estimated growth into capacity planning and be aware to adjust process steps to provision additional capacity as well: First add disks of same type and capacity as those already used for the storage pools. Different ones, especially larger ones require extra considerations. For example, plan for additional lead times to get new enclosures and/or racks and infrastructure as necessary, when all drive slots in existing ones are filled. 20
Estimation and capacity planning - 3 Compression factor/efficiency File types: include all file types in planning and evaluate their compressibility Blacklist of file types within V7000 Unified For others perform compression tests with tools like gzip/winzip, Adjust blacklist with your own incompressible file types if needed Use the NAS Compression Estimation Utility to determine the expected compression efficiency in your NAS environment. (-> see next slide) Based on the files found and examined, it will return the expected compression rates RtC works with fixed 32 kb blocks written to disk. Updates to existing data are written to a new block location. Partial updated blocks are freed up (combined to 32 kb and written to a new location) by a housekeeping process. However, until this is done some additional capacity remains allocated. 21
Estimation and capacity planning - 4 Compression factor/efficiency Use the NAS Compression Estimation Utility to determine the expected compression efficiency in your NAS environment. Download the utility from: http://www14.software.ibm.com/webapp/set2/sas/f/comprestimator/nas_compression _estimation_utility.html Install this utility on one host (different platforms are supported) which has access to the NAS shares to be determined. These could also be from a test system with appropriate sample data. When running the command line utility, it will scan the connected shares and perform a read access to determine the file types used. Based on the files found and examined, it will return the expected compression rates: 22
Estimation and capacity planning - 5 Calculating required physical storage capacity Physical Capacity PC (in TB) = VC * (1-CR)/CT where: PC = overall calculated Physical Capacity needed VC = overall maximum virtual capacity used for file system(s) creation CR = Compression efficiency savings, e.g. 0,6 CT = Contingency Threshold, e.g. 0,8 for monitoring and warning levels Based on the results from the NAS Compression Evaluation Utility, you can use the equation to calculate required capacity by file type and also to apply different growth rates for different file types. For example, by using an Excel spreadsheet. For experienced RtC users with a stable compression efficiency, add a minimum of 25% contingency capacity to the expected maximum physical capacity allocation PC. If all works as expected, utilization stays below 80%. For all others, start with 100% physical storage capacity of the uncompressed data: Compression gains need to be monitored. These can be used to defer new hardware/capacity allocation. This also simplifies management since no later additions need to be handled in the first place. 23
Configuration best practices 24
Configuration best practices - 1 Create selectively compressed file system one file system pool for metadata and non-compressible data and one file system pool for compressible data Use placement rule to control initial file placement. Blacklist of file types is built-in. Adjust if needed. Option to use compressed file system pool as one tier of an ILM file system One or multiple non-compressed file system pools as well File placement policy to determine initial placement based on file types File migration policy based on thresholds File migration policy based on last access time (inactive data to compressed file system pool) Optional HSM in addition (needs TSM server and Tapes) Important: Do not configure file system policy to migrate data out of the compressed system pool. Even though files are migrated to another pool, the underlying storage space will NOT be freed-up by the migration. 25
Configuration best practices - 2 Clear recommendation to reserve contingency buffer in storage pool with compressed volumes: Amount is determined by compression efficiency risk, planned growth, and changed usage. Develop staged plan to add capacity when needed with appropriate lead times: Adjust, define thresholds and notifications accordingly with enough lead times to ultimately bring in new storage hardware (enclosures, physical disk drives). Note: A new mdisk can only be added to one storage pool. It might require enough drives for multiple mdisks to be added to different storage pools to prevent an out-ofspace situation. 26
Configuration best practices - 3 Configuring RtC with CLI: create separate NSDs for data and metadata: Metadata should never be compressed. GUI ensures this automatically. RtC is only enabled when Create separate NSDs... is specified. -> (see picture on page 33) Use mkdisk and mkfs / or mkfs commands to create volumes. --autoexpand is automatically set in R1.4.1. Creating volumes with realistic rsize shows realistic overall picture initially and defers need for additional capacity allocations of the volumes: Ignore initial thin provisioning related space savings, focus on compression related space savings only. In the example below, compression ratio is 50%, although file system is only 70% full. After some deletions the asssociated volume remains 100% allocated. File system (with deleted content in green) NSD Volume: virtual size Physical storage allocated: rsize 27
Configuration best practices - 4 Starting with R 1.4.1 (contains R7.1 for V7000) RtC and Easy Tier are supported in conjunction. Nevertheless, primary recommendation for SSDs in a V7000 Unified is to use it for file system metadata first. If enough SSD capacity is available, consider specific applications/use cases of Easy Tier within hybrid pools. 28
File system compression - implementation guidance Configure remote caching placement rules: Place dynamic files which are changed and deleted and files which do not compress well on non-compressed system pool. Place static files which are appended and not deleted on compressed pool. Configure remote caching migration rules: Migrate compressible files which have not changed for x days from system to compressed pool. Important: Incompressible files should neither be placed nor migrated into a compressed pool. Optional: Use remote caching to migrate files from compressed pool to tape System Pool One GPFS file system Compressed Pool NSD NSD NSD NSD NSD Vol Vol Vol V7000 Pool 1 ILM V7000 Files File shares Remote Caching Comp. Vol Comp. Vol V7K Pool 2 HSM HSM Tape Pool Disk Tape TSM Server 29
Configuring compression via the GUI - 1 Compression schema Use same or different V7000 pool within one file system pool; metadata is not compressed 30
Configuring compression via the GUI - 2 Custom Preset provides most choice of options Separate NSDs/disks for metadata needed to enable Compressed selection for data disks Choice to compress data, choice if same or different pool 31
Alerting is crucial set up event notification Important: Configure to get alerts for Storage and Utilization thresholds as well: Make sure there is enough lead time to react after receiving the alert. Event notifications can be send via SNMP, syslog, and emails to customer. Event notifications can be send for informational, warning, and error events: Health status changes Event and system log entries Storage alerts for the V7000 block system Utilization thresholds (file system) TSM client backup Quota threshold Event notifications can be configured via GUI: Setting Event Notifications Event notifications can be configured via CLI: mkemailuser username [--utilization <level>] [--gui <level>] [-- status <level>] [--systemlog <level>] [--storagealerts <level>] [--storageinventory <on/off>] [--quotathreshold <percentage>] 32
Monitoring 33
Monitoring Configure notifications to be sent automatically: Set up notifications. Specify threshold value to trigger usage notification to be sent. Threshold value chosen must ensure enough lead time to react before running out of physical space. Best Practice is to have spare capacity ready and have a staged plan to get/add additional capacity. Align planning for contingency capacity with overall capacity growth plan. Make sure to adjust for changed usage and new filetypes as required. Monitor notification infrastructure for function and test regularly to prevent notifications from being missed or received with unacceptable delays. Monitor usage capacity: Use integrated GUI view to display storage pool usage along with file systems. Select Storage pools in file system view. -> (see next page) Monitor individual storage pools. Exceeding thresholds create warning in logs and change file system status (warning, critical). 34
Monitoring compression in the GUI Files > File systems: integrated view for File and Block when selecting Storage pools: Includes view on remaining capacity available in underlying storage pools. Status shown as Critical when storage pool utilization is above the defined threshold. 35
Monitoring compression in the GUI Monitoring > Capacity -> File System Pools: view for file system Pools using RtC: Shows achieved average compression savings of the data per file system pools. A value below 45% indicates that the data in this pool does not compress well. Adjustment of placement policy might be needed and housekeeping on existing data requires an appropriate migration policy. 36
Monitoring a specific storage pool Detailed view on block and file volumes and capacities in an individual storage pool: 37
Other views for monitoring compression in the GUI GUI displays compression savings on volume, pool, and system basis: 38
Upgrades and growth 39
Upgrades and growth Incorporate growth rate into capacity planning: Use upper limit and monitor growth rate incurred. Adjust if necessary. Ensure that enough contingency buffer is available at all times. Plan for infrastructure ahead of time: Based on growth rate and contingency buffer, plan for enough lead time to add hardware including required infrastrcuture. Floor space, power, cooling, connectivity, rack space, etc. 40
Out of space situations 41
Out of space situations Out of space situation of a storage pool needs to be avoided: By proper planning and provisioning of physical capacity By monitoring usage and taking appropriate actions to provision more capacity Out of space situation of a storage pool triggers an impact to all users of the affected storage pool: Could be multiple file systems and their associated shares/clients If a compressed volume related to a file system cannot expand, it will go offline. This causes the file system to go offline (unmounted, all I/O from clients will fail): If CLI is used, make sure autoexpand is ON for all compressed volumes (this is set by default in R1.4.1). Recovery requires to add or free up additional space in that storage pool: For example, migrate block volume(s) to a different storage pool with enough space. For example, delete block volume(s) from that storage pool. For example, add an additional mdisk into that storage pool. Note: This situation is different from the file system filling up, with or without compression. File system remains accessible, but it might enter the read-only mode. 42
Recovery from out of space situations Step 1: On a storage pool level: Solve underlying capacity shortage in that storage pool. Free up existing capacity in that pool. Add new capacity to that pool. This will re-enable compressed volumes to allocate physical space if they need. Step 2: On a volume level: Run DMP for the volumes which are offline. Ensure all the volumes belonging to NSDs for a file system are back online. Associated entries in Block Event log should have been cleared by completed DMP. In System log, mark associated File events as fixed. Step 3: On an NSD level: Ensure all NSDs belonging to a file system are started. If not, use GUI (see next page) and/or CLI to start them (chdisk --action start). Step 4: On a file system level: Mount the file system again. Verify clients have access to their shares again. If not, recovery steps on GPFS level are required (See next section, Further troubleshooting). 43
Out of space situations Step 3: restarting NSDs Recovering from an out of space situation: On an NSD level: Ensure all NSDs belonging to a file system are started. If not, use GUI and/or CLI to start them (chdisk --action start): All disks of a file system or single NSD at a time 44
Further troubleshooting 45
Further troubleshooting If the recovery steps described do not solve the problem, please contact IBM support. The next steps to resolve the problem require guidance from IBM support and might involve root access to the system. Depending on the scenario, individual NSDs might have to be restarted with GPFS commands and file system checks/recoveries might need to be initiated. 46
References related to V7000 Unified and Real-time Compression Real-time Compression in SVC and V7000 Redbook: http://www.redbooks.ibm.com/abstracts/redp4859.html?open V7000 Unified Implementation Redbook: http://www.redbooks.ibm.com/abstracts/sg248010.html?open V7000 Unified RtC Getting Started Guide (V1.08 to be available soon): http://pic.dhe.ibm.com/infocenter/storwize/unified_ic/topic/com.ibm.storwize.v7000.uni fied.141.doc/v7000unifiedreal-timecompressiongettingstartedguidev106.pdf V7000 Unified Infocenter: http://pic.dhe.ibm.com/infocenter/storwize/unified_ic/index.jsp 47
Reference links IBM Storwize V7000 Unified Home page: http://www-03.ibm.com/systems/storage/disk/storwize_v7000/index.html V7000 Unified Info Center: http://pic.dhe.ibm.com/infocenter/storwize/unified_ic/index.jsp V7000 Unified Quick Installation Guide: http://www-01.ibm.com/support/docview.wss?uid=ssg1s7003728 V7000 Unified Problem Determination Guide: http://www.ibm.com/support/docview.wss?uid=ssg1s7003729 V 1.4 Supported Interoperability List of Operating Systems, File Systems and Levels: http://www.ibm.com/support/docview.wss?uid=ssg1s1004228 V 1.4 Configuration Limits and Restrictions: http://www.ibm.com/support/docview.wss?uid=ssg1s1004227 48
Disclaimer This information is provided on an "AS IS" basis without warranty of any kind, express or implied, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. Some jurisdictions do not allow disclaimers of express or implied warranties in certain transactions; therefore, this statement may not apply to you. Important notes: IBM reserves the right to change product specifications and offerings at any time without notice. This publication could include technical inaccuracies or typographical errors. References herein to IBM products and services do not imply that IBM intends to make them available in all countries. IBM makes no warranties, express or implied, regarding non-ibm products and services, and any implied warranties of merchantability and fitness for a particular purpose. IBM makes no representations or warranties with respect to non-ibm products. Warranty, service and support for non-ibm products is provided directly to you by the third party, not IBM. All part numbers referenced in this publication are product part numbers and not service part numbers. Other part numbers in addition to those listed in this document may be required to support a specific device or function. When referring to storage capacity, GB stands for one billion bytes; accessible capacity may be less. Maximum internal hard disk drive capacities assume the replacement of any standard hard disk drives and the population of all hard disk drive bays with the largest currently supported drives available from IBM. IBM Information and Trademarks The following terms are trademarks or registered trademarks of the IBM Corporation in the United States or other countries or both: the e-business logo, IBM, system x, system p, System Storage Microsoft Windows is a trademark or registered trademark of Microsoft Corporation. Linux is a registered trademark of Linus Torvalds. Other company, product, and service names may be trademarks or service marks of others. 49