Quantum QXS-4 Series Hybrid Storage with Q-Tier and Q-Tools

Transcription

1 Enterprise Strategy Group Getting to the bigger truth. ESG Lab Validation Quantum QXS-4 Series Hybrid Storage with Q-Tier and Q-Tools Hybrid Enterprise-class Storage Functionality at Mid-Range Prices By Tony Palmer, Senior ESG Lab Analyst June 2017 This ESG Lab Report was commissioned by Quantum and is distributed under license from ESG.

2 Lab Validation: Quantum QXS-4 Series Hybrid Storage with Q-Tier and Q-Tools 2 Contents Introduction... 3 Background... 3 Quantum QXS-4 Series with Q-Tier and Q-Tools... 3 The Quantum QXS-4 Series: Dense, Reliable, and Serviceable... 4 Q-Tier and Q-Tools Software: Simple, Smart, and Efficient... 4 ESG Lab Validation... 5 Simply Smart and Efficient... 5 ESG Lab Testing... 5 Q-Tier... 7 Cache... 7 Quick... 8 Easy to Deploy and Manage... 9 ESG Lab Testing... 9 Highly Available and Serviceable ESG Lab Testing ESG Lab Validation Highlights Issues to Consider The Bigger Truth ESG Lab Reports The goal of ESG Lab reports is to educate IT professionals about data center technology products for companies of all types and sizes. ESG Lab reports are not meant to replace the evaluation process that should be conducted before making purchasing decisions, but rather to provide insight into these emerging technologies. Our objective is to go over some of the more valuable feature/functions of products, show how they can be used to solve real customer problems and identify any areas needing improvement. ESG Lab's expert third-party perspective is based on our own hands-on testing as well as on interviews with customers who use these products in production environments.

3 Introduction Lab Validation: Quantum QXS-4 Series Hybrid Storage with Q-Tier and Q-Tools 3 This ESG Lab Validation documents the results of expert, independent, hands-on testing of the Quantum QXS Series next generation hybrid system with a focus on the value of Q-Tier and Q-Tools software and the QXS-4 Series high-density storage enclosure. Background Businesses today must adjust to the fact that most of their information from , financial records, and employee data to sales orders, video surveillance files, and shipping/tracking details has gone digital. This is not just true for back-end processes; on the front-end, databases are used to drive web-based operations, file sharing and collaboration is improving productivity, and mobile apps are helping businesses connect with customers and compete in their respective markets. As the value and ubiquity of digital information rises within the business, IT managers are struggling with the challenges associated with managing growing volumes of digital data as they work top modernize their data centers. As shown in Figure 1, increasing use of server virtualization and data center consolidation were near the top of data center modernization spending priorities in ESG s 2017 IT Spending Intentions Survey. 1 Organizations recognize the opportunity to improve efficiency by consolidating applications on fewer physical servers. ESG research also indicates that storageintensive big data and data analytics initiatives are on the rise with a goal of accelerating business intelligence and customer insight. Figure 1. Top Ten Data Center Modernization Spending Priorities We would like to learn more about your specific spending plans for data center modernization. In which of the following areas will your organization make the most Increasing use of server virtualization Improving data backup and recovery IT infrastructure orchestration/automation tools Data center consolidation Increasing use of solid-state/flash storage technology Deploying converged infrastructure platform Implementing a software-defined data center strategy Building a private cloud infrastructure Deploying hyperconverged infrastructure Increasing use open source platform software 20% 28% 27% 26% 26% 25% 24% 32% 32% 36% Quantum QXS-4 Series with Q-Tier and Q-Tools The Quantum QXS-4 Series with Q-Tier and Q-Tools is a modular, high-density hybrid storage system that delivers enterprise-class features and performance at mid-range price points. The new and improved software capabilities of Q-Tier and Q-Tools and the flash-optimized QXS-4 Series high-density hardware platform cost-effectively addresses the data 1 Source: ESG Research Report, 2017 IT Spending Intentions Survey, March 2017.

4 Lab Validation: Quantum QXS-4 Series Hybrid Storage with Q-Tier and Q-Tools 4 growth and performance challenges associated with a wide variety of applications and workloads including server virtualization, high-performance computing (HPC), media and entertainment, surveillance, big data, and analytics. The Quantum QXS-4 Series: Dense, Reliable, and Serviceable The QXS-4 Series is the cornerstone of the Quantum QXS family of hybrid disk array platforms. The QXS-4 Series is an extremely reliable and serviceable hardware platform that s a ninth-generation field-proven storage hardware architecture. More than 650,000 storage systems based on this architecture have been sold. Some of the largest IT systems vendors in the world have come to rely on the enterprise-class reliability and field-proven % availability of this architecture. Quantum QXS series is designed to comply with both NEBS Level 3 and MIL-STD-810GA standards. These standards document that Quantum QXS storage can withstand extreme environmental conditions like heat, significant shaking, and dust without impact to operations. Several valuable, patented technologies are built into the architecture, including Q-Turbo and EcoStor, which provide low-latency mirroring and battery-free protection from data loss after a power failure. The QXS-4 Series combines two QSX-4 controllers with dual power supplies and a mix of up to SAS HDDs and SSDs in a small 2U footprint (see Figure 2). With optional SAS-connected expansion, the QXS-4 platform supports up to 192TB of usable capacity in only 8U of rack space. Up to eight ports of 16G Fibre Channel, iscsi over 10Gb Ethernet, or 12Gb SAS per dual controller system are provided for server attachments. The drives are completely hot-swappable and are housed in three drawer-like trays that are designed to be easily replaced. Figure 2. Quantum QXS-4 Series Powered by Q-Tier, Q-Turbo, and Q-Tools Q-Tier, Q-Turbo, and Q-Tools Software: Simple, Smart, and Efficient Optional Q-Tier is automated storage tiering software that offers SSD and HDD tiering in real time, moving data based on usage every five seconds. Q-Turbo, integrated with the QXS controllers, increases ingest speed by 50% by simultaneously writing to both controllers. Q-Tools is second generation storage virtualization software that delivers simple, smart, and efficient storage management services. This latest version of the software that s included with the QXS-4 Series fuels a powerful set of tools including SSD caching, thin provisioning, zero-impact snapshots, and quick RAID rebuilds. A new and improved management interface streamlines deployment and simplifies routine storage configuration and management tasks.

5 Lab Validation: Quantum QXS-4 Series Hybrid Storage with Q-Tier and Q-Tools 5 ESG Lab Validation ESG Lab performed hands-on evaluation and testing of the QXS-4 Series with Q-Tier and Q-Tools. Testing was focused on the value of the new and improved software capabilities of Q-Tier and Q-Tools software and the high availability and serviceability of QXS-4 Series hardware. Simply Smart and Efficient Q-Tools storage virtualization software was designed with a goal of improving the efficiency and manageability of pools of storage, while Q-Tier automated storage tiering software was built to enhance storage performance and efficiency. Pools of hard disk drives (HDDs), solid-state drives (SSDs), and a hybrid mix of both are supported. Q-Tier and Q-Tools both work automatically in real time with a goal of optimizing an organization s storage environment. After configuring storage into one or more pools, Q-Tier and Q-Tools kick in and work automatically as they deliver a simply powerful range of smart and efficient storage services, including: Q-Tier: Improves performance and reduces costs. Frequently accessed 4MB chunks of application data are automatically moved in real-time to a higher performing tier (e.g., SSD) for faster performance, and infrequently accessed chunks are automatically moved in real-time to a lower cost tier (e.g., near-line SAS) for reduced costs, in real time. Q-Tools: o o o o o Pool, which simplifies the definition and management of tiered pools of storage with a powerful, yet simple to manage, storage virtualization layer. Thin, which reduces the cost of storage capacity with thin provisioning. Instead of reserving and consuming storage when host volumes are created, thin provisioning consumes capacity as needed by applications. Snap, which provides zero-impact data copies for quick and easy recovery and test development. Read-only and writeable snapshots are supported with redirect-on-write technology and no need for zero block recovery. Cache, which provides a quick real-time performance boost for business-critical applications with SSD configured as a caching layer for frequently accessed read data. Quick, which reduces the risk of data loss after a drive failure with fast RAID rebuilds. Instead of rebuilding an entire RAID group, only the capacity that has been allocated to storage volumes is rebuilt to restore data protection more quickly. Q-Turbo: QXS Q-Turbo provides a high bandwidth, low latency, active-active write-back cache, designed to approach the performance of dual independent cache mode. QXS Q-Turbo writes to the first controller and simultaneously mirrors the write to the cache of the second controller. Quantum claims a 50% write improvement over traditional methods. ESG Lab Testing ESG Lab testing began with a tour of the new and improved Q-Tier and Q-Tools management interface with a goal of providing a quick introduction to some of the key new capabilities that are built into the Q-Tier and Q-Tools software stacks.

6 Lab Validation: Quantum QXS-4 Series Hybrid Storage with Q-Tier and Q-Tools 6 A home screen shot that was captured during ESG Lab testing is shown in Figure 3. Note how Thin has been used to provision more capacity than is physically available (green versus blue bars), and Pool has been used to configure three pools of storage with different performance characteristics (performance, standard, and archive). The arrows toward the bottom show how Q-Tier is delivering intelligent tiering in real time. In this example, 4MB chunks of hot data are moving from the standard to the performance tier, and cold chunks are moving to the archive tier. Figure 3. Q-Tier and Q-Tools Management Interface Home Screen Figure 4 illustrates the effect of Q-Tier during ESG Lab testing. The screenshot below shows a home screen status summary shortly after starting an online transaction processing (OLTP) database workload. The counters and bar graphs indicate that nearly all the performance is being served by ten disk drives in the green standard pool. It s important to note that data movement began immediately after starting the OLTP workload. The screenshot on the right, which was taken short while later, shows how most of the performance is being served by a pair of solid-state drives in the red performance pool. Figure 4. Q-Tier in Action

7 Response Time (ms) Lab Validation: Quantum QXS-4 Series Hybrid Storage with Q-Tier and Q-Tools 7 Q-Tier Q-Tier moves frequently accessed 4MB chunks of application data to high-performance storage media for faster performance and infrequently accessed chunks to lower cost media for reduced costs. Q-Tier works in real time as it monitors and moves data between tiers of storage in minutes to hours versus competitive tiering implementations that can take hours to days to move bigger, and more inefficient, chunks of application data. ESG Lab tested Q-Tier with a goal of quantifying the benefits of tiering in real time. ESG Lab tested an industry-standard workload that simulates the activity of a typical OLTP database application in a pool with sixty 600GB RAID-10 10K RPM hard drives and eight 400GB RAID-1 solid-state drives. 2 Real-time tiering kicked in and started migrating frequently accessed data to solid-state within minutes after starting the test. Aggregate performance increased approximately 2.5x over the course of the next 14 hours. Tiering up to solid-state not only improved the number of database transactions, but it also improved I/O response times, as shown in Figure 5. Figure 5. Delivering Faster Response Times for an OLTP Database Workload Cache Faster Response Times with SSD and Q-Tier (8 KB OLTP workload, 8 400GB RAID-1 SSD, GB RAID-10 10K RPM HDD) Faster Response Times ,020 Elapsed Time (minutes) Cache delivers a quick real-time performance boost for business-critical applications with SSD configured as a caching layer for frequently accessed read data. Cache is configured at the pool level with a goal of turbo charging the read performance of business-critical application workloads. Read performance can be improved by a factor of up to 10x with SSD and Cache compared with a pool of spinning hard drives. While Cache does not accelerate write performance because Cache lacks the non-volatile capabilities of Q-Tier, it can accelerate performance for business-critical read-intensive workloads in a matter of seconds or minutes compared with Q-Tier, which can take from minutes to hours to fully optimize I/O, based on data set size. 2 Test bed details: Quantum QXS-4 Series, 6 x R GB 10K & 4 x R GB SSD, 4x8G FC, Dell R610 Xeon 8 CPUs 12GB RAM, Microsoft Windows Server 2008 R2, 8K OLTP database workload.

8 I/O Response Time (ms) Lab Validation: Quantum QXS-4 Series Hybrid Storage with Q-Tier and Q-Tools 8 ESG Lab tested Cache with the industry-standard Iometer workload generation utility and a read-intensive OLTP database workload. 3 A 4GB volume was tested with a goal of demonstrating how Cache can be used to promote a hot range of highly referenced data into SSD in a matter of minutes. As shown in Figure 6, Cache provided a real-time performance boost in a matter of seconds and reduced the average I/O read response times from more than five milliseconds to less than one millisecond in less than two minutes. Figure 6. Cache: Increasing the Performance of Read-intensive Workloads with SSD Caching 7 Faster Read Response Times with SSD and Cache (8 KB OLTP workload, 1x400GB SSD, 10x600GB RAID-10 10K RPM HDD) Faster Response Times Elapsed Time (seconds) Quick Quick reduces the risk of data loss after a drive failure with fast RAID rebuilds. Instead of rebuilding an entire RAID group, only the capacity that has been allocated to storage volumes is rebuilt. While the level of acceleration narrows as the storage system fills up, it also reduces the risk of data loss after a multiple drive failure to some level. Quick rebuild also reduces the duration of potential application performance degradation. Figure 7 shows how a Quick rebuild on a partially full RAID group completed 14.6x faster than a full rebuild during ESG Lab testing. While the level of acceleration narrows as the storage system fills up, it also reduces the risk of data loss after a multiple drive failure. Quick rebuild also reduces the duration of potential application performance degradation. 3 Test bed details: Quantum QXS-4 Series, 10 x R GB 10K HDD, 1 x 400GB SSD, 4x8G FC, Dell R610 Xeon 8 CPUs 12GB RAM, Microsoft Windows Server 2008 R2, Iometer version , 8K random 75/25 read/write.

9 Lab Validation: Quantum QXS-4 Series Hybrid Storage with Q-Tier and Q-Tools 9 Figure 7. Quick: Reducing the Performance and Impact of RAID Rebuilds Quick RAID Rebuilds (8+2 RAID-6, 10K RPM Drives) Traditional full Rebuild RealQuick Rebuild (10% full) 14.6X faster RAID Rebuild Time (minutes) Easy to Deploy and Manage The new and improved Q-Tier and Q-Tools management simplifies storage deployment with intuitive GUI and streamlines routine configuration changes with a table-driven user interface. Granular performance monitoring with deep historical trending provides a powerful set of monitoring and debugging capabilities. ESG Lab Testing Why This Matters The complexity and challenges associated with meeting the growing storage requirements of the business can lead to increased capital equipment costs (buying more storage than needed) and operational expenses (spending too much time managing storage). This is especially true within highly consolidated virtual server environments serving a mix of application workloads. Given the automatic, intelligent management, IT managers don t have to think about the benefits of the Q-Tier and Q- Tools software. Capacity, efficiency, and performance are automatically optimized in real time. ESG Lab was particularly impressed with the cost-effective real-time efficiency that can be achieved with Q-Tier and a hybrid mix of solid-state and traditional hard disk drives. ESG Lab tested how easy it is to configure a QSX-4 Series with Q-Tier and Q-Tools from scratch. Screenshots taken at the beginning and end of the intuitive GUI interface are shown in Figure 8. Starting on the left with a blank configuration, it took just five minutes and 26 mouse clicks to configure the system from scratch, and define and map a storage volume to a server. ESG Lab was not only impressed with the intuitive simplicity of the GUI but also noted that the home screen provides an intuitive, top-down overview of the configuration, health, and utilization of the system from hosts, through host ports and overall capacity, to a great view of back-end storage pool consumption.

10 Lab Validation: Quantum QXS-4 Series Hybrid Storage with Q-Tier and Q-Tools 10 Figure 8. Quick and Easy Deployment Highly Available and Serviceable The QXS-4 Series with Q-Tier, Q-Turbo, and Q-Tools leverages more than two decades of hardware and software innovation and a field-proven track record of % availability. The QXS-4 Series hardware has passed some of the most rigorous certification tests in the industry (e.g., the NEBS Level 3 compliance Telco industry standard). The modular and fully redundant QXS-4 Series hardware platform is highly available, field serviceable, and extremely dense with a supported mix of up to 48 hard drives and solid-state disk drives in an industry-standard 2U package. ESG Lab Testing Why This Matters Data growth is driving up the cost of deploying and managing storage systems. Thus, the 1-button configuration has become a welcome area of focus and innovation in the storage industry in recent years. ESG Lab has confirmed that the new and improved Q-Tier and Q-Tools management can be used to dramatically reduce the time, effort, and potential for error inherent in deploying and managing complex storage environments. The 2U QXS-4 Series enclosure contains three field-serviceable shelves, each containing a total of 16 drive slots. By loosening a thumb screw on the shelf, ESG Lab was easily able to slide the shelf out toward the front of the array, exposing the 16 SAS drives. ESG noted the unique design of the shelves, as the outer shelves were designed with the drives facing the center of the array. This allowed the rack to easily be serviced, even if placed tight against a wall on either side of the array.

11 Lab Validation: Quantum QXS-4 Series Hybrid Storage with Q-Tier and Q-Tools 11 ESG Lab validated the reliability and serviceability of the QXS-4 Series enclosure by injecting system faults, monitoring the system for alerts, and non-disruptively replacing failed components with an I/O-intensive simulated application workload running in the background. As shown in Figure 9, the following errors were injected during ESG Lab testing: Unplug a redundant FC host interface cable. Unplug a redundant power supply cable. Remove a field replaceable fan. Remove a field replaceable hard drive. Remove a field replaceable controller. Figure 9. Error Injection Testing First, ESG Lab pulled one of the drives in the shelf to simulate a drive failure. An alert was quickly displayed in the QXS Q- Tools management interface, and ESG Lab was easily able to identify the enclosure, drive-tray shelf, and individual drive that had failed. This visual indication made locating the drive very simple, helping to reduce the chance of human error, and minimizing the time to failure resolution. ESG Lab then tested the serviceability of other components of the QXS-4 Series shelf by simulating the failure of host ports, power supplies, fans, and even a system controller. Each failure was quickly identified and resolved, and with just a click, ESG Lab reviewed the event log entries that identified the alerts and resolution of component failures along with the associated timestamps. Figure 10 shows the QXS Management Interface and Event Log entries while identifying the controller failure. In all cases, the application workload continued running without error and no entries were found in operating system event logs.

12 Lab Validation: Quantum QXS-4 Series Hybrid Storage with Q-Tier and Q-Tools 12 Figure 10. Fine-grained Historical Performance Monitoring Why This Matters The ability of IT end-users to access information anytime from anywhere is not simply desired, it is expected. Downtime not only hurts business productivity, but in many cases it can also impact customer satisfaction and revenue. Because of this, it s no surprise ESG research indicates that one of the top priorities for IT in 2015 is improving data backup and recovery. ESG Lab validated that the fully redundant and field-serviceable architecture of the QXS-4 Series with Q-Tier and Q-Tools is highly available and serviceable. ESG Lab was particularly impressed with the innovative mechanical design of the hot pluggable and field replaceable hard drive trays that support up to 48 drives in only 2U of rack space. The alternating orientation of the drive trays that enables hot plug replacement of any drive, even when the rack is up against a wall, is an example of the unique design to ensure the utmost in high availability and serviceability. Price-Performance One of Quantum s key claims is that QXS can provide 90% of the performance of an all-flash array (AFA) at 30% of the cost. To validate the price-performance and total cost of ownership of Quantum QXS as compared to other solutions on the market, ESG Lab created a simple model comparing QXS with an all-flash array. This representative system comes from ESG's analysis of several vendors, choosing representative median values for each component and parameter. List prices were used for the comparison. Configurations were built using vendor guidelines to meet the following requirements: Total IOPS required: 100,000 Applications to be supported: 100 virtualized OLTP databases, 1,000 virtual desktops, 500 additional servers supporting all other virtualized workloads Usable capacity required for all applications: 146TB

13 Cost of Ownership Lab Validation: Quantum QXS-4 Series Hybrid Storage with Q-Tier and Q-Tools 13 Hardware and software maintenance included for three years Tiering, Replication, and Snapshot software included Figure 11. Quantum QXS-4 vs. AFA Cost of Acquisition, Maintenance, and Infrastructure over Three Years $800,000 $700,000 $600,000 $500,000 $400,000 $300,000 80% Savings $200,000 $100,000 $0 Typical All-Flash Arrray Quantum QXS-4 Hardware Software Software Maintenance Infrastructure As Figure 11 shows, Quantum QSX-4 s cost of ownership over three years is significantly lower than ESG s all-flash array model, with the cost of software and maintenance contributing most to the delta. This compares very favorably to Quantum s claim of 90% of the performance at 30% of the cost of an AFA, with the cost for the Quantum solution coming in at just over 20.5% of the AFA in this model. Why This Matters In ESG research, cost reduction was called out as a business initiative driving IT spending by 32% of respondents. 4 In the same survey, data center modernization was called out as an important IT initiative. Organizations are also prioritizing spending on increasing the use of server virtualization and data center consolidation. In short, organizations are working to improve efficiency by consolidating applications on fewer physical servers and storage arrays. What is needed is a solution that can deliver the performance applications need, simply and cost efficiently. Through a combination of hands-on testing and cost modeling, ESG Lab validated that Quantum QSX offers organizations a high-performance alternative to all-flash arrays at a significantly lower cost of ownership. 4 Source: ESG Research Report, 2017 IT Spending Intentions Survey, March 2017.

14 Lab Validation: Quantum QXS-4 Series Hybrid Storage with Q-Tier and Q-Tools 14 ESG Lab Validation Highlights Q-Tier, Pool and Thin increased the efficiency and ease of managing a cost-effective hybrid mix of SSD and HDD storage. Q-Tier started to increase performance with a hybrid mix of SSD and HDD in a matter of minutes for an OLTP database workload. Cache with SSD increased performance for a read-intensive database workload in a matter of seconds. Quick reduced RAID group rebuild times by 14.6x for a volume that was 10% full compared with a traditional full volume rebuild. The new and improved Q-Tools management interface was used to configure a QXS-4 Series from scratch in five minutes and 26 mouse clicks. ESG Lab confirmed that the fully redundant and field-serviceable QXS-4 Series was highly available as it rode through a barrage of simulated hardware failures. Issues to Consider Real-world application workloads tend to have more locality of reference than the workload generators that were used during this ESG Lab validation. As a result, ESG Lab is confident that intelligent real-time tiering can be used to exceed the 2.5x performance improvements presented in this report. Improvements of up to 5x are realistically achievable depending on the cache friendliness of an application and the ratio of solid-state to hard disk capacity. While ESG Lab testing of intelligent real-time tiering focused on the performance advantages that can be achieved in real time with a relatively modest amount of SSD, it can also be used to reduce costs as it moves less frequently used data to a more cost-effective tier of storage. While the QXS is not a unified solution that supports network-attached storage (NAS) and block-based storage area network (SAN) protocols, it is often deployed as part of a unified solution with the file system layer implemented in a cost-effective virtual server or appliance.

15 The Bigger Truth Lab Validation: Quantum QXS-4 Series Hybrid Storage with Q-Tier and Q-Tools 15 The Quantum QXS-4 Series includes a ninth-generation custom storage processor that delivers world-class performance at the lowest price possible especially when compared with the growing number of midrange storage systems that are built with a layer of software running on an industry-standard Intel processor. Purpose-built hardware with field-proven % availability and more than 650,000 systems shipped forms the rock-solid foundation for the QXS-4 Series with Q- Tier, Q-Turbo, and Q-Tools software. The pooling and automatic tiering of Q-Tier and Q-Tools is seriously smart and efficient. The fine-grained 4MB efficiency and real-time frequency of Q-Tier provide best-in-class efficiency and performance compared with competitive tiering systems with larger page sizes (e.g., 1GB) and sluggish monitoring and migration frequencies (e.g., 24 hours). Intelligent, automatic tiering continually pumps data throughout the organization in real time when and where it s needed most, at the lowest possible cost. ESG Lab testing showed significant improvements in response times for workloads mirroring typical business environments. Q-Tier and Q-Tools software quickly adapted to rapidly changing workloads. We believe customers will welcome Quantum s simply elegant approach to storage management that saves time and money, delivers great performance at a compelling price point, and eliminates the hassle of dealing with RAID.

16 All trademark names are property of their respective companies. Information contained in this publication has been obtained by sources The Enterprise Strategy Group (ESG) considers to be reliable but is not warranted by ESG. This publication may contain opinions of ESG, which are subject to change from time to time. This publication is copyrighted by The Enterprise Strategy Group, Inc. Any reproduction or redistribution of this publication, in whole or in part, whether in hard-copy format, electronically, or otherwise to persons not authorized to receive it, without the express consent of The Enterprise Strategy Group, Inc., is in violation of U.S. copyright law and will be subject to an action for civil damages and, if applicable, criminal prosecution. Should you have any questions, please contact ESG Client Relations at Enterprise Strategy Group is an IT analyst, research, validation, and strategy firm that provides market intelligence and actionable insight to the global IT community by The Enterprise Strategy Group, Inc. All Rights Reserved. P