Over provisioning in solid state hard drives: benefits, design considerations, and trade-offs in its use

Transcription

1 Over provisioning in solid state hard drives: benefits, design considerations, and trade-offs in its use

2 Conditions of use: Intended to provide the reader with some background on over provisioning, this paper is for internal use only and should not be shared outside the company. Abstract: Solid state hard disks (SSDs) may appear very similar to conventional rotating hard drives, but the similarity is only superficial. If one examines the internal workings of an SSD, you would find that it shares very little in common with legacy drives. Just as the storage industry has developed best practices to coax as much performance and robustness from rotating drives as possible like short stroking, larger caches, RAID, etc. SSDs have their own methods of improving performance and reliability. Unlike rotating hard drives (in which techniques like short stroking reduce the drive capacity substantially), the methods used in SSDs are transparent to the end user, but can affect performance and endurance significantly. This brief paper examines one such strategy used in SSDs over-provisioning. Background: The Basics of SSD Design: The first thing to understand is how SSDs are constructed. Any SSD (Micron or otherwise) has the following basic components: Host interface Printed circuit board NAND Controller NAND Second, one needs to understand how NAND is organized on an SSD. The smallest unit of storage is called a page and is typically 4K or 8K in size (other sizes are also used). Pages are then logically grouped into larger structures called blocks which are typically 128 to 256 pages. One writes to NAND differently than one writes to a rotating hard drive: NAND which contains data (valid or invalid) must be erased first, only then it can have data written to it. There is a good blog post explaining this located here ( In a nutshell, the erase process erases an entire block, only then can the pages in that block have data written to them.

3 The two fundamental processes in any storage device read and write operate at different speeds on NAND devices. Reads are much faster than writes and writes are much faster than erases. If writes are minimized or if a more efficient method of managing writes can be found, overall drive performance improves. Since NAND that has data in it already must be erased before new data can be written to it, the write sequence for a device that is full (i.e. has no pre-erased blocks) is very different from that of a device that is empty (or has enough pre-erased blocks to absorb the incoming write traffic). No pre-erased blocks: Suppose we had no pre-erased blocks. Every data element that is to be written to the SSD requires that cells be erased immediately before that write happened: Data comes into the SSD from the host interface, resulting in a write request (that will eventually be written to the NAND). 2. Are there any pre-erased blocks (that is, blocks whose pages are ready to be written to)? In this example, there are none 3. Locate block whose data has been invalidated and is an erase candidate 4. Erase this block <Repeat as necessary to store the data in the write request> 5. Write the data to pages in the newly available pre-erased blocks Step 4 is typically very slow to complete because erasing NAND takes much longer than writing to it. Any NAND erase cycle is slow (typically it takes 250 microseconds to write to an MLC cell, but 1500 microseconds to erase it the erase takes much longer). In addition, the drive must invoke garbage collection in order to erase the block (see later section on garbage collection for details). This garbage collection process decreases the write performance further.

4 Pre-erased are blocks present: Now suppose we were able to ensure we had pre-erased blocks ready at all times. As the write came into the SSD, steps 3 and 4 would be avoided completely and the process flow would look like this: Data comes into the SSD from the host interface, resulting in a write request (that will eventually be written to the NAND). 2. Are there any pre-erased blocks (that is, blocks whose pages are ready to be written to)? In this example, there are pre-erased blocks available 3. Write the data to the pages in the already available pre-erased blocks As you can see this write process is much faster and the erase step, the slowest step, has been completely removed. New drives/used drives: If you are using a brand new drive, this really is not an issue. A new drive is empty it will have pre-erased blocks ready to receive new data and the erase step is rarely performed as data is being written. However, as the drive fills the number of these pre-erased blocks decreases and will be consumed eventually. There is a background process in SSDs that frees up pages whose data has been invalidated (marked as deleted or moved) making them good candidates to be erased and added to a pre-erased pool of blocks. This background process is called garbage collection. Garbage collection: Garbage collection is a process by which the NAND controller on the SSD manages the SSD s available storage. As data is written to and removed from an SSD, NAND pages on the SSD are filled (data writes) and are marked as available for erase (data deletes). The pages marked for erase are actually not erased yet because NAND can t be erased one page at a time, it has to be erased in blocks. The NAND controller looks at blocks in which at least some pages have been marked as available for erasure (invalid data), looks at the data that remains in the other pages in that block (valid data), then moves the valid data to free pages and invalidates the data in the former location, converting an entire block into an erase candidate. Once the block has only erase candidates (pages with invalid data), the NAND controller can erase the block in preparation for new data s arrival.

5 How it works: This abstracted diagram shows an SSD that contains user data and free areas. The green boxes are pages in the drive that contain valid data, data the user wants to keep. The blue pages are empty. One column represents a block of NAND (the smallest unit that can be erased) ): The user now erasess the data contained in the pages shown in black. The user could have deleted a spreadsheet, editedd a Word document, etc. The data in those pages is now invalidated and the NAND controller knows that these pages could be erased to prepare them for new data. However, we have to erase all the pages in a block at once (represented by a column in this diagram). The controller moves the data in the 2 pages directly above the erase candidates into available pages (which have no data and are thus available to store this information). This frees up an entire block (column), enabling that block to be erased, making it available for new data.

6 The controller has been able to erase the entire block (erased pages shown in light blue), making that block available for new data, as shown in the next diagram. New data can now be written to the (newly erased) blue pages. The full drive problem: Provided that a NAND based SSD is empty, or at least has sufficient pre-erased pages to meet the incoming write demands of the host, there is no performance degradation. When the drive is nearly empty, this above process of garbage collection routine is rarely activated. As the drive fills, the number of pages available to absorb incoming write traffic decreases. As there are fewer and fewer such pages, the garbage collection routine is activated more regularly and performance degrades. As the drive continues to be written and the garbage collection routine is more active, the drive write performance should reach a plateau called steady state after which little to no further performance degradation occurs. Drive endurance: Another consideration with NAND based SSDs is drive endurance. The basic storage elements of NAND SSDs the NAND cells themselves are subject to wear out and will not last forever. Each time a NAND cell is erased/programmed its life expectancy is reduced. Micron SSDs use different NAND types depending on the intended use of the SSD (this is very common in the industry): both multi- MLC NAND level cell ( MLC ) NAND and single-levestores 2 bits (4 charge states) per cell and is rated at a lower program/erase ( P/E ) cycle count when cell ( SLC ) NAND are used. The basic differences: compared to SLC NAND. SLC storess 1 bit per cell (2 charge states) and is rated at a substantially greater number of P/E cycles. The NAND controller manages writes to the cells (regardless of drive/nand type) to maximize drive life, but NAND cells (and hence SSDs) do not have infinite life. This lifespan is referred to as drive endurance.

7 Write amplification: Another phenomenon unique to SSDs is their write amplification being greater than one. Write amplification ratio of the write traffic the NAND cells experience divided by the write traffic that comes from the host through the drive interface. The additional writes that happen due to wear leveling (a background process in which the NAND controller distributes the writes across the NAND cells to ensure that no single cell gets written to more than any other), garbage collection, and the fact that large areas must be erased before data is written to them create write amplification. Usually write amplification will be expressed as a number, say This number reflects the number of physical writes that occur in the drive. As noted above, this can also be thought of as a ratio, like 1:1.83 either way what is meant is that more physical writes occur to the NAND than were actually sent by the host. Rotating hard drives always have a write amplification of 1 because they simply write the data as it comes in and do not require the erase cycle that a NAND based SSD does. Rotating hard drives do not need to move data internally to ensure level wearing of the drive storage media. The table below gives an example of 10 bits being written by the host and the effect of different write amplification values on the number of bits actually written to the NAND: Number of bits written by host Write Amplification of drive 10 bits bits 10 bits bits 10 bits bits Actual writes that go to NAND Ideally, solid state drive designers strive for a write amplification as low as possible. Because the NAND cells have a finite lifespan, write amplification is an important consideration. Over-provisioning: Reserving some portion of NAND for use by internal processes on the SSD processes like garbage collection is a common practice called over-provisioning. Depending on the intended use of the driver there may be as little as 6%* or as much as 50%** (or more) of the total available NAND capacity. By having physically more NAND on the device that the host can utilize, the SSD NAND controller is ensured of an adequate amount of working area to manage and optimize its overall write performance, and balance servicing incoming host traffic with its background operations.

8 6% to 50+% of total NAND capacity is reserved for by the NAND controller The yellow areas represent the amount of space on the NAND that is reserved for use by the NAND controller. The physical cells that are reserved can be located anywhere on the NAND array the boxes just show the portion of the NAND that is reserved. Because this space is only used by the controller, the amount of total space on the drive differs from the amount of space the host can utilize. These differences vary based on the amount (percentage) of over-provisioning. *6% is reserved for client drives is because the majority of their traffic is reading data which is less affected by write amplification. **50% is reserved on some enterprise drives, Micron reserves 27% Enterprise drive traffic is much more write intensive which increases write amplification. The effect of over-provisioning on drive performance: As noted previously, as a drive fills the number of free (erased) pages decreases and so does drive write performance. Empirical data indicates that by over-provisioning SSDs this problem can be mitigated and the drive performance can be stabilized over the useful life of the drive. The diagram below compares the performance of an SSD without over-provisioning to that of an SSD that is over-provisioned. How full the drive is (expressed as a percentage of total capacity) is shown on the x-axis; relative drive performance is shown along the y-axis.

9 Enterprise Over-provisioning This plot shows enterprise drive performance (yaxis) as a function of how full the drive is (x-axis). The firmware with over-provisioning is the top line, the drive without it is the bottom line. Drive performance for the over-provisioned drive remains nearly constant regardless of how full the drive is. Drive performance for the non-overprovisioned drive falls off sharply as the drive fills. Client Over-provisioning This plot shows the same configuration, but a client drive is used instead. Because client drives have less over-provisioning (as a percentage of overall drive capacity), the drive performance is not linear with over-provisioning. However, the plot clearly shows the benefit of overprovisioning even on client devices. The effect of over-provisioning on write amplification: As noted previously, the basic storage elements of NAND based SSDs the NAND cells themselves have a finite lifespan. As we ve shown, the phenomenon of write amplification can increase the number of actual writes that occur to the NAND cells over and above the number of writes sent to the SSD by the host. These extra writes are a product of wear leveling and the basic structure of NAND. With an over-provisioned drive, the effect of write amplification can be reduced. Over provisioning a drive effectively increases the working area available for the controller s background tasks. As this working room increases, the write amplification for a given workload decreases.

10 The fraction of free space available on a sample drive is plotted along the x-axis; the write amplification is plotted on the y-axis. To increase drive endurance, write amplification should be as small as possible. Shown at left, write amplification decreases as a function of increasing fractional free space. By minimizing write amplification, the number of writes to the physical NAND cells is decreased which in turn increased drive endurance The effect of over-provisioning on user available drive space: Since an over-provisioned drive has a portion of the total capacity reserved for use by the NAND controller exclusively, the available capacity (what the host can use) will differ from the total NAND capacity of the drive. The table below shows representative examples of this difference: Total drive capacity (decimal) Percentage over provisioned 64GB 4% (client) 60GB 64GB 27% (enterprise) 50GB 128GB 4% (client) 120GB 128GB 27% (enterprise) 100GB Nominal drive capacity available to host (decimal) When over provisioning is used, this extra space is hidden from the host. An SSD with 64GB of NAND capacity which employs 27% over provisioning will appear to the host OS as a 50GB drive. Conclusion: We have seen that NAND based SSDs have unique design considerations when compared to conventional hard drives. Factors such as drive performance are common to both SSDs and rotating drives, but maximizing them is achieved differently. Other factors like cell P/E cycle limits, drive endurance, and drive performance degradation as the drive fills are unique to NAND based SSDs. One key technology to optimize NAND based SSDs for all of the above is over-provisioning. By reserving a portion of the physical NAND for use exclusively by the NAND controller (and not made available to the host), all of these unique concerns can be abated, and both drive performance and lifespan increased.