Persistent Memory and Media Errors

Transcription

1 Persistent Memory and Media Errors Vishal Verma Vault

2 Or How to have your Poison and (not) consume it too 2

3 NVDIMM software stack Regular Block IO Application Standard Raw Device Access DAX Application Standard File API Application Standard File API File System Load/Store User Persistent Memory Aware File System MMU Mappings Kernel Libnvdimm Drivers Cache Line I/O NVDIMM 3

4 What is poison Persistent Memory == Persistent Poison What we (storage people) would like What needs to be done in Linux How we did it 4

5 What is Poison 5

6 What is Poison Bad cell in memory Transient or hard/uncorrectable error How platforms deal with it Machine Check Exception Recoverable on high-ras platforms (page is sequestered, app get SIGBUS) OS crash on other platforms If transient, rebooting typically makes the poison page go away If permanently degraded cell, replace DIMM 6

7 What is poison Persistent Memory == Persistent Poison What we (storage people) would like What needs to be done in Linux How we did it 7

8 What is Persistent Poison Bad cell in Persistent memory Will not go away on reboot Without any changes in Linux: Trying to write to it will trigger a machine check Deleting the file won't help either A bad cache line is now a bad filesystem block * Implies data has been lost * Drivers/firmware can easily 'fix' the bad location, but it is imperative to let userspace know of the data loss. The 'fix' has to be triggered by the user/app 8

9 What is Persistent Poison Behavior we want to prevent Application calls read() Hits poison Crashes/gets SIGBUS (reboots) App starts up Tries to access its data (read()) Crash Reebootus-infinitus DAX Application Standard File API mcheck handler Unmap Notify Crash Load/Store Persistent Persistent Memory Memory Aware Aware File File System System User MMU Mappings Kernel Libnvdimm driver Cache Line I/O NVDIMM 9

10 What is poison Persistent Memory == Persistent Poison What we (storage people) would like What needs to be done in Linux How we did it 10

11 What we would like Instead of a SIGBUS/crash, return -EIO Way to expose known poison to Software Ability for Software to clear poison 11

12 What is poison Persistent Memory == Persistent Poison What we (storage people) would like What needs to be done in Linux How we did it 12

13 What needs to be done: Exposing poison Start an Address Range Scrub (ARS) Or harvest results from a previous, automatically started scrub DAX Application Load/Store Expose to App User Libnvdimm gets a list of poison Make it available for libnvdimm drivers and file systems to check on I/Os Expose to FS Persistent Memory Aware File System MMU Mappings Kernel Libnvdimm ARS Expose it to userspace Firmware NVDIMM 13

14 What needs to be done: Clearing poison App provides new data DAX Application Filesystem detects write to a poison Load/Store range and goes through driver Provide new data User Driver calls the clear_poison DSM, and then writes data Provide new data Persistent Memory Aware File System MMU Mappings Kernel Libnvdimm Clear Poison Firmware (handles _DSM) NVDIMM 14

15 What is poison Persistent Memory == Persistent Poison What we (storage people) would like What needs to be done in Linux How we did it 15

16 How we did it: Exposing poison Harvest results from an Address Range Scrub Get SPA relative list of poison Convert SPA poison ranges to bad disk sectors Make md-raid's badblocks code generic $ cat /sys/block/pmem1/badblocks gendisk->badblocks Expose to FS Expose to App /dev/pmem0 /dev/pmem1 Add bad blocks to gendisk Also expose them in sysfs Libnvdimm ARS Firmware NVDIMM 16

17 How we did it: Handling Driver I/O Regular Block IO In the pmem driver, check if a BIO is for a bad sector Application If reading: fail with an -EIO Standard Raw Device Access Standard File API Send clear_poison DSM Clear the sector from gendisk->badblocks Write the new data User File System If writing: Application Kernel Libnvdimm Drivers READ: Write: Check disk->badblocks Return -EIO Check disk->badblocks Send DSM: c lear_poison Clear disk badblocks Write data NVDIMM 17

18 How we did it: Handling DAX I/O DAX ->direct_access() checks for badblocks at Application fault time Standard File API User Load/Store If found, DAX mapping fails Persistent Memory Aware Aware File File System System If writing: Fallback to blockdev_do_direct_io() All zeroing goes through the driver XX Kernel dax_map_atomic Fails with -EIO Libnvdimm If we hit a latent error, SIGBUS/crash, and it will be a 'known' error the next time MMU Mappings Cache Line I/O NVDIMM 18

19 How we did it: 'Blast Radius' DAX 4k The poison is on a cache line granularity Block layer rounds up to a sector (512B) User Fs/DAX will round up to a page (4K) If an app hits a bad page: look up bad sector from sysfs do a write() to clear it Bad Blocks 512 Machine Machine Check Check Handler Handler pmem-aware File System MMU Mappings Libnvdimm Kernel NVDIMM 4k 64 19

20 Q&A