SSD Failures in Datacenters: What? When? And Why?

Transcription

1 SSD Failures in Datacenters: What? When? And Why? Iyswarya Narayanan, Di Wang, Myeongjae Jeon, Bikash Sharma, Laura Caulfield, Anand Sivasubramaniam, Ben Cutler, Jie Liu, Badriddine Khessib, Kushagra Vaid The 9 th ACM Systems And Storage Conference (SYSTOR 2016) 1

2 Why SSD Reliability? Data reliability SSDs popularity % annual growth* Datacenter decision support Limited field data *Source: IDC, Dec

3 Why SSD Reliability? Data reliability SSDs popularity Large scale Datacenter decision support Field data 46.5% annual growth* Limited field data *Source: IDC, Dec

4 SSD Failures Flash failures - Media wear-out - Data Retention - Program disturb - Erase disturb FTL Mechanisms - Wear levelling - Error detection - Error correction - Flash correct and refresh, etc. 4

5 SSD Failures Flash failures - Media wear-out - Data Retention - Program disturb - Erase disturb FTL Mechanisms - Wear levelling - Error detection - Error correction - Flash correct and refresh, etc. 5

6 SSD Failures Flash failures - Media wear-out - Data Retention - Program disturb - Erase disturb FTL Mechanisms - Wear levelling - Error detection - Error correction - Flash correct and refresh, etc. 6

7 SSD Failures Flash failures - Media wear-out - Data Retention - Program disturb - Erase disturb FTL Mechanisms - Wear levelling - Error detection - Error correction - Flash correct and refresh, etc. 7

8 SSD Failures Flash failures - Media wear-out - Data Retention - Program disturb - Erase disturb FTL Mechanisms - Wear levelling - Error detection - Error correction - Flash correct and refresh, etc. 8

9 SSD Failures Flash failures - Media wear-out - Data Retention - Program disturb - Erase disturb FTL Mechanisms - Wear levelling - Error detection - Error correction - Flash correct and refresh, etc. Fail-stop failures 9

10 Annualized Failure Rate % SSD Reliability AFR=0.61 AFR= A 1-B 1-C 1-D 2-A SSD Model Consumer Enterprise 10

11 Annualized Failure Rate % SSD Reliability AFR=0.61 AFR= A 1-B 1-C 1-D 2-A SSD Model 11

12 Annualized Failure Rate % SSD Reliability AFR=0.61 AFR= A 1-B 1-C 1-D 2-A SSD Model 5 large datacenters 12

13 Annualized Failure Rate % SSD Reliability AFR=0.61 AFR= A 1-B 1-C 1-D 2-A SSD Model 4 major workloads 13

14 Annualized Failure Rate % SSD Reliability AFR=0.61 AFR= A 1-B 1-C 1-D 2-A SSD Model 6 different rack SKUs 14

15 Annualized Failure Rate % Various factors in production environment could affect SSD failure trends very differently from lab test conditions SSD Reliability AFR=0.61 AFR= A 1-B 1-C 1-D 2-A Can we understand SSD failures in the presence of SSD Model various factors? 15

16 Understanding SSD Failures An analogy SSD Reactive Proactive 16

17 What are the symptoms? SSD Fever Unexpected weight loss Reallocated sectors ?00101? Data errors Low blood pressure Program and erase failure SATA downshift 17

18 AFR % SSD Failure Symptoms Reallocated Sector Count Program and Erase Fail Count CRC and Uncorrectable Error Count SATA Downshift Count X Reallocated Sector Count w Symptom w/o Symptom 2.76X Program and Erase Failure Count 18X CRC and Uncorrectable Error Count 3.91X SATA Downshift Count 18

19 % of devices Insufficiency of symptom only diagnosis Reallocations Program and Erase Fail Failed Healthy Symptoms seen only in 62% of failed devices Data Errors SATA Downshift Any 19

20 What are the factors? SSD Lifestyle Workload Environmental agents Production environment Genetics Design decisions 20

21 >50 AFR % Device level correlating factors Average write rate of a device Average read rate of a device Total read and/or write usage Write Amplification Read Write Ratio 0 Increasing failure trend at higher write rates Avg. host writes per day More results in the paper 21

22 AFR% Server level correlating factors SSD space utilization Disk space utilization Memory utilization Processor utilization Decreasing failure trend at high disk space usage Avg. Disk Space Utilization More results in the paper 22

23 AFR % Datacenter factors Same model different behavior Rack SKU Datacenter Facility D 2-A 1-D 2-A S1-3a S1-3b SKU and SSD model More results in the paper 23

24 Understanding SSD Failures An analogy MULTI FEATURE ANALYSIS SSD Symptoms Factors Symptoms Factors 24

25 Understanding SSD Failures An analogy SSD Symptoms Factors Symptoms Factors Random forest based binary classification Permutation feature ranking 25

26 Understanding What? What are the important factors? is their order of importance? are the important combinations? 26

27 Understanding What? Feature Importance SYMPTOMS DataErrors ReallocSectors TotalNANDWrites HostWrites TotalReads+Writes AvgMemory AvgSSDSpace UsagePerDay TotalReads ReadsPerDay 27

28 Understanding What? Feature Importance DEVICE WORKLOAD DataErrors ReallocSectors TotalNANDWrites HostWrites TotalReads+Writes AvgMemory AvgSSDSpace UsagePerDay TotalReads ReadsPerDay 28

29 Understanding What? Feature Importance SERVER WORKLOAD DataErrors ReallocSectors TotalNANDWrites HostWrites TotalReads+Writes AvgMemory AvgSSDSpace UsagePerDay TotalReads ReadsPerDay 29

30 Understanding What? Combinations of top 8 important features SYMPTOMS Frequent Combinations Condition Data Errors <=1 & Reallocated Sectors<=5 Data Errors<=1& WAF<=1 Media Wear-out=100 & WAF<=1 Avg. SSD space >=10 Class H H H F 30

31 Understanding What? Combinations of top 8 important features SYMPTOMS + WORKLOAD Frequent Combinations Condition Data Errors <=1 & Reallocated Sectors<=5 Data Errors<=1& WAF<=1 Media Wear-out=100 & WAF<=1 Avg. SSD space >=10 Class H H H F 31

32 Understanding What? Combinations of top 8 important features WORKLOAD Frequent Combinations Condition Data Errors <=1 & Reallocated Sectors<=5 Data Errors<=1& WAF<=1 Media Wear-out=100 & WAF<=1 Avg. SSD space >=10 Class H H H F 32

33 Understanding When? What is the duration between detection and failure? signatures characterize SSD survivability? 33

34 CDF(x) Understanding When? % of failures > 4 months Sufficient time to intervene Time To Fail (months) 34

35 CDF(x) Understanding When? % of failures Early failures (< 1 month): Rules include symptoms and their thresholds > 4 months Late failures: Rules contains only workload factors Time To Fail (months) 35

36 Understanding SSD Failures An analogy SSD Symptoms Factors Symptoms Factors Observation based causal estimate Probabilistic causal models and Pearl s do-calculus 36

37 Understanding Why? What factors impact SSD reliability? is their magnitude of impact? 37

38 Understanding Why? SSD model and symptoms have direct impact Workload impacts failures through media wearout 38

39 Concluding Remarks SSD Failures in the field Factors -> Symptoms -> Failures Important Symptoms: Data Errors and Reallocated Sectors High intensity and rapid progression fails early Important factors: NAND Writes, Total Reads and Writes, etc. Direct impact: SSD Model and Symptoms Indirect impact: Workload through wear-out Future direction: prediction and control 39