IDO: Intelligent Data Outsourcing with Improved RAID Reconstruction Performance in Large-Scale Data Centers

Transcription

1 IDO: Intelligent Data Outsourcing with Improved RAID Reconstruction Performance in Large-Scale Data Centers Suzhen Wu *, Hong Jiang*, Bo Mao* Xiamen University *University of Nebraska Lincoln

2 Data Deluge Social Network 2,300 tweets per second Scientific Simulation How to safely store such a huge data volume proposes a big challenge to Business Intelligence Mobile Apps the system administrators! 275 EB data flowing per day in

3 Where Are We? Laptop and Desktop Interruptible Event Data Center Common Case 3

4 Disk Failure in the Real World Higher error rates than expected Complete disk failures, 2%~4% on average; Latent sector errors, 3.45%; CorrelaBon in drive failures e.g., acer one disk fails, another disk failure will likely occur soon. RAID reconstrucbon becomes an operabonal state in data centers Increasing disk capacity and number of drives 4

5 More Observations Linux software RAID (MD) mailing list: too many complains about the slow recovery speed. Storage at Exascale: Some thoughts from Panasas CTO Garth Gibson. Disk failure is a normal case in exascale storage systems. 5

6 RAID Reconstruction Challenges Online RAID Reconstruction: User I/O Requests How many user I/O requests can be eliminated from degraded RAID directly Reconstruction I/O Requests affects the reconstruction performance. Two challenges: Real-time user performance; Window of vulnerability. 6

7 The State of the arts Optimizing the reconstruction workflow: DOR (CMU PDL) Live-block recovery (USENIX FAST 04) PRO (USENIX FAST 07) Optimizing the user I/O requests: MICRO (IEEE TC 08) WorkOut (USENIX FAST 09) VDF (USENIX ATC 11) 7

8 Compare with State of the arts Characteris*cs ProacBve PRO (FAST 07) WorkOut (FAST 09) VDF (USENIX 11) Temporal Locality SpaBal Locality User I/O ReconstrucBon I/O IDO (LISA 12) 8

9 Observation 1 RAID reconstruction is an operational state in large-scale data centers which means reactive scheme is inefficient. Reactive vs. Proactive? Existing studies are all reactive schemes. 9

10 Example 1: Reactive vs. Proactive 10

11 Example 1: Reactive vs. Proactive 11

12 Observation 2 With large RAM and SSDs, the temporary locality is poor at HDD level. However, the spatial locality is good due to the sequential accesses of HDDs. Temporal locality vs. Spatial locality? Existing studies mostly focus on temporal locality and ignore spatial locality. 12

13 Example 2: Temporal vs. Spatial (a) Request-based approach a b c d Migrate requested a to Surrogate Set a (b) Zone-based approach a b c d Migrate hot zone to Surrogate Set a b c d 13

14 The Motivation User I/O traffic removed from degraded RAID 100% 80% 60% 40% 20% 0% Reactive-request Reactive-zone Proactive-request Proactive-zone WebSearch2 Financial2 Microsoft Project 14

15 IDO: Intelligent Data Outsourcing The main idea: Proactively identify the hot data zones; Upon disk failure, Recovery the hot data zones first; Migrate the hot data zones to surrogate set; Redirect the user I/O requests. The design objectives Reducing reconstruction time; Improving the user I/O performance; Applicable to other background tasks. 15

16 System Overview Hot Zone Identifier Hot Zone Identifier Request Distributor Request Distributor Data Migrator Data Migrator Data Reclaimer Data Reclaimer Network Storage Node IDO Software RAID Controller RAID Reconstruction Storage Node IDO Software RAID Controller RAID Reconstruction Failed Disk New Disk Working / Degraded RAID Surrogate RAID Working / Surrogate RAID Data Migration 16

17 Performance Evaluation IDO prototype is a built-in module in Linux MD, compared with WorkOut and VDF. Intel Xeon 3440 processor, 8GB DDR memory, WDC WD1600AAJS SATA disks. Trace-driven evaluations 17

18 RAID5 Results Average Response Time (ms) WorkOut VDF IDO Fin1 Fin2 Web2 Proj Reconstruction Time (s) WorkOut VDF IDO Fin1 Fin2 Web2 Proj (a) Average Response Time during Recovery (b) Reconstruction Time 18

19 RAID6 Results Average Response Time (ms) WorkOut VDF IDO Fin1 Fin2 Web2 Proj Reconstruction Time (s) WorkOut VDF IDO Fin1 Fin2 Web2 Proj (a) Average Response Time during Recovery (b) Reconstruction Time 19

20 Detailed Real-time Results Shorter Reconstruction Times User Response Time (ms) WorkOut VDF IDO WorkOut ends IDO ends VDF ends Reconstruction Time (s) User Response Time (ms) WorkOut VDF IDO IDO ends VDF ends WorkOut ends Reconstruction Time (s) (a) WebSearch2.spc (b) Microsoft Project Shorter Lower Reconstruction user response times Time 20

21 Reduce I/Os and Sensitivity Study Reduced I/Os: Percentage (% of Total) WorkOut VDF IDO Fin1 Fin2 Web2 Proj Sensitivity & overhead analysis (in the paper). 21

22 Extendibility Evaluation Re-synchronization Time (s) Default WorkOut IDO Fin1 Fin2 Web2 Proj Average Response Time (ms) Default WorkOut IDO Fin1 Fin2 Web2 Proj (a) Re-synchronization Time (b) Average Response Time 22

23 Summary of IDO RAID reconstruction is an operational state in large-scale data centers! Salient features of IDO: Proactive; Exploit both temporal and spatial localities; Optimize both user and reconstruction IOs; Portability and extendibility. 23

24 Thanks! 24