Duy Le (Dan) - The College of William and Mary Hai Huang - IBM T. J. Watson Research Center Haining Wang - The College of William and Mary

Transcription

1 Duy Le (Dan) - The College of William and Mary Hai Huang - IBM T. J. Watson Research Center Haining Wang - The College of William and Mary

2 Virtualization Games Videos Web Games Programming File server Web server Database server Mail server [Boutcher-Hotstorage 9] Different [Tang-ATC 11] I/O scheduler combinations Performance Implications Disks Disks Disks of File Systems [Jujjuri-LinuxSym 1] VirtFS - File system passthrough File Systems File Systems File Systems [Tang-ATC 11] Storage space allocation and tracking dirty blocks functionality optimization Storage Storage 2

3 Selected file systems are based on workloads Only true in physical systems File systems for guest virtual machine Workloads Deployed file systems (at host level) Investigation needed! Guest File Systems Ext2, Ext3, Ext4, ReiserFS, XFS, and JFS Host File Systems Ext2, Ext3, Ext4, ReiserFS, XFS, and JFS 3

4 For the best performance? Best and worst Guest/Host File System combinations? Guest and Host File System Dependency Varied I/Os and interaction File disk images and physical disks 4

5 Experimentations Macro level Throughout analysis Micro level Findings and Advice 5

6 Experimentations Macro level Throughout analysis Micro level Findings and Advice 6

7 9 x 16 blocks Qemu.9.1; 512MB RAM Linux Guest File Systems vdc1 Ext2 vdc2 Ext3 vdc3 Ext4 ReiserFS vdc4 vdc5 XFS vdc6 JFS VirtIO File Systems sdb1 Ext2 sdb2 Ext3 sdb3 Ext4 ReiserFS sdb4 sdb5 XFS sdb6 JFS sdb7 BD Host 6 x 16 blocks Sparse disk image Pentium D 3.4 GHz, 2GB Ram Linux Qemu-KVM TB, SATA 6Gb/s, 64MB Cache Raw partition as block device (BD) 6 x 16 blocks 7

8 Filebench File server, web server, database server, and mail server. Throughput Latency I/O Performance Different abstraction consideration Via block device (BD) Via nested file systems Relative performance variation BD as baseline 8

9 Relative performance to baseline of guest file systems Baseline Host file systems 9

10 ReiserFS guest file system 1

11 Throughput (MB/s) BD Ext2 Ext3 Ext4 ReiserFS XFS JFS Percentage (%) R ReiserFS Ext4 guest file system 11

12 Guest file system Host file systems Varied performance Host file system Guest file systems Impacted differently Right and wrong combinations Bidirectional dependency I/Os behave differently Writes is more critical than Read (mail server) 12

13 Throughput (MB/s) BD Ext2 Ext3 Ext4 ReiserFS XFS JFS Percentage (%) R Ext2 guest file system Ext3 guest file system 13

14 Guest file system Host file systems Varied performance Host file system Guest file systems Impacted differently Right and wrong combinations Bidirectional dependency (mail server) I/Os behave differently Writes is more critical than Read (mail server) 14

15 Throughput (MB/s) BD Ext2 Ext3 Ext4 ReiserFS XFS JFS Percentage (%) R Ext2 guest file system 15

16 Guest file system Host file systems Varied performance Host file system Guest file systems Impacted differently Right and wrong combinations Bidirectional dependency I/Os behave differently Writes is more critical than Reads 16

17 Throughput (MB/s) BD Lower than 1% Ext2 Ext3 Ext4 ReiserFS XFS JFS Percentage (%) R More WRITES 17

18 Guest file system Host file systems Varied performance Host file system Guest file systems Impacted differently Right and wrong combinations Bidirectional dependency I/Os behave differently WRITES are more critical than READS 18

19 Guest file system Host file systems Varied performance Host file system Guest file systems Impacted differently Right and wrong combinations Bidirectional dependency I/Os behave differently WRITES are more critical than READS Latency is sensitive to nested file systems 19

20 Experimentations Macro level Throughout analysis Micro level Findings and Advice 2

21 Same testbed Primitive I/Os Reads or Writes Random or Sequential FIO benchmark Description Total I/O size Parameters 5 GB I/O parallelism 255 Block size I/O pattern I/O mode 8 KB Random/Sequential Native async I/O 21

22 Throughput (MB/s) BD Ext2 Ext3 Ext4 ReiserFS XFS JFS Percentage (%) R Random Throughput (MB/s) BD Ext2 Ext3 Ext4 ReiserFS XFS JFS Percentage (%) R Sequential Ext3 guest file system 22

23 Read dominated workloads Unaffected performance by nested file systems Write dominated workloads Heavily affected performance by nested file systems 23

24 Throughput (MB/s) BD Ext2 Ext3 Ext4 ReiserFS XFS JFS Percentage (%) R Random Throughput (MB/s) BD Ext2 Ext3 Ext4 ReiserFS XFS JFS Percentage (%) R Sequential 24

25 Read dominated workloads Unaffected performance by nested file systems Write dominated workloads Heavily affected performance by nested file systems 25

26 Throughput (MB/s) 1 Throughput (MB/s) 1 Ext3 guest file system Read 8 dominated workloads Unaffected performance by nested file systems 6 4 Write 4dominated Ext3 workloads guest file system 2 2 Heavily affected performance by nested 2 file systems BD R JFS guest file 12 system Ext2 Ext3 Ext4 ReiserFS XFS JFS BD Ext2 Ext3 Ext4 ReiserFS XFS JFS Percentage (%) Percentage (%) R Sequential Reads: Ext3/JFS vs. Ext3/BD Sequential Writes: Ext3/ReiserFS vs. JFS/ReiserFS (same host file systems) JFS/ReiserFS vs. JFS/XFS (same guest file systems) I/O analysis using blktrace 26

27 Findings: Readahead at the hypervisor when nesting FS Long idle times for queuing 27

28 Different guests (Ext3, JFS) same host (ReiserFS) I/O scheduler and Block allocation scheme 28

29 Long waiting in the queue Well merged Low I/Os for journaling Ext3 causes multiple back merges JFS coalescences multiple log entries 29

30 Different guests (Ext3, JFS) same host (ReiserFS) I/O scheduler and Block allocation scheme Findings I/O schedulers are NOT effective for ALL nested file systems I/O scheduler s effectiveness on block allocation scheme 3

31 Different guests (Ext3, JFS) same host (ReiserFS) Same guest (JFS) different hosts (ReiserFS, XFS) Block allocation schemes 31

32 Longer waiting in queue Fairly similar CDF of disk I/Os ReiserFS XFS Long distance seeks Normalized seek distance Repeated logging 32

33 Different guests (Ext3, JFS) same host (ReiserFS) Same guest (JFS) different hosts (ReiserFS, XFS) Block allocation schemes Findings: Effectiveness of guest file system s block allocation is NOT guaranteed Journal logging on disk images lowers the performance 33

34 Experimentations Macro level Throughout analysis Micro level Findings and Advice 34

35 Advice 1 Read-dominated workloads Minimum impact on I/O throughput Sequential reads: even improve the performance Advice 2 Write-dominated workloads Nested file system should be avoided One more pass-through layer Extra metadata operations Journaling degrades performance 35

36 Advice 3 I/O sensitive workloads I/O latency increased by 1-3% Advice 4 Data allocation scheme Data and Metadata I/Os of nested file systems are not differentiated at host Pass-through host file system is even better! Advice 5 Tuning file system parameters Non-smart disk Noatime and nodiratime 36

37 37

38 Devices Blocks (x1 6 ) Speed (MB/s) Type sdb Ext2 sdb Ext3 sdb Ext4 sdb ReiserFS sdb XFS sdb JFS sdb Block Device 38