ZFS Benchmarking. eric kustarz blogs.sun.com/erickustarz

Transcription

1 Benchmarking eric kustarz blogs.sun.com/erickustarz

2 Agenda Architecture Benchmarks We Use Tools to Analyze Some Examples

3

4 FS/Volume Model vs. FS/Volume I/O Stack Block Device Interface Write this block, then that block,... Block Device Interface Object-Based Transactions FS Write each block to each disk immediately to keep mirrors in sync Loss of power = resync Synchronous and slow Loss of power = loss of on-disk consistency Workaround: journaling, which is slow & complex I/O Stack Make these 7 changes to these 3 objects All-or-nothing Transaction Group Commit Volume Again, all-or-nothing Always consistent on disk No journal not needed Transaction Group Batch I/O ZPL Schedule, aggregate, and issue I/O at will No resync if power lost Runs at platter speed DMU SPA

5 FS/Volume Model vs. Traditional Volumes Abstraction: virtual disk Partition/volume for each FS Grow/shrink by hand Each FS has limited bandwidth Storage is fragmented, stranded FS FS FS Volume Volume Volume Pooled Storage Abstraction: malloc/free No partitions to manage Grow/shrink automatically All bandwidth always available All storage in the pool is shared Storage Pool

6 Dynamic Striping Automatically distributes load across all devices Writes: striped across all four mirrors Reads: wherever the data was written Block allocation policy considers: Capacity Performance (latency, BW) Health (degraded mirrors) 2 3 Writes: striped across all five mirrors Reads: wherever the data was written No need to migrate existing data Old data striped across 1-4 New data striped across 1-5 COW gently reallocates old data Storage Pool 1 Storage Pool Add Mirror

7 Copy-On-Write Transactions 1. Initial block tree 2. COW some blocks 3. COW indirect blocks 4. Rewrite uberblock (atomic)

8 Self-Healing Data in 1. Application issues a read. 2. tries the second disk. 3. returns good data mirror tries the first disk. Checksum reveals that the block is corrupt on disk. Checksum indicates that the block is good. Application Application Application mirror mirror mirror to the application and repairs the damaged block.

9 Benchmarks We Use FileBench One offs Whole assortment of other benchmarks postmark kenbus bigdir2 tar mkfile iozone dd bonnie etc. specsfs netbench full build OLTP-net specweb99

10 Tools available on OpenSolaris Dtrace precise answers to arbitrary questions Lockstat 'lockstat -P -D 20 sleep 60' 'lockstat -kgiw -D 20 sleep 60' fsstat # fsstat -f /tank kstats vmstat

11 vq_max_pending and Dtrace maximum num of I/Os sends to each device does the I/O scheduling defaults to 35 zfs_vdev_max_pending tunable Use dscript to see if avg. # of pending I/Os is hitting max

12 VOPs and Dtrace Dscript to find time spent in each VOP call #./zvop_times.d dtrace: script './zvop_times.d' matched 66 probes ^C CPU ID FUNCTION:NAME 17 2 :END COUNT zfs_fsync 61 zfs_write 494 zfs_read 520 AVG TIME zfs_read zfs_write zfs_fsync SUM TIME zfs_read zfs_write zfs_fsync

13 ARC kstats # kstat zfs::arcstats fsh-suzuki# kstat zfs::arcstats module: zfs instance: 0 name: arcstats class: misc... hits l2_hits 0 l2_misses 0... prefetch_data_hits 0 prefetch_data_misses 0 prefetch_metadata_hits 0 prefetch_metadata_misses 0 size fsh-suzuki#

14 vdev cache kstats # kstat zfs::vdev_cache_stats fsh-suzuki# kstat zfs::vdev_cache_stats module: zfs instance: 0 name: vdev_cache_stats class: misc crtime delegations 1008 hits 89 misses 28 snaptime fsh-suzuki#

15 vdev cache example Software track buffer per vdev Inflate I/Os under 16KB to 64KB Best case: Worst case: 128 back to back 512B reads into 1 64KB read Isolated 512B read Used to be for all I/Os, now only for metadata

16 vdev cache results FileBench's OLTP workload vdev cache not needed ops/s (data and metadata inflated) ops/s (only metadata inflated) FileBench's multi-stream read workload Inflation of random reads hurts no inflation done on 1MB reads inflation of metadata I/Os helps, vdev cache needed ops/s (data/metadata potentially inflated) ops/s (vdev cache disabled)

17 NCQ enbaled vs. disabled FileBench's RandomRead workload iosize of 128KB, 1 40GB file, 32 threads 1 32 disk RAID0 more iops and more BW for NCQ enabled FileBench's MultiStreamRead workload iosize of 1MB, 40GB files, 1 stream per file, 1 thread per stream 46 disk RAIDO 1 6 files/streams worse BW with NCQ enabled 500GB Hitachis with AJOA firmware

18 DNLC Should improve lookup performance replacing specific znode cache How to measure benefit of adding DNLC? specsfs is very lookup dependent specsfs 3.0 has 27% lookups, % small system went from 2.5k ops to 17k tar is lookup dependent time to tar up 25k files (zero length) no DNLC: 3.9 real, 0.9 user, 2.9 sys with DNLC: 1.6 real, 0.8 user, 0.8 sys

19 What I would like to have More FileBench workloads real world officially verified Tools to measure power consumption have ops/s, latency, CPU usage today Standard output for various benchmarks including what should be included in the output

20 Benchmarking eric kustarz blogs.sun.com/erickustarz