L79. Linux Filesystems. Klaus Bergmann Linux Architecture & Performance IBM Lab Boeblingen. zseries Expo, November 10-14, 2003 Hilton, Las Vegas, NV

Transcription

1 L79 Linux Filesystems Klaus Bergmann Linux Architecture & Performance IB Lab Boeblingen zseries Expo, November 10-14, 2003 Hilton, Las Vegas, NV

2 Trademars The following are trademars of the International Business achines Corporation in the United States and/or other countries. Enterprise Storage Server ESCON* FICON FICON Express HiperSocets IB* IB logo* IB eserver Netfinity* S/390* V/ESA* WebSphere* z/v zseries * Registered trademars of IB Corporation The following are trademars or registered trademars of other companies. Intel is a trademar of the Intel Corporation in the United States and other countries. Java and all Java-related trademars and logos are trademars or registered trademars of Sun icrosystems, Inc., in the United States and other countries. Lotus, Notes, and Domino are trademars or registered trademars of Lotus Development Corporation. Linux is a registered trademar of Linus Torvalds. icrosoft, Windows and Windows NT are registered trademars of icrosoft Corporation. Penguin (Tux) compliments of Larry Ewing. SET and Secure Electronic Transaction are trademars owned by SET Secure Electronic Transaction LLC. UNIX is a registered trademar of The Open Group in the United States and other countries. * All other products may be trademars or registered trademars of their respective companies.

3 Agenda Journaling file systems easurement setup easurement results LPAR V 31 / 64 bit single dis and LV DASD statistics CPU load and CP overhead journaling options Outloo

4 Problems of non-journaling file systems data and meta-data is written directly and in arbitrary order no algorithm to ensure data integrity after crash, complete structure of file system has to be checed to ensure integrity file system chec times depend on size of file system i ris of data loss i long and costly system outages

5 advantages of journaling data integrity ensured in case of system crash only journal has to be replayed to recover consistent file system structure file system chec time depends on size of journal i much higher data integrity i much shorter system outages but there is a cost...

6 Journaling file systems in SuSE SLES8 ext3 v jfs reiserfs For reference : ext2 v0.5 (non-journaling)

7 ext3 developed by Andrew orton and others based on ext2 extended by journaling features supports full data journaling resizing (only with unmount) possible

8 jfs developed by IB Austin Lab ported from OS/2 Warp Server only metadata journaling max. file system size 4 PB

9 reiserfs developed by a group around Hans Reiser SUSE's default choice only metadata journaling dis space optimization algorithm online enlargement of file system

10 easurement setup Hardware (z900) 1.09ns (917Hz) 2 * 16 B L2 Cache (shared) 64 GB 6 FICON channels 2105-F20 (Shar) 384 B NVS 16 GB Cache 128 * 36 GB diss RP FICON (1 Gbps) Software SUSE SLES8 Dbench

11 easurement setup dbench 128B main memory 1, 2 and 4 CPUs LPAR and z/v bit and 64-bit Single 3390 model 3 dis 6 pac of using striped LV. Attached via 6 FICON channels Running 8 and 16 processes

12 Dbench File I/O Emulation of Netbench benchmar, rates windows fileservers Large set of mixed file operations worload for each process: create, write, read, append, delete Scaling for Linux with 1, 2, 4 PUs Scaling for 8 and 16 clients (processes) simultaneously forced to do I/O while memory is filling up with data

13 easurement results

14 LPAR and V Throughput [B/s] single dis, LPAR and V, 31bit, 4 CPUs 8 proc., LPAR 16 proc., LPAR 8 proc., V 16 proc., V Ext2 Ext3 Jfs Reiserfs filesystem

15 31-bit and 64-bit Throughput [B/s] single dis, V, 4 CPUs Ext2 Ext3 Jfs Reiserfs filesystem 8 proc., 31bit 16 proc., 31bit 8 proc., 64bit 16 proc., 64bit

16 /proc/dasd/statistics Example cat /proc/dasd/statistics dasd I/O requests with sectors(512b each) < _128 _256 _ _16 _32 _ _256 _ _16 _32 _ G 2G 4G _>4G Histogram of sizes (512B secs) Histogram of I/O times (microseconds) Histogram of I/O times per sector Histogram of I/O time till ssch Histogram of I/O time between ssch and irq Histogram of I/O time between ssch and irq per sector Histogram of I/O time between irq and end # of req in chanq at enqueuing (1..32)

17 < _12 8 _25 6 _ _16 _32 _ _25 6 _ _16 _32 _ G 2 G 4 G _>4 G ext2, 8 Processes Histogram of I/O times (microseconds)

18 < _12 8 _25 6 _ _16 _32 _ _25 6 _ _16 _32 _ G 2 G 4 G ext2, 16 Proceses Histogram of I/O times (microseconds)

19 < _12 8 _25 6 _ _16 _32 _ _25 6 _ _16 _32 _ G 2 G 4 G ext3, 8 Processes Histogram of I/O times (microseconds)

20 < _12 8 _25 6 _ _16 _32 _ _25 6 _ _16 _32 _ G 2 G 4 G ext3, 16 Processes Histogram of I/O times (microseconds)

21 ext3, 16 Processes Histogram of I/O time before SSCH (IOSQ) < _12 8 _25 6 _ _16 _32 _ _25 6 _ _16 _32 _ G 2 G 4 G

22 18000 Ext3, 16 Processes Histogram of I/O time between SSCH and IRQ < _1 28 _2 56 _ _1 6 _3 2 _ _2 56 _5 12 _1 _3 _ G 2G 4G

23 Ext3, 16 Processes number of requests in subchannel-queue at enqueuing

24 Logical Volume anager (LV) Linux software raid with raid levels 0,1, 4 and 5 excellent performance excellent flexibility (resizing, adding/removing diss) available in SLES7, SLES8, and RedHat RHEL 3 on zseries, support multipath and PAV (under z/v)

25 (journaled) file system LV system structure Logical Volume anager physical dis physical dis RAID adapter RAID array

26 Improving dis performance with LV striped datastream physical volume physical volume physical volume With LV and striping parallelism is achieved

27 Throughput [B/s] LV results single dis - LV comparison, z/v, 31bit, 2 CPUs Ext2 Ext3 Jfs Reiserfs filesystem 8 proc., LV 16 proc., LV 8 proc., single dis 16 proc., single dis

28 filesystem options single dis, ext3 optimizations, z/v, 31bit, 4 CPUs 45 single dis, reiserfs optimizations, z/v, 31bit, 4 CPUs Throughput [B/s] Throughput [B/s] processes 16 processes 5 0 ordered writebac full data external journal 100 journal 200 journal 400 journal 5 0 default no hash relocation no border allocation no unhashed relocation single dis, jfs optimizations, z/v, 31bit, 4 CPUs Throughput [B/s] default external journal 100 journal 200 B journal

29 CPU load 100 LPAR, 1 CPU, 8 processes, single dis CPU load [%] idle system user 0 ext2, 31bit ext2, 64bit ext3, 31bit ext3, 64bit reiserfs, 31bit filesystem reiserfs, 64bit jfs, 31bit jfs, 64bit

30 V overhead LV CPU consumption CPU consumption CP Guest ext2, 31bit ext2, 31bit, LV ext3, 31bit ext3, 31bit, LV reiserfs, 31bit reiserfs, 31bit, LV jfs, 31bit jfs, 31bit, LV

31 recovery times

32 Outloo on ernel 2.6 Throughput [B/s] journaling filesystem in ernel version 2.4 vs Kernel preliminary results jfs was not compiled into 2.6 ernel Kernel test5 ext3 jfs reiserfs

33 Summary journaling file systems increase data integrity significantly journaling file systems dramatically reduce system outage times performance cost is at least 30% reiserfs is slightly faster than ext3, but needs much more CPU journaling file systems profit from LV jfs has fastest recovery times 2.6 will bring more improvements (increased throughput, reduced CPU load, iostat for ECKD)

34 Questions?