Linux on System z - Disk I/O Alternatives

Transcription

1 Mustafa Mesanovic 3/21/11 Linux on System z - Disk I/O Alternatives visit us at Linux on System z Performance Evaluation

2 Trademarks IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at Copyright and trademark information at Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Oracle and Java are registered trademarks of Oracle and/or its affiliates in the United States, other countries, or both. Other product and service names might be trademarks of IBM or other companies. 2 Linux on System z - Disk I/O Alternatives

3 Agenda Introduction Systems OS Configurations Storage Server Results ECKD SCSI ECKD vs SCSI 3 Linux on System z - Disk I/O Alternatives

4 Introduction This presentation should give an overview about: what types of disks can be provided by the DS8x00 what types of setups are possible which setup and discipline shows the best performance All measured with Linux on System z running in LPAR. 4 Linux on System z - Disk I/O Alternatives

5 Configuration: System z Linux running in a LPAR z9 LPAR: 8 CPUs 512MiB 8 FICON Express4 Linux: SLES11 SP1 (with HyperPAV) Kernel: default Device-mapper: multipath: version striped: version multipath-tools Linux on System z - Disk I/O Alternatives

6 Configuration: DS8700 DS8700: 359.8GiB Cache High Performance FICON, Read Write Track Data and HyperPAV feature connected with 8 FCP/FICON channels/paths Firmware level: Disks storage pool striped (SPS) Extent pool over multiple ranks in our case over 4 ranks Single rank (SR) One Extent pool per rank Further details about DS8800 here: 6 Linux on System z - Disk I/O Alternatives

7 Benchmark description Workload Threaded I/O benchmark Each process writes or reads to a single file, volume or disk Benchmark can be configured to run with and without page cache (direct I/O) Operating modes: Sequential write/rewrite/read + Random write/read Setup Main memory was restricted to 512MB File size (overall): 2GB, Record size: 64kB Run with 1, 8 and 32 processes Sequential run: write, rewrite, read Random run: write, read (with previous sequential write) Optionally using Direct I/O (bypass the page cache) Sync and Drop Caches prior to every invocation to reduce noise 7 Linux on System z - Disk I/O Alternatives

8 Measurement scenarios For FICON/ECKD (8paths) 1 single disk no aliases (1sd) vs. 1 storage pool striped disk using 7 aliases with HyperPAV (1sps+hpav) 8 single disks, 7 aliases, as striped logical volume (8sd+hpav+lv) vs. 2 storage pool striped disks, 7 aliases, as logical volume (2sps+hpav+lv) For FCP/SCSI (1path per disk/8paths per disk using multipath-tools) 1 single disk (1sd) vs. 1 sps disk (8 paths, multipathing using multibus) (1sps+mb) 8 single disks, 8 paths per disk, multipathing using multibus, as logical volume (8sd+mb+lv) vs. 2 sps disks 8 paths per disk, multipathing using multibus, as logical volume (2sps+mb+lv) 8 Linux on System z - Disk I/O Alternatives

9 FICON/ECKD dasd I/O to a single disk Application program VFS Assume that subchannel a corresponds to disk 2 in rank 1 The full choice of host adapters can be used Only one I/O can be issued at a time through subchannel a All other I/Os need to be queued in the dasd driver and in the block device layer until the subchannel is no longer busy with the preceding I/O DA ranks Block device layer Page cache I/O scheduler dasd driver Channel subsystem a! chpid 1 chpid 2 chpid 3 Switch HBA 1 HBA 2 HBA 3 Server chpid 4 HBA 4 Server Linux on System z - Disk I/O Alternatives

10 Flashback: DS8000 storage pool striped volume A storage pool striped volume (rotate extents) is defined on a extent pool consisting of several ranks It is striped over the ranks in stripes/extents of 1 GB As shown in the example The 1 GB stripes/extents are rotated over all ranks The next storage pool striped volume would start in rank volume ranks Disk drives 10 Linux on System z - Disk I/O Alternatives

11 FICON/ECKD dasd I/O to a storage pool striped volume with HyperPAV (SLES11) Application program VFS A storage pool striped volume makes only sense in combination with PAV or HyperPAV VFS sees one device The dasd driver sees the real device and all alias devices The storage pool striped volume spans over several ranks of one server and overcomes the limitations of a single rank and / or a single device adapter Storage pool striped volumes can also be used as physical disks for a logical volume to use both server sides. To ensure that I/O to one dasd is not limited by one subchannel, PAV or HyperPAV should be used Block device layer Page cache I/O scheduler dasd driver Channel subsystem a b c d chpid 1 chpid 2 chpid 3 chpid 4 Switch HBA 1 HBA 2 HBA 3 HBA 4! Server 0 Server 1 DA ranks extent pool Linux on System z - Disk I/O Alternatives

12 ECKD (page cache) - 1 sd vs. 1sps+hpav Throughput for initial writers Throughput for readers 1sd 1sps+hpav Throughput number of processes Combination of storage pool striped disk and HyperPAV boosts a single disk enormously Improves write up to 4x and read up to 7x 12 Linux on System z - Disk I/O Alternatives

13 ECKD (page cache) - 1 sd vs. 1sps+hpav Throughput for random writers Throughput for random readers 1sd 1sps+hpav Throughput number of processes A storage pool striped performs better than a single disk due to having a 4x more NVS space Random read is up to 7x better, random write is still 3.6x better with 32 processes 13 Linux on System z - Disk I/O Alternatives

14 ECKD (dio) - 8sd+hpav+lv vs. 2sps+hpav+lv Throughput for initial writers Throughput for readers 8sd+hpav+lv 2sps+hpav+lv Throughput number of processes Sequential I/O using 8 disks improves up to 1.8x for read and 1.2x for write The number of disks and stripes in the striped logical volume, do make the difference 14 Linux on System z - Disk I/O Alternatives

15 ECKD (dio) 8sd+hpav+lv vs. 2sps+hpav+lv Throughput for random writers Throughput for random readers 8sd+hpav+lv 2sps+hpav+lv Throughput number of processes Random read is up to 1.8x better for 8 single disks Number of Ranks involved is the same, 8 in 2 sps and 8 for 8 sd 15 Linux on System z - Disk I/O Alternatives

16 FCP/SCSI LUN I/O to a single disk Application program VFS Assume that disk 3 in rank 8 is reachable via channel 6 and host bus adapter 6 Up to 32 (default value) I/O requests can be sent out to disk 3 before the first completion is required The throughput will be limited by the rank and / or the device adapter There is no high availability provided for the connection between the host and the storage server DA ranks Block device layer Page cache I/O scheduler SCSI driver zfcp driver qdio driver 16 Linux on System z - Disk I/O Alternatives! chpid 5 chpid 6 chpid 7 chpid 8 Switch HBA 5 HBA 6 HBA 7 HBA 8 Server 0 Server 1!

17 FCP/SCSI LUN I/O to a storage pool striped volume with multipathing Application program VFS Storage pool striped volumes make no sense without high availability VFS sees one device The device mapper sees the multibus or failover alternatives to the same disk The storage pool striped volume spans over several ranks of one server and overcomes the limitations of a single rank and / or a single device adapter Storage pool striped volumes can also be used as physical disks for a logical volume to make use of both server sides. Multipath dm! Block device layer Page cache I/O scheduler SCSI driver zfcp driver qdio driver 17 Linux on System z - Disk I/O Alternatives chpid 5 chpid 6 chpid 7 chpid 8 Switch HBA 5 HBA 6 HBA 7 HBA 8 Server 0 Server 1! DA ranks extent pool

18 SCSI (dio) - 1sd vs. 1sps+mb Throughput for initial writers Throughput for readers 1sd 1sps+mb Throughput number of processes SCSI benefits from Storage pool striped disks too, especially with multipathing using multibus Sequential write is up to 2x better, sequential read up to 4.8x 18 Linux on System z - Disk I/O Alternatives

19 SCSI (dio) 1sd vs. 1sps+mb Throughput for random writers Throughput for random readers 1sd 1sps+mb Throughput number of processes SCSI benefits from Storage pool striped disks too, especially with multipathing using multibus Random write is up to 5.5x better, sequential read up to 4.1x 19 Linux on System z - Disk I/O Alternatives

20 FCP/SCSI LUN I/O to a linear or striped logical volume Application program VFS LVM dm Block device layer Page cache I/O scheduler SCSI driver zfcp driver qdio driver 20 Linux on System z - Disk I/O Alternatives! VFS sees one device (logical volume) The device mapper sees the logical volume and the physical volumes Additional processor cycles are spent to do map the I/Os to the physical volumes. Striped logical volumes require more additional processor cycles than linear logical volumes With a striped logical volume the I/Os can be well balanced over the entire storage server and overcome limitations from a single rank, a single device adapter or a single server To ensure high availability the logical volume should be used in combination with multipathing chpid 5 chpid 6 chpid 7 chpid 8 Switch HBA 5 HBA 6 HBA 7 HBA 8 Server 0 Server 1 DA ranks

21 FCP/SCSI LUN I/O to a storage pool striped volume with multipathing Application program VFS Storage pool striped volumes make no sense without high availability VFS sees one device The device mapper sees the multibus or failover alternatives to the same disk The storage pool striped volume spans over several ranks of one server and overcomes the limitations of a single rank and / or a single device adapter Storage pool striped volumes can also be used as physical disks for a logical volume to make use of both server sides. Multipath dm! Block device layer Page cache I/O scheduler SCSI driver zfcp driver qdio driver 21 Linux on System z - Disk I/O Alternatives chpid 5 chpid 6 chpid 7 chpid 8 Switch HBA 5 HBA 6 HBA 7 HBA 8 Server 0 Server 1! DA ranks extent pool

22 SCSI (page cache) 8sd+mb+lv vs. 2sps+mb+lv Throughput for initial writers Throughput for readers 8sd+mb+lv 2sps+mb+lv Throughput number of processes The scenario with the 2 storage pool striped disks performs comparable to 8 single disks 22 Linux on System z - Disk I/O Alternatives

23 SCSI (page cache) 8sd+mb+lv vs. 2sps+mb+lv Throughput for random writers Throughput for random readers 8sd+mb+lv 2sps+mb+lv Throughput number of processes The scenario with the 2 storage pool striped disks performs comparable to 8 single disks 23 Linux on System z - Disk I/O Alternatives

24 ECKD vs. SCSI (8KiB, direct I/O OLTP like workload) Throughput for random writers Throughput for random readers ECKD 8sd+hpav+lv ECKD 2sps+hpav+lv SCSI 8sd+mb+lv SCSI 2sps+mb+lv Throughput For the OLTP like workload, the SCSI setup performs better 24 Linux on System z - Disk I/O Alternatives

25 ECKD vs. SCSI (64KiB, page cache) Throughput for initial writers Throughput for readers Throughput 8sd+hpav+lv 2sps+hpav+lv 8sd+mb+lv 2sps+mb+lv number of processes Common sequential workloads ECKD can outperform SCSI when setup correctly 25 Linux on System z - Disk I/O Alternatives

26 Questions??? Some links the presentation referred to Linux on System z Tuning hints and tipps Live Virtual Classes for z/vm and Linux Mustafa Mesanovic Linux on System z Performance and Evaluation Schönaicher Strasse Böblingen, Germany mustafa@de.ibm.com 26 Linux on System z - Disk I/O Alternatives

27 BACKUP z/vm HyperPAV Linux on System z - Disk I/O Alternatives