Three Generations of Linux Dr. Alexander Dunaevskiy

Transcription

1 Three Generations of Linux Server Systems as TSM-Platform at Dr. Alexander Dunaevskiy d

2 LRZ in Words LRZ is the biggest supercomputing centre in Germany LRZ provides a wide range of services for German und Bavarian universities and research institutes. Amongst others, they are compute, archive and backup, online data storage and mail services. The staff contains nearly 170 persons. LRZ is located in Scientific Research Area in Garching near Munich since LRZ belongs to Bavarian Academy of Sciences and Humanities

3 LRZ Numerically I Compute Server of LRZ: SGI Altix x512 System 9728 Cores, 39 TB RAM, 60 TB NAS for /home and 600 TB SAN for scratch directories 62.3 TFlop/s peak 56.5 TFlop/s LINPACK performance, No. 10 in the current Supercomputer Top 500 list. Different parts of LRZ Linux Clusters with peak performance between 0.4 and 1.6 TFlop/s and sum of peaks over 4.5 Tflop/s

4 LRZ Numerically II Archive and Backup System with potential to store up to 10 PB of data. Mail Platform for more than 30,000 Users (Students and Researchers). Munich Scientific Network: Munich and surroundings High Performance Network up to 10GE m² air-conditioned d rooms for hardware, 2MW electricity facilities, 2MW air-conditioners, Argon Gas fire-extinguishing extinguishing system.

5 LRZ in Pictures I

6 LRZ in Pictures II Photo Helmut Payer, produced by gsicom

7 LRZ in Pictures III

8 Basic Data of TSM under Linux I Linux is supported as server platform for TSM Server from version SuSE: Professional 8.0, SLES7, RedHat: Red Hat 7.2, RHEL 2.1 A thread Error in TSM delayed the beginning of production => Call by Tivoli Disaster Recovery for Linux was implemented first in TSM

9 Basic Data of TSM under Linux II

10 Basic Data of TSM under Linux III

11 TSM under AIX on LRZ

12 1.Generation of TSM Server based on Linux at LRZ

13 The first tests (since March 2003) Compared with RedHat, SuSE SE performs better. The crucial factor : fewer problems with device driver and fewer kernel problems with dsmserv. Compute Services group of LRZ prefers SuSE and has many more systems with SuSE than with RedHat Linux.

14 3584-L32 (1. Frame)

15 First Tests and First Problems Support matrix of TSM and support matrix of 3584 with LTO2 have an empty intersection. Solution: IBM was cooperatively and has made available an extra kernel patch (2418S ( SMP) for LRZ.

16 First Tests and First Problems SAN Failover for Linux: HBAs from Qlogic and from Emulex, sdd for ESS SHARK, Qlogic qla-failover driver. Ethernet redundancy (bonding). switches did not work correctly...

17 TSM for Linux and for AIX The differences between TSM for Linux and for AIX are marginal. Examples: Disaster Recovery in the Linux version of TSMfirstin5200 in In Linux, ps grep dsm shows many more processes than in AIX. AIX displays data volume in Kbytes, Linux in bytes. Raw devices are not supported by Linux.

18 Comparison between TSM for AIX and for Linux AIX: High price for hardware, especially for fiber channels and network, very high price for service, very good manageability of online operations. Linux: Moderate price for hardware, attractive price for service, continuous updates, battle against the specified and strongly overpatched kernel. In the recent time many things have turned for the better: 64 Bit, Kernel 2.6, LVM A few things could be improved : persistent binding, scsi device manageability.

19 Error Classification Error Classification for a TSM System OS Induced TSM as Software Induced Hardware Induced To many Threads under Linux Almost no differences under different OS TSM Log Overflow Tape Problems Slightly differences under different OS Strong differences under different OS TSM Storage pools are full Big Number of Sessions in "Media Wait" State Tape-Library (Gripper)- Problems SAN-Problems HD Reconstruction under strong demand.

20 Table Summary Generation 1 Generation 2 Generation 3 Generation 3+ Server Adv Unibyte Oxygen 2200 IBM X346 Sun SunFire4200 Sun SunFire4200m2 CPU 2xXeon 2,8GHz 2xXeon (EMT 64) 3,0 GHz 2xOpteron 2,6 GHz 2xOpteron 2,8 GHz Dual Core Ram 2-8GB 4GB 6GB 16GB HBA 3xQlogic xIBM - FC133 (Qlogic 2312) 2xQlogic 2342(Dual Port) 2xQlogic 2462(Dual Port PCI-E) OS 1 SuSE Prof. 8.0 SLES8 SLES9x64 SLES9x64 Kernel SMP (IBM) smp smp smp OS2 SLES8 SLES8 SLES9x64 SLES9x64 Kernel SMP smp smp smp Storage Server STK D280 3xIBM FastT 900 2xSTK Flex 380 2xSTK Flex 380 HDD FC 73GB FC 146GB + SATA 250GB FC 146GB + FC 300GB FC 146GB + FC 300GB HDD Size 3TB 22TB FC + 63TB SATA 80TB 80TB Tape Library IBM 3584 (1-7Frame) 2xIBM 3584 (je 12 Frame) STK SL85100 STK SL85100 Tape Drives 8xLTO2 (IBM Ultrium 3580) 40xLTO2 (IBM Ultrium 3580) 16xTi10000A (STK) 16xTi10000A (STK) Failover Qlogic RDAC Qlogic Qlogic STG&DB LVM LVM w/o partitionstable LVM2 w/o partitionstable LVM2 w/o partitionstable

21 Complexity of Linux Handling as TSM Base Intersection of Hardware and Software support matrix. Updates within an OS release can be as complex as a new installation. Updates of an OS release are very complex. No authentic certification for hardware under Linux, among other things for a server.

22 Consequences Long and numerous tests are required before you can update or buy a system! Security could be increased by smaller kernels or you need external solutions.

23 LABS

24 HABS r200 HABS FC-Network Overview sunstk01 datsrv31 datsrv32 datsrv33 datsrv34 datsrv35 datsrv36 datsrv37 datsrv38 datsrv46 datsrv45 datsrv44 datsrv43 datsrv42 datsrv41 datsrv40 datsrv39 datsrv38 datsrv48 datsrv47 fcdir1 fcdir2 One FC Cable (2 Gbit) Two FC Cables (2 Gbit) Two FC Cables (4 Gbit) Four FC Cables (4 Gbit) NAS3050a NASGX3 NASGX4 NAS3050b Drive0 Drive1 Drive2 Drive3 Drive4 Drive5 Drive6 Drive7 flx1 flx2 NAS3050c NASGX1 Drive8 NASGX2 Drive9 Drive10 Drive11 Di Drive12 Drive13 Drive14 Drive15 Version 2,

25 HLRB2 High Performance Archive

26 High Performance Archive Storage Pools 7xFC hard disks configured as RAID5 with Store Size 512KB LVM stripe over 4 hard disks and over 2 HBA ports with XFS FS Read ahead set to 8192KB Kernel I/O Scheduler = deadline

27 High Performance Archive System Benchmark Results Benchmark Iperf (raw network I/O) TSM Archive to Disk (single file) TSM Backup to Disk (4 parallel files) TSM Archive to Tape (single file) TSM Archive to Tape (4 parallel files/tape drives) Results* 5.2 Gbit/s 300 MB/s 480 MB/s 120 MB/s 400 MB/s *Results for one server

28 NDMP Backup on LRZ Mail nas3050a nas3050b File-Services Volume Snapmirror nas3050c Linux Cluster NDMP nas270a/b R200 HABS Test-System Replication Tape-Backup p

29 About NDMP Backup on Linux Control commands are sent by LAN, the data by SAN. Building of TOC needs a lot of time for the volumes with a large number of files. Two changeable types of SCSI reservation on Netapp Filer (SCSI2 or SCSI3). TSM implements only parts of SCSI3 protocol. Problems with labels (ANR8355E): the faster the drives the more often the problem. Workaround: enable the traces on the filer and on the TSM library manager or on the TSM server with defined library and drives.

30 Conclusions Experiences with Linux as a base for TSM are predominantly positive Differences between AIX and Linux as TSM base are marginal The error behaviour is similar