Dimensioning storage and computing clusters for efficient high throughput computing

Transcription

1 Journal of Physics: Conference Series Dimensioning storage and computing clusters for efficient high throughput computing To cite this article: E Accion et al 2012 J. Phys.: Conf. Ser View the article online for updates and enhancements. Related content - The MAGIC data processing pipeline R Firpo Curcoll, M Delfino, C Neissner et al. - The Grid Enabled Mass Storage System (GEMSS): the Storage and Data management system used at the INFN Tier1 at CNAF. Pier Paolo Ricci, Daniele Bonacorsi, Alessandro Cavalli et al. - An alternative model to distribute VO software to WLCG sites based on CernVM-FS: a prototype at PIC Tier1 E Lanciotti, G Merino, A Bria et al. This content was downloaded from IP address on 05/11/2018 at 10:29

2 Dimensioning storage and computing clusters for efficient high throughput computing E. Accion 1,3,A. Bria 1,2,G. Bernabeu 1,3,M. Caubet 1,3,M. Delfino 1,4,X. Espinal 1,2,G. Merino 1,3,F. Lopez 1,3,F. Martinez 1,3,E. Planas 1,2 1 Port d Informació Científica (PIC), Universitat Autònoma de Barcelona, Edifici D, ES Bellaterra (Barcelona), Spain 2 Also at Institut de Física d Altes Energies (IFAE), Universitat Autònoma de Barcelona, Edifici Cn, ES Bellaterra (Barcelona), Spain 3 Also at Centro de Investigaciones Energéticas Medioambientales y Tecnológicas (CIEMAT), Madrid, Spain 4 Also at Universitat Autònoma de Barcelona, Department of Physics, ES Bellaterra (Barcelona), Spain Abstract. Scientific experiments are producing huge amounts of data, and the size of their datasets and total volume of data continues increasing. These data are then processed by researchers belonging to large scientific collaborations, with the Large Hadron Collider being a good example. The focal point of scientific data centers has shifted from efficiently coping with PetaByte scale storage to deliver quality data processing throughput. The dimensioning of the internal components in High Throughput Computing (HTC) data centers is of crucial importance to cope with all the activities demanded by the experiments, both the online (data acceptance) and the offline (data processing, simulation and user analysis). This requires a precise setup involving disk and tape storage services, a computing cluster and the internal networking to prevent bottlenecks, overloads and undesired slowness that lead to losses cpu cycles and batch jobs failures. In this paper we point out relevant features for running a successful data storage and processing service in an intensive HTC environment. 1. Introduction Scientific experiments are experiencing an explosion of digital data production, both in quantity and size. Detectors and electronic devices in general are continuously increasing their intrinsic resolutions delivering huge amounts of digital data. The Large hadron Collider (LHC) at CERN envisaged this and started the WLCG[1] (World-wide LHC Computing Grid) project, a computing infrastructure involving more than 140 sites around the world and having a potential of six thousand users. The Port d Informació Científica (PIC) is one of the eleven first level centers (known as Tier1 centers) and supports three out of the four experiments in the LHC: ATLAS, CMS and LHCb. PIC is also providing computing services for research groups in astrophysics, cosmology, neuroimaging and genomics, but their requirements are usually smaller than the LHC experiment ones. The LHC delivers 15 PB of data every year, that is eventually stored, analyzed, archived and reprocessed. This means the same data is not only processed once, but several times, using disk storage as online data and a different concept of long term storage: nearline data. The Published under licence by IOP Publishing Ltd 1

3 interaction among the processing nodes, and the storage (online or nearline) has to be correctly synchronized to minimize CPU cycles lost due to I/O waits and to maximize the efficiency, this means that online data has to flow quickly and reliably and nearline data has to be intelligently pre-staged on buffer disk areas in front of the tape robots before the jobs start to run. One of the keys to reach the required performance is the correct dimensioning of the network among all the parties: processing nodes, disk servers and tape servers. PIC is running a computing farm of 4000 cores, and the experiments estimate on the jobs average data I/O rate is of about 5MB/s. This would translate in a constant data flux of 20GB/s between the nodes and the disk servers for a 100% occupied farm. Measurements show a yearly mean usage of 2GB/s, which translates into approximately 60 PB of internal data exchange per year. The acceptance and replication of data among the WLCG computing centers proceeds simultaneously to the internal data processing flows. The incoming data is steered automatically into nearline data or online data depending on its nature. The rate of exported and imported data in the WAN averaged at around 2 Gbps during last year, having peak values where all the available bandwidth (12 Gbps) was saturated for short periods of time. To illustrate this, Fig. 1 shows the WAN global traffic in and out from PIC during 5 weeks in Figure 1. WAN traffic. Data reception and exportation during one month in The storage setup at PIC is not implemented using commercial solutions but innovative disk system managers. Disk storage is managed by dcache [2] and tape storage is managed by Enstore [5] (developed at FNAL, Chicago). The details are shown in section 2 The requirements to achieve a high throughput performance in a heterogeneous hardware environment need a proper study and a fine tuning for most of the nodes, such that an homogeneous network performance per TB can be achieved. This is discussed in section 3. Interaction between disk and tape should be transparent from the user and experiment point of view. The tape storage is not the classical backup mainframe system but an automated robotic library used as a high latency disk, the details on the implementation of the tape system mechanism and its interaction with the disk storage are covered in section 4. 2

4 2. Handling online and nearline storage Two different kinds of storage are in place: disk and tape. Disk is normally used for short and mid-term storage, or storage that is planned to be accessed frequently. Tape is used for long-term storage which is planned to be accessed in a controlled manner and not too frequently. dcache handles the disk storage and Enstore handles the tape storage. dcache is a software system for storing and retrieving huge amounts of data, distributed among a large number of heterogeneous disk nodes and presented under a single name space with different access protocols. It features an interface for a tertiary storage system, space management, pool attraction, dataset replication, hot spot determination and recovery from disk or node failures. The dcache installation at PIC is roughly separated in three components: pools, head nodes and doors. Pools are servers which provide the raw disk storage capacity, normally composed of disks with some degree of redundancy. The redundancy at the level of disk server is achieved with hardware and software RAID technologies - via hardware RAID controllers in some cases, or with software RAID systems like ZFS or mdraid. The configuration of the pools is as homogeneous as possible, though still heterogeneous to cater for the different hardware deployed and different usage needs. Doors are software components that are responsible for translating the requests in any protocol to internal dcache brokers. Currently PIC has doors in production which are specific for gridftp, dcap, http/webdav and xrootd protocols. The dcache nodes that run the auxiliary services are known as head nodes. They are responsible for brokering requests across the pools, publishing information and other administrative tasks. In the case of PIC, servers either run as doors, pool nodes or head nodes. Other levels of redundancy beyond the one given by RAID can be achieved through the feature of dcache supporting copies of the same file on different disk pools. That allows to achieve redundancy of files, to keep critical files available even in the case of one disk server failure, and also fosters throughput as files accessed from many different clients can be replicated into different disk servers and thus requests load can be balanced. To do this, dcache uses a thermodynamic approach, allowing files to be dynamically copied to different disk servers in case of detecting that a given file is hot (accessed by many clients), and removing the copies once that space is needed for another purpose. One of the critical aspects of dcache operation is its ability of handling disk server failures smoothly; a critical failure in a pool very rarely has effects on the overall system, beyond losing availability of those files which were not duplicated. Even if dcache is not able to manage tape directly, it has features to interact with a tertiary storage system, normally implemented by tape. When a given set of conditions is met, it launches a script that triggers the migration/recall from the tertiary storage system. Enstore was chosen for this purpose. Enstore is a software system that provides distributed access to the data stored on tape and its management. It provides a generic interface so the user can access data in a similar way as they access native file systems. It features tape and robot management, scheduling the requests and handling quotas. Enstore uses a single name space that can be shared with other storage systems to provide an interface for users. Enstore installation at PIC can be roughly separated in tape servers, head servers and client machines. Tape servers are machines with HBA (Host Bus Adapter) controllers directly attached to tape drives: LTO3, LTO4, LTO5 and T10KC are currently in production. The current setup at PIC uses 8 servers, each of them with 2 HBA controllers with 2 ports each, hence connected 3

5 to 4 tape drives. Head servers run the brain of the system, providing the brokering among the clients and the tape servers, running services responsible for keeping quotas, grouping tape drives in different libraries and managing queues. The scheduling of the queue is done in a non-fifo way to allow for optimizations by trying to keep a given tape mounted (to alleviate mount/dismount latency penalties). PIC s HSM solution is therefore the integration of both dcache and Enstore systems. dcache handles disk servers and uses the tertiary storage interface to call a script that handles the translations needed from the dcache file to Enstore, which ends up calling a client command: encp, the main Enstore interface that copies data from/to disk to/from tape. In this way, dcache can migrate files to tape or recall them from tape per request of the user. The script runs on the pool node, and by using the encp command it generates an Enstore request handled by the Enstore servers. When a suitable tape drive is available, the tape is mounted there, and the file is staged into the pool node and from there it is served to the worker node. All meta-information of the file is stored in a shared name space system named PNFS [3]. A new version of the namespace called Chimera [4] is available in dcache and PIC plans to migrate PNFS to it before the end of This allows Enstore and dcache to collaborate without having a big dependence on each other. 3. Disk storage There is a large number of solutions for disk storage on the market. They can be categorized in two different branches: big, monolithic solutions, where one or multiple controllers manage a big set of disks in an opaque manner, normally achieving high resilience via RAID hardware controllers cheap, highly decoupled disks that are presented independently. These are usually referred to as JBOD systems (for Just a Bunch Of Disks). On those, the resilience is provided at a higher software level. This can be achieved, for instance, with multiple independent copies on disk instead of relying on each copy on disk to be always there. The implementation chosen, with dcache, is in between these two extremes, showing characteristics of both systems. For example, a double level of resilience is implemented by providing redundancy at the level of device (ZFS, RAID6) and at the application level. dcache handles widely accessed files triggering multiple and independent copies in case one of them becomes inaccessible or the pool is under high load. This is done because the first kind of redundancy allows to have a very low operational cost - disk failures have very low interference with the rest of the system. The second kind of redundancy is also used to be able to achieve faster read speeds for situations when a file is hot (being accessed by many clients) by allowing clients to read from different sources rather than only one. Three different approaches to hardware solutions are in place: DAS with s/w RAID: this first type of disk server hardware is represented by SunFire x4500 servers. Those are basically servers with 48 hard drives which are independently presented to the OS. Volume manager and redundancy capabilities normally offered by controllers are implemented with Solaris ZFS. Basically, the main CPU of the machine is the one working as RAID controller, as everything is done at OS level. That means tuning is also done at OS level, which is convenient as it is centralized. Data is served through the network with 4x1GE interface aggregated through LACP (Link Aggregation Control Protocol). 4

6 SAN: A second solution for hardware was a Data Direct Networks (DDN) S2A9900. This is a Storage Area Network (SAN) type system: 600 disks per system are served through a pair of controllers providing high availability for the system. Controller manages disks setting RAID6-like systems of 8+2 disks, that then are served through fiber channel using SCSI protocols to a set of blade servers, all of them running dcache. In this case the tuning is done at two levels: at the level of the controller and at the level of the fiber channel communication, controlled by the OS of the blade servers. Due to legacy reasons, the chosen operating system is Solaris 10 with ZFS just adding stripping. Tests have been done with Linux together with XFS and grouping disks with LVM (Logical Volume Manager) to get similar environments with good performance results. DAS with h/w RAID: These are SGI or Supermicro servers providing high density of disks (36 disks in a 4u server) presented as a single virtual device through an internal controller, with redundancy. Every server is currently organized in three different devices each one composed by 10+2 disks, arranged in RAID 6. Multiple performance tests has been performed to decide the exact configuration of the disks. A RAID 60: grouping 12 disks as one RAID 6, and then aggregating 3 of those groups as one block device. There was mainly one problem related to the controller: the scalability with multiple streams suffered high degradation because of saturation of the CPU of the raid controller. Three different RAID 6 and grouping them using LVM: An improvement has been observed in throughput basically because a bigger block size (4MB) could be setup. It was observed that the multiple streams writes were relatively slow (order of 400 MB/s) but the reads were fast (1 GB/s) in this setup. This is acceptable as it is relatively easy to parallelize writes into different disk servers to achieve the desired throughput, but it is not as easy to parallelize reads as they require a copy from a pool to another pool that can generate more problems than it solves. Incoming and outgoing traffic to WAN shows roughly a 1:1 read:write ratio, while LAN traffic shows a clean tendency towards reading, with a 2:1 read:write ratio. It can also be observed that traffic to/from tape is not high, but scalability to absorb eventual high throughput should be guaranteed, as tape drives are the most scarce resource. To ensure this scalability, the dcache system is configured such that all of the pools are eligible to be used for all of the purposes: WAN transfers, LAN transfers and tape recall or migrations. This configuration choice has proven to deliver optimum performance, since it potentially gives access to the maximum number of available spindles at any given time for any requested action. It is also a configuration that enables efficient use of resources, since for instance all of the available disk space at a given time can be used as cache for files on tape. The pool costs feature in dcache allows to select algorithms to balance the loads, so the system can adapt to the different kinds of loads each server has. Pool costs are basically a way to assign dynamically weights to pools. The load of the server can be taken into account (number of concurrent transfers to the disk server), and other variables like available space to calculate the weight. Then, for write requests, weight is used to select the less loaded server to write the data in. At the same time that the disk system is deployed and tuned for optimal performance, it is important to monitor which is the usage that the LHC experiments are making of the service, and their access patterns. Fig. 2 shows the fraction of the data stored on disk which was actually uniquely read every month. This data has been obtained from the dcache Billing DB, where an exhaustive accounting of every data transferred in or out of the disk system is recorded. The 5

7 results show that only around 20-30% of the data stored on disk is read every month for the large experiments. This seems to indicate that the system is running far from its full capacity, hence it suggests that there is room for improvement in the overall efficiency delivering the large disk storage service for the Tier1. These are preliminary results which now motivate triggering a deeper analysis of such access patterns, which is considered as future work. Figure 2. Percentage of the data stored on disk which is accessed by each experiment every month at PIC Tier1. 4. Tape storage Tape storage is managed by Enstore. The setup can be categorized in three components: head servers, tape servers and tape libraries Head servers There are four head servers deployed in each of the two instances (production and test), both running in separated environments. Configuration server: contains the Enstore central configuration. It is responsible of maintaining and distributing all the information about the system configuration across all the components of the tape system. This server also contains a centralized log file service for all the components. A web server also runs in the configuration server, acting as an interface to the tape system monitoring: accounting, rates, system status, etc. It also runs a real time application showing the state of the system, drive rates, drive buffer occupation, etc. Library manager (LM): runs two different groups of processes, the first group is formed by the virtual libraries defined by the combination of a physical library and media type. These processes are the responsible of managing queues of requests. The second group of processes running in the LM are the media changers that need to be defined for each physical library. These processes are responsible to launch the actual commands that operate the robot, mounting and dismounting tapes from drives. Backup: running backups of the data and providing storage space needed for migrations. Database: running Postgresql databases needed by the other services. It used to be one instance of Postgresql serving 3 different databases: file and volume catalog, drive status 6

8 Tape Tape Library Technology TS3500 SL8500 LTO3 drives 8 LTO3 tapes 1490 LTO4 drives 4 16 LTO4 tapes LTO5 drives 4 LTO5 tapes b drives b tapes 2494 Table 1. Number of tapes and tape drives per library and accounting, but it has been recently split into three different instances of Postgresql serving one of the DB each, to avoid interferences among the databases. A lot of the information regarding files is not exclusively stored in this database, but it is also exported to the PNFS, as a way to share information with dcache. Every tape file has an internal and an external ID. PNFS ID is used by dcache and BFID is meaningful for Enstore so the storage system can handle both disk and tape requests Tape servers Tape servers are currently Dell R710 with two Fiber Channel HBA controllers having two 4Gbps fiber-channel ports each, thus being able to connect to 4 different tape drives. Each tape server controls drives of different tape technologies, to avoid the failure of few tape servers to affect completely one technology. As network requirements with tape drives on modern technologies are around 150 MB/s, 10GE connections are needed for each tape server in order to prevent bottlenecks. Aggregation of 1GE links is not optimal, as individual streams can be more than 120 MB/s. With the current implemented networking technology it is not possible to efficiently distribute one stream among different network interfaces. Tape servers need well dimensioned memory to scale with the numbers of tape drives they are controlling. Production tape servers have 32GB of RAM, split into 12GB for the system and 5GB per tape drive controlling process. Each tape drive is managed by a single python process called a mover. Our current configuration of the tape servers involves four of these processes running concurrently, each of them responsible for handling a different tape drive. For load balancing purposes and high availability, each tape server runs different drive media (LTO3 LTO4, LTO5 and T10KC) Tape libraries The system handles two different tape libraries: IBM TS3500 and Oracle/STK SL8500, in Table 4.3 a breakdown is shown of the different tape libraries and technologies in use. There are tunable key features provided by Enstore for optimizing performance. Some of the problems already addressed are enumerated below. Data distribution on tapes: Data placement is relevant to exploit locality. Given the physical restriction that only one tape can be mounted in a given tape drive at a given time, if you place all relevant data in a single tape (not so strange considering sizes of tapes nowadays get up to 5TB) then retrieving that data must be done sequentially and only with one 7

9 tape drive. This can lead to suboptimal performance, specially when there are not enough concurrent requests to make use of a high number of tapes, as the tape drive utilization will drop substantially. To solve this, a parameter (file family width) can be configured that indicates the maximum number of streams a given set of data will use to migrate to tape. Thus, if 5TB of data are migrated to tape, and specify file family width=5, then five tapes will be mounted, and five streams written, storing 1TB of data on each tape. One wants to limit that number because using the maximum number of available tape drives, one could easily end up running into starvation problems (a single migration process monopolizing the resources). The number is also tuned having into account the resources assigned to each project. One should also be careful that a higher number of streams can lead to unwanted excess of fragmentation of data, having data scattered among too many different tapes. Saturation of disks servers on disk-to-tape copies: It can happen that only one of the disk servers gets most of the data that is going to be recalled/migrated from/to tape. One could run into contention of the disk, stalling tapes. This is clearly unwanted as tape drives are the slowest and thus one wants to maximize utilization making tape drives the bottleneck. To control this, Enstore discipline feature is used to limit the number of concurrent accesses to tape drives from a given disk server. In this way, disk servers throughput capability is throttled to prevent them using the tape drives in a sub-optimal manner. Avoid tape dismounting: Enstore s library manager has optimizations in order to avoid paying the dismount penalty. It implements a HAVE BOUND state that is configured to last for about two minutes when a tape is not requested anymore. That means that after a file has been served from a tape, if a new request appears before the 2 minutes timeout, the tape will still be there saving the dismount/mount time. Enstore handles a queue of requests, which is auto-shuffled to group requests accessing to the same tape, hence allowing to save some extra time in mount/dismount operations. Also, at the level of dcache, the minimum amount of data needed to trigger a migration can be tuned, thus preventing a constant leakage of small quantities of data that would force a tape to be mounted/dismounted all the time. 5. Network Networking is the layer that holds everything together. Careful dimensioning of the network flows is important to avoid making the network the bottleneck. The topology of choice is starlike, to be able to follow a simple structured cabling, where the central component is a Cisco 6509-E. There is an ongoing migration of this component to be moved into a Cisco Nexus 7009, providing the required scalability in terms of 10GE ports. Data flows with significant bandwidth requirements are basically three: WAN connections, worker nodes reading/writing from/to the disk system and those reading/writing from/to tape servers. Roughly, the bandwidth is dominated by the worker nodes interacting with the storage system, and it is estimated to be close to the 5-10 MB/s per job. Due to the Cisco 6509-E having a 40Gb bandwidth limitation between modules, it is not able to cope with the requirements. Getting a new switch with more bandwidth was a too expensive option at that time, and given the fact that most of the traffic comes from the interaction of worker nodes and the storage system, it was decided to buy a new set of two switches (Arista 7148SX) that operate as one virtual switch with 92 effective 10Gbps ports, wire-speed and with low latency in the nanosecond range. All L3 traffic among worker nodes, storage subsystem and tape servers was handled at the level of this switch, off-loading the most intensive network bandwidth from the main switch/router. Intensive data movement makes most of the Ethernet frames to be the size of the MTU (Maximum Transmission Unit). The usual default value of the MTU is 1500 bytes and this can have an impact on the CPU of the intervening parts, and can significantly drop the performance 8

10 of transfers. Thus, the possibility of using Jumbo Frames was investigated and adopted after seeing improvements in both CPU utilization and data throughput when using MTU value of 9000 bytes. From the worker nodes side the improvement in the combined throughput was 30% and the improvement on the disk servers throughput was 40% (DDN case). It has also been observed that default kernel values found in most common kernels do not match HTC environment requirements. As an example, TCP max buffer size, a kernel parameter that marks the size of the buffer to be used as the TCP window, tends to default to tiny values (in the order of 16KB), that do not provide enough buffer to be able to sustain high throughput transfers over WAN. As seen in ESnet web [6], recommended value is 16MB but a lower value of 8MB was chosen due to the high number of connections needed to the disk servers. This is because of memory restrictions on disk server machines. Another interesting value to tune is the max backlog. This number is the maximum number of packets that can be left unprocessed before the kernel starts discarding new ones. By default, this value is 1000 packets, which is too low for HTC environments; the recommendation found in ESnet web [6] is to set this value close to packets. Similarly the transmit queue length, defined as the transmit queue length of the device, should be increased to packets as recommended for HTC environment. An HTC environment handles large quantity of data, and that can lead to congestion. The algorithm used up to now in the case of congestion was BIC (Binary Increase Congestion) which is used by default in Linux kernels through There is an ongoing evaluation of CUBIC [7], an alternative algorithm available in Linux kernels and above that is an enhanced version of BIC: it simplifies the BIC window controls and improves its TCP-friendliness and RTT-fairness. Overall, it should be emphasized the fact that, for correctly dimensioning the network in an HTC environment like PIC, one needs to have into account all the available tuning at different levels, and the analysis of the network flows by throughput, to be able to optimize the economical component of the solution. 6. Conclusions An HTC service in production has been described. It has been pointed out that growing in capacity is not a brute force game of adding disks but a subtle strategy for pursuing right scaling process involving many layers in the data center. Public centers usually have a mix of hardware due in part to the public tender procurement procedures, this adds a level of complexity and performing a certain level of R&D is needed when new hardware is being deployed. For this reason is of great importance to have the storage management layers as decoupled as possible from the low level settings peculiarities, such as for instance the OS. This allows to wrap everything up in a single framework and co-operate transparently at the application layer. Also it has been shown that, once the disk management application is defined, the interconnection with the tape system has to be as simple as possible, considering the tape Libraries as high latency disk. As discussed in section 5 the main characteristic of an HTC environment is an intensive use of data on disk and tape. Successful processing can only be achieved if the network among tape system, disk system and the worker nodes is optimized and correctly dimensioned to be capable to handle I/O bursts. For instance, at the time a heavy data processing campaign starts, usually a full usage of the batch system is required with a huge number of jobs starting almost at the same time. For this workload profile, the handling of hot files and an intelligent way of replicating them as they are issued, it is crucial to prevent inefficiencies or an eventual unintended 9

11 Denial of Service, as was pointed out in section 2. Science is starting a high resolution phase in the current digital era, the amount of data collected is growing exponentially and the size of data is also in constant expansion. For this reason data centers providing support to scientific experiments with huge demands in computing, naturally tend to an HTC environment with the best possible performance. This results in a data center that runs many tasks where all the performance metrics matters (CPU usage, jobs/h, MB/s processed, I/O rates). 7. Acknowledgments The Port d Informació Científica (PIC) is maintained through a collaboration between the Generalitat de Catalunya, CIEMAT, IFAE and the Universitat Autònoma de Barcelona. This work was supported in part by grant FPA C02-01/02 and FPA C02-01/02 from the Ministerio de Educación y Ciencia, Spain. We would like to specially thank the dcache team in DESY, FNAL and NDGF, and the Enstore team in FNAL for their hard work, support and co-operation. 8. References [1] WLCG computing TDR available from [2] dcache project web page: [3] dcache PNFS: [4] dcache Chimera: [5] Fermilab - Enstore group web-page: [6] A high-speed network serving thousands of Department of Energy scientists and collaborators worldwide: [7] Injong Rhee, and Lisong Xu CUBIC: A New TCP-Friendly High-Speed TCP Variant In Proceedings of the third PFLDNet Workshop (France, February 2005) 10