Setting Up a High Performance Computing Cluster: A Case Study at the University of Craiova

Transcription

1 Setting Up a High Performance Computing Cluster: A Case Study at the University of Craiova CATALINA MANCAS DAN ANDREI, MIHAI MOCANU, DAN MANCAS Faculty of Automation, Computers and Electronics, blvd. Decebal, no. 107, , Craiova, Dolj, Romania Catalina.Mancas@ucv.ro, ado@ucv.ro, mmocanu@software.ucv.ro, dmancas@ucv.ro Abstract: - The term grid computing originated in the early 1990s as a metaphor for making computer power as easy to access as an electric power grid. The first definition of grid computing was that A computational grid is a hardware and software infrastructure that provides dependable, consistent, pervasive, and inexpensive access to high-end computational capabilities. Grid computing is today applied in a variety of application domains. Grid architectures are very much used in executing applications that require a large number of resources and the processing of a significant amount of data. This paper focuses on the HPC cluster installed at University of Craiova (CeGO) describing various aspects related to the hardware architecture, network infrastructure, basic system services, cluster middleware, and administration applications. Key-Words: - grid computing, high performance computing cluster 1 Introduction The term Grid computing (or simply Grid) refers to the way of combining multiple computing resources across a network in order to enable the execution of complex computations, by making them shareable across applications. Broadly speaking, the exploitation of several computers to process big workloads - Grid computing has revolutionized the parallel computing by both extending the life of existing systems and exploiting rapid advances in processing power, storage capacity, network bandwidth, as well as energy and space efficiency. [1] Apparently, the Grid computing advent was predicted ahead of time by Len Kleinrock in 1969 in [2]: We will probably see the spread of computer utilities, which, like present electric and telephone utilities, will service individual homes and offices across the country. Later, several definitions of the concept have been developed as Grid computing started to expand. In 1998, Carl Kesselman and Ian Foster state in the book The Grid: Blueprint for a New Computing Infrastructure [3] that: A computational grid is a hardware and software infrastructure that provides dependable, consistent, pervasive, and inexpensive access to high-end computational capabilities. In 2001, they refine this definition in order to address social and policy issues, together with Steve Tuecke in The Anatomy of the Grid [4]: Grid computing is concerned with coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations. Today, the literature abounds in attempts to provide meaningful definitions of Grid computing. The authors of [5] select ten definitions from literature and examine them so they produce a set of essential characteristics of Grid computing. Nowadays, the IT organizations lay on a rapidly, constantly changing and unpredictable context. Due to the challenges generated by such an environment there appears the urge need to adapt to rapid changes while keeping up the quality level of the delivered services. Grid computing proves to successfully comply with/conform to such requirements by meeting a series of characteristics beneficial for academic and business organizations. Thus, the adoption of grid solutions is accelerating and Grid computing and the technologies that enable it are being increasingly applied within large datacenters. Grids represent a consistent solution to running powerful applications and managing large amounts of data, as we shall illustrate through examples/stats in the end of this paper. The paper starts by introducing the Grid computing paradigm; the goal is to spot its main characteristics and highlight the benefits that may be achieved by applying Grid solutions. In Section 2 the focus switches towards the evolution of Grid computing identifying its key factors. Section 3 presents the architecture/ infrastructure of the Grid center set-up at University of Craiova, Romania. ISBN:

2 2 Grid Computing - key factors In the previous section we offer an overview on how Grid computing evolved in time and how definitions have been updated over the years according to the ascension of the Grid paradigm. In search for a comprehensive definition of Grid computing, Miguel Bote-Lorenzo, Yannis Dimitriadis and Eduardo Gomez-Sanchez argue in Grid Characteristics and Uses: a Grid Definition [5] that it is of a great importance for the definition of Grid to touch all its main characteristics. Not only it shows how Grid technologies can be used in various fields of application, but it also allows studying the potential benefits that the use of a Grid can report to application fields still unexplored by Grid researchers. [5] Anyhow, Grid computing is today applied in a variety of application domains. Grid architectures are very much used in executing applications that require a large number of resources and the processing of a significant amount of data. In this category lays the Particle Physics applications, Biomedical applications, Automobile and Aeronautical industries, Astrology and many others. But behind this various range of applications stands the ability of the Grid of solving complex problems, difficult to solve within a single body or company. Another key factor for Grid computing is the resource access and management. Multiple ways of combining resources results in increasing performance and reducing costs. The possibility of creating virtual organization and to dynamically reallocate resources according to the performance peaks are efficient cost cutting strategies enabled by Grids. Nowadays, Grid architectures turn out being the solution for many organization in need of managing an increased workload without adding more servers, supporting more users and managing more equipment with a smaller IT staff, while keeping the IT budget at minimum and cutting off capital and operational costs as much as possible. Hence, the Grid represents a good candidate for revolutionizing not only the supercomputing, but also the science and the engineering. 2.1 The Grid center at University of Craiova, Romania a good candidate for the European Grid Infrastructure (EGI) A Grid is a mechanism of enabling multiple computing and storage resource sharing and administration for geographically distributed organizations ensuring transparent and secure access. A set of characteristics of a consistent Grid system are presented in [5]. By simply connecting it to Internet a Grid becomes a means for sharing computer power and data storage capacity. Europe's largest production Grid infrastructure, namely Enabling Grids for E-Science (EGEE) gathers experts from more than 50 countries and aims at developing a service Grid infrastructure that is available to scientists 24 hours-a-day 1. EGEE project closed in April 2010, but it constitutes the foundation for a new organization European Grid Infrastructure (EGI). At its close, EGEE represented a world wide infrastructure of approximately to CPU cores, collaboratively hosted by more than 300 centers around the world. By the end of the project, around 13 million jobs were executed on the EGEE grid each month. The European Grid Infrastructure enables access to computing resources for European researchers from all fields of science, from High Energy Physics to Humanities. A good candidate for adhering to EGI is the Grid center recently set up at University of Craiova, Romania. It is currently integrated in RoGrid -NGI 2 (Romanian National Grid Initiative), which is part of EGI. The infrastructure provider of RoGrid is IBM 3 - industry leader in Cloud Computing. IBM installed the computing infrastructure consisting in BladeCenter 4 servers at six universities in Romania, linked together to provide RoGrid with an overall computing capacity of 25.7 trillion calculations per second. [6] As described by IBM, the BladeCenter is an ideal platform for server and network virtualization and consolidation because of its modular design. The IBM BladeCenter technology saves approximately 50% physical space and eliminates up to 92% of cabling and therefore it greatly increases server density versus rack servers. Moreover, it reduces costs by using a shared power and cooling infrastructure, it significantly contributes to the energy consume and nevertheless, BladeCenter simplifies the management of the datacenter. 1 More information about the EGEE project available online at: 2 More information about RoGrid-NGI available online at: More about IBM BladeCenters can be found online at: index.html?lnk=tab ISBN:

3 3 CeGO Architecture 3.1 Hardware Architecture The Oltenia Grid Center (CeGO) set-up at University of Craiova contains 528 Xeon cores, 408 GB RAM and approximately 40 TB of storage. The main hardware components of the HPC cluster are: - Firewall; - GRID gateway; - Two head nodes; - Disk storage; - 64 computing nodes; - Interconnection (network) infrastructure. The hardware architecture is presented in Fig Firewall The firewall ensures the necessary network security for the entire HPC cluster. It is based on the open-source Untangle Linux distribution. Untangle has been chosen because it contains an entire family of applications which help provide the required security level while still being easy to manage. Untangle comes with a common graphical user interface, logging and reporting. It basically plays the following roles: - web filter; - spam and spyware blocker; - virus blocker; - phishing blocker; - Attack blocker; - intrusion prevention; - protocol-level control of the traffic; - firewall; - OpenVPN access point; - routing and quality of service; - reporting Grid Gateway The GRID gateway provides the users and administrators the means to access the cluster Head Nodes The two head nodes are the place where users log in in order to access cluster resources (computation resources and storage). They are the only way for the normal user to access the cluster, being in fact an interface between the user and the HPC cluster Disk Storage The disk storage is accessible only through the two head nodes. The disk storage is connected to the head nodes via two redundant SAN (4Gbps) links which provide the necessary bandwidth and redundancy. It is based on NFS v3 and it provides a central repository for: - installation and update packages; - cluster-wide configuration files; - libraries; - executables which are not needed at boot time; - users homes; - general-purpose storage Computing nodes The computing power of the cluster relies on 64 blade-based Linux computers (nodes), having a total of 512 Xeon cores (8 cores/ node). To mark the impact of this multi-core solution architecture in cluster computing, we had in mind the further optimization issues required, such as: the multi-core awareness for cluster middleware, administration and user applications, or the optimization of intranode communication (just as important as the optimization of inter-node communication). We also took in account the scalability of a multi-core cluster, which, according to technical data available to us at the time of decision, could be improved by appropriate selections of user tasks and memory placement schemes, using the large page support available for multi-core processors [7] Network infrastructure As it can be seen in Fig. 1, the communication infrastructure of the HPC cluster comprises four virtual networks: - Login network; - Management network; - Cluster fabric network; - SAN. There are four VLANs defined: - one for the public access from Internet to the cluster; - one for the access environment (login network); - one for the management network; - a private VLAN for intra-cluster communication (for the cluster fabric network). All communication infrastructure is redundant for ensuring an acceptable fault tolerance level. The HPC cluster is connected to the Internet via two 10Gbps links. Inside the cluster there is a redundant core router having a bandwidth of 80 Gbps. Every node is ISBN:

4 connected to the router via two 1Gbps links (also for redundancy). There are also four uninterruptible power supplies (not shown in Fig. 1), which provide redundant power for all equipment. Fig. 1 ISBN:

5 3.2 Software Architecture The software architecture of CeGO HPC cluster is presented in Fig Operating system The operating system represents the first level of the software architecture of the cluster. RedHat Enterprise Linux 5.5 has been chosen because it is a stable and maintained distribution, which has proven to be stable and safe. All these services are part of the RedHat Enterprise Linux distribution KUSU basic cluster services Kusu is an open-source project which has as its objective to provide a simplified cluster deployment, management and operation framework. It supports a bunch of operating systems, including RedHat Enterprise Linux 5.5 used by us Cluster middleware This software level contains: - Service-Oriented Architecture Middleware (SOAM); - OpenFabrics Enterprise Distribution (OFED); - Platform Lava and LSF workload management; - Portals; - Compilers (INTEL: icc, ifc/ifort; PGI: pgcc, pgf77; GNU: gcc, g77, g95); - Message Passing Interface (OpenMPI). Fig Basic system services This software level consists of the basic services needed in order to operate the cluster: - DHCP (Dynamic Host Configuration Protocol); - NTP (Network Time Protocol); - HTTP - LDAP (Lightweight Directory Access Protocol); - MySQL (a database management system); - NFS (Network File System); - DNS (Domain Name System); - IPMI (Intelligent Platform Management Interface); - NIS (Network Information Service); - PFS (Parallel File System) Administration applications At this level we find applications for: - cluster management; - cluster monitoring; - cluster reporting; - workload management. There are several applications to manage the cluster, part of them being provided by Kusu: - addhost add and remove nodes from a node group; - boothost tool to create the PXE configuration files used when starting up a node; - cfm configuration file manager, a tool used to synchronize node group packages and files; - repoman repository manager; - nghosts a tool to move hosts between node groups; - ngedit node group editor; - etc. For monitoring purposes, we use two distributed monitoring applications: 4 Ganglia an open-source project which is used world-wide on a large scale for monitoring HPC clusters; 5 CeGO Mon a monitoring application with ticketing that we have designed User applications The primary user applications we started to develop in order to be run on our new HPC cluster concern the management of the digital content and cryptography. ISBN:

6 4 Conclusions The grid center installed at University of Craiova is one the largest high performance computing clusters in Romania. Due to its architecture it is appropriate for various types of applications, CPU intensive as well as memory intensive. The installed infrastructure will allow us to test various algorithms and apply different techniques for parallel computing, secure procedures for data and service access. References [1] Oracle, Oracle Grid Computing, An Oracle White Paper, pdf [2] University of California, UCLA to be first station in nationwide computer network, press.html [3] Ian Foster and Carl Kesselman, The Grid: blueprint for a new computing infrastructure, Morgan Kaufmann Publishers Inc., [4] Ian Foster, Carl Kesselman and Steve Tuecke, The Anatomy of the Grid: Enabling Scalable Virtual Organizations, International Journal Supercomputer Applications, 15(3), [5] Miguel L. Bote-Lorenzo, Yannis A. Dimitriadis and Eduardo Gomez-Sanchez, Grid Characteristics and Uses: a Grid Definition, Proceedings of the 1st European Across Grids Conference (CD), [6] IBM, Romanian National Computing Initiative Selects IBM, NewsBlaze.com, Article published online at: pnw/topstory.html [7] Lei Chai, Qi Gao, Dhabaleswar K. Panda, "Understanding the Impact of Multi-Core Architecture in Cluster Computing: A Case Study with Intel Dual-Core System" 7 th IEEE Int. Symposium on Cluster Computing and the Grid (CCGrid '07), 2007, pp ISBN: