Practical Methods to Begin Parallel Processing Education and Research

Transcription

1 Practical Methods to Begin Parallel Processing Education and Research Yili TSENG Department of Electronics, Computer, and Information Technology North Carolina A & T State University Greensboro, NC 27411, U.S.A. ABSTRACT In the past, parallel processing was not widely adopted because of the complexity and lengthy duration of parallel software development and the amazing advancement of processing speed of uniprocessors. However, the design of processors has hit the physical limitation and parallel processing is the only way to build more powerful computer systems with current technology. That is why the major processor manufacturers introduce multicore processors. But little did people know that the extra cores of a multicore processor cannot be utilized unless programs specifically written for multicore processors are executed. Therefore, the need for parallel processing education is imperative. Most educators and researchers are hindered from initiating parallel processing courses and/or researches by a misconception: affordability of costly parallel computer systems. With the introduction of affordable personal computers equipped with multicore processors, programming languages supporting thread can be used to write small-scale parallel programs taking advantages of extra cores. Parallelism can be explored with the multithreaded programs. As for larger scale parallel programs, they can be carried out by an affordable cluster. Thanks to the contribution of open-source software developers, major parallel software libraries have been successfully ported to inexpensive personal computer platform. Along with other free open-source software for PC, they can make PCs networked by low-cost switches a cluster, an affordable platform for parallel processing education and research. Keywords: Parallel Processing, Computing Education, Cluster, Open-source Software, Parallel Programming, Multicore Programming 1. INTRODUCTION In the past, parallel processing was not widely adopted because of the complexity and lengthy duration of parallel software development and the amazing advancement of processing speed of uniprocessors. However, clock speed of contemporary uniprocessors can hardly be further pushed because of speed-og-light argument. [18][1] That is why the major processor manufacturers like Intel and AMD introduce multicores processors. But little did people know that the extra cores of a multi-core processor cannot be utilized unless programs specifically written for multi-core processors are executed. In other word, parallel software is mandatory to take advantage of the multicore processors. In short, the age of parallel computers has arrived and parallel processing is the only way to build more powerful computer systems with current technology. Therefore, the need for parallel processing education is imperative. Most educators and researchers are hindered from initiating parallel processing courses and/or researches by a misconception: affordability of costly parallel computer systems. Formerly only major institutions can afford expensive parallel machines. With the introduction of affordable personal computers (PC) equipped with multicore processors, programming languages supporting thread can be used to write smallscale parallel programs taking advantages of extra cores. Parallelism can be explored with the multithreaded programs. The first part of this paper introduces the more popular programming languages supporting multithread or multicore programming. As for larger scale parallel programs, they can be carried out by an affordable cluster. Thanks to the contribution of open-source software developers, major parallel software libraries have been successfully ported to inexpensive personal computer platform. Along with other free open-source software for PC, they can make PCs networked by low-cost switches a cluster, an affordable platform for parallel processing education and research. With clusters built with the retired PCs and free software, any institution and individual can own its tools with minimal cost and start parallel processing education and/or research. Although an affordable cluster can be built with the free opensource software and inexpensive PCs, it cannot work correctly without some vital configurations. Unfortunately, the free software does not provide supports, necessary background knowledge, or correlated information for constructing a cluster. The second part of this paper presents the practical experience to build an affordable cluster. Both approaches together provide low-cost solutions for all

2 institutions to effectively offer their parallel processing courses. Both approaches, namely multithread programming and affordable clusters, provide practical and effective way to initiate parallel processing education and research. Microsoft Windows Threads is a threading library supporting C++ for Microsoft Windows operating systems. It is included in Microsoft Windows SDK. Some programmers prefer it to Pthreads because of the simplicity of use. [24] 2. MULTICORE PROCESSORS AND THREADING LIBRARIES The easiest way to start parallel processing is to carry out multithreaded programs on multicore PCs. Although multithreaded programs can be executed on uniprocessor PCs, the performance gain will not be as close to be linear as observed on multicore PCs due to the unavailability of extra cores. Luckily, multicore PCs are very affordable these days. The realistic effects of parallel processing can be attained by the combination of multicore PCs and multithreaded programs. The drawback is that all threads can only be executed on the cores of the same chip. Consequently, only small scale multithreaded programs can be executed considering the number of cores incorporated into a chip allowed by current technology. However, the complete characteristics of parallelism still can be explored with this approach. The popular threading libraries and concurrent programming languages are introduced below. Intel Threading Building Block Intel Threading Building Block (TBB) [14] is the newest threading library which supports C++. It is downloadable at for free. It targets to provide high-level parallelism in contrast to low-level parallelism offered by raw threads libraries and Message Passing Interface (MPI). Issues like optimal management of a thread pool, proper distribution of task with load balancing, and cache affinity are automatically taken care of by TBB. Therefore, it is easier for beginning parallel programmers in addition to better performance than the alternatives. POSIX Threads POSIX (Portable Operating System Interface) Threads (Pthreads) is a threading library for C++ as well. It has been existing for more a decade and was the most popular threads library in the past. Hence, more documentations and textbooks are available for Pthreads. [12][13][15] Threads in Java and C# and Tasks in Ada Java, C#, and Ada are concurrent programming languages which directly support concurrency without having to use API. [16] The concurrent activities are called threads in Java [20][16] and C# [23] while called tasks in Ada. Different thread and tasks can be executed in parallel on different cores of a multicore processor. 3. CLUSTERS AND MPI LIBRARIES The second approach is to use clusters and execute MPI programs. MPI is a standard for parallel libraries supporting C++ and FORTRAN. [25][26] This approach is capable to support large scale parallelism as long as sufficient nodes are built into the cluster. Several books have been written about building a cluster with Linux. [2][3][4][5][6] However, all of them fail to point out the vital configurations required to make the cluster work correctly. The practical issues [11] in building an inexpensive cluster are addressed in the following subsections respectively. Hardware Requirements As Linux operating systems come with plenty of hardware drivers which cover almost all legacy and new hardware, any PC with 128B RAM or more and an Ethernet network interface card can work as a node. All nodes have to be connected with a network hub or switch. Generic Ethernet NICs and switches can be acquired with little cost. The logical layout is shown in Figure 1 and the physical configuration is displayed in Figure 2. KVM (keyboard, video, and mouse) switches shown in Fig. 3 can be installed to save space and devices. Multicore PCs can also be used to construct a cluster. The extra cores than uniprocessor still contribute to performance gains. Microsoft Windows Threads Fig. 1, Logical layout of the cluster

3 Fig. 2, Physical configuration of a 4-node cluster Open MPI, etc. Nevertheless, only Open MPI is still under active development and growing more powerful. Therefore, Open MPI is the best choice. The steps to install Open MPI are quite straightforward. They are described as follows. First, download openmpi- 1.3.tar.gz from Open MPI s website, Log into the root account to install it. Copy the compressed file to the root directory. Uncompress the file with the command line [8]: shell$ gunzip -c openmpi-1.3.tar.gz tar xf Then change to the directory openmpi-1.3. Configure and make Open MPI with the commands below. [7] Replace the directory after prefix option if you want to install into another directory. shell$ cd openmpi-1.3 shell$./configure - -prefix=/usr/local shell$ make all install Fig. 3, KVM (keyboard, video, mouse) switch Operating System and Compilers Among all operating systems, only Linux can be acquired for free. Although several implementations of open source Linux operating systems are available, not all of them work well with the MPI library. Sometimes, a version of Linux works with MPI, but the new features of the newer version demand more computing power and even interfere with the normal operations of MPI. After extensive experiments, the author chose and installed CentOS Linux V4.7 which is a clone of commercial Red Hat Enterprise Linux Version 4 [9] and downloadable at Firewall and SELinux should be turned off during installation as they cause difficulty for communications among nodes which are required for executing MPI programs. Fortunately, security is not a concern as long as the cluster is not connected with other networks. The editor, gcc C++ and FORTRAN compilers have to be installed to facilitate MPI programming process. After Linux is installed, networking should be correctly configured so that all nodes of the cluster can communicate among one another. In general, one user account with the same name needs to be created on each node because a parallel program is dispatched to each node under the same user account. Network Information Service (NIS) which is addressed in later sections can take care of this and other issues. MPI library The MPI library is an extension to C++ and FORTRAN language. [8][25][26] The next major step is to install the MPI library. Again, there are several implementations of MPI, such as MPICH, LAM/MPI, The make process may take up to one hour on an old PC. Disabling Password Authentication As Open MPI is configured by default to use ssh (secured shell) to dispatch parallel tasks, it is ssh that asks for password to authenticate the connection. Extra steps below will prevent ssh from requesting passwords [7]. Because the measure should only work for the user account which intends to run MPI applications, log into the user account of one node, say first node, to configure. First, generate the private and public key for the user account by executing: shell$ ssh-keygen -t dsa Next, log into the user accounts of all other nodes respectively and connect to itself by typing: shell$ ssh own_hostname That will generate the hidden.ssh directory on each node with the necessary attribute. Then go back to the first node to finish the remaining procedures. Change into the.ssh directory under the user account and do the following [7]. shell$ cd /home/user/.ssh shell$ cp id_dsa.pub authorized_keys

4 Next, repeat the following steps for all other nodes in the cluster. chkconfig nfs on chkconfig nfslock on shell$ scp authorized_keys remotehost:/home/user/.ssh shell$ scp id_dsa remotehost:/home/user/.ssh shell$ scp id_dsa.pub remotehost:/home/user/.ssh With these procedures done, all private, public, and authorized keys are duplicated on each node. They will be used for authentication for all future connections without passwords being requested from other nodes. Now the MPI applications can be executed on multiple nodes of this cluster without being asked for passwords. NFS and NIS Before we run a parallel program on our cluster, we need to dispatch a copy of the program binary code onto every node. Copying the binary code to all nodes is impractical. Network File System (NFS) is the solution to help us to complete this mission. With NFS the programs binary code only need to be saved on the file server and a copy of the program is automatically copied to the other computers. While NFS can solve the copying problem, there are some potential security problems. Each user has access to the shared directory, meaning any user can run, correct, save, and delete others programs. To remedy this problem, Network Information System (NIS) was used to manage the shared directory. Set Up the NFS Server [9] 1) From the NFS Server Configuration window, click File Add Share. The Add NFS Share window appears. In the Add NFS Share window Basic tab, type the following information: Directory Type the name of the directory you want to share. (The directory must exist before you can add it.) Host(s) Enter one or more host names to indicate which hosts can access the shared directory. Host names, domain names, and IP addresses that are allowed here. Separate each name with a space. Basic permissions Click Read-only or Read/Write to let remote computers mount the shared directory with read access or read/write access, respectively. 2) To permanently turn on the NFS service, type: Set Up the NFS Client [9] To set up an NFS file system to mount automatically each time you start your Linux system, you need to add an entry for that NFS file system to the /etc/fstab file. The /etc/fstab file contains information about all different kinds of mounted (and available to be mounted) file systems for your Linux system. The format for adding an NFS file system to your local system is the following: host:directory mountpoint options 0 0 The first item (host:directory) identifies the NFS server computer and shared directory. Mountpoint is the local mount point on which the NFS directory is mounted, followed by the file system type (nfs). Any options related to the mount appear next in a comma0separated list. For our system just need to add the following NFS entries to /etc/fstab: aggie1:/home /home nfs rsize=8192,wsize= Set Up the NIS Client [9][10] 1) Defining an NIS domain name: Our NIS domain name were Aggie, we can set it by typing the following as the root user from the shell: domainname Aggie To make the NIS domain name permanently, you need to have the domainname command run automatically each time your system boots. We can do it by adding the command line (domainname Aggie) to a run-level script that runs before the ypbind daemon is started. We edited the /etc/init.d/network file and added the following lines just after the first set of comment lines. # Set the NIS domain name. domainname Aggie 2) Setting up the /etc/yp.conf file: We had an NIS domain called Aggie and a master server called Aggie1, we should have the following entries in our /etc/yp.conf file: domain Aggie server Aggie1 domain Aggie broadcast ypserv Aggie1

5 3) Configuring NIS client daemons: We need set up an existing run-level script called ypbind to start automatically at boot time. To do this, you can run the following command (as root user from a Terminal window): chkconfig ypbind on 4) Using NIS maps: For the information being distributed by the NIS server to be used by the NIS client, you must configure the /etc/nsswitch.conf file to include nis in the search path for each file you want to use. In most cases, the local files are checked first (files), followed by nisplus. The following are examples of how some entries appear: passwd: files nisplus shadow: files nisplus group: files nisplus hosts: fiels nisplus dns For our purposes, we need change above entries into the following entries: passwd: files nis shadow: files nis group: files nis hosts: fiels nis dns As soon as the /etc/nsswitch file is changed, the data from NIS maps are accessible. No need to restart the NIS service. the following line into /etc/ypserv.conf to allow all hosts access to all maps: * : * : * : none 4) Configuring NIS serve daemons We can use the following chkconfig command to set ypserv and yppasswdd scripts to start automatically at boot time. chkconfig ypserv on chkconfig yppasswdd on 5) Updating the NIS maps If you modify the sources for NIS maps (for example if you create a new user by adding the account to the passwd file), you need to regenerate the NIS maps. This is done by a simple make C /var/yp This command will check which sources have changed, creates the maps new and tell yperv that the maps have changed. Running MPI applications On Single Node: Now the MPI applications can be executed on a single node. You should log into the user account to compile and execute MPI programs. MPI applications can be compiled with the command [7]: shell$ mpicc mpi_program.c o mpi_program Set Up the NIS Server [10][9]: 1) To configure your Linux system as an NIS master server, you should first configure it as an NIS client (That is, set the NIS domain name, set up /etc/yp.conf, and configure client daemons as described earlier.) 2) Creating NIS maps To create NIS maps so that your Linux system can be an NIS master server, start from the /var/yp directory from a Terminal window as root user. In that directory, a Makefile enables you to configure which files are being shared with NIS. All default configurations in Makefile are ok for our purposes, so we don t need change them. 3) Configuring access to maps In the /etc/ypserv.conf file, you can define rules regarding which client host computers have access to which maps. For our purposes we just need add MPI applications should be executed with the command below [7]. Replace the number in the np option (number of process) with any value you want. shell$ mpirun -np 4 mpi_program If you do not add the path to your login profile, you should add the path in front of mpicc and mpicrun [7]. The path is /usr/local/bin in this example. On Multiple Nodes: In order to run the MPI applications on multiple nodes, you need to specify the hostnames of all participating nodes in a file, my_hostfile in the example, and execute with the option hostfile [7]. shell$ mpirun - -hostfile my_host -np 4 mpi_program

6 4. CONCLUSIONS As the design of processors has met the physical limitation, namely the light-of-speed argument, and the speed of processor can no longer be further pushed, parallel processing is the only way to advance the computing performance. Despite the recognition of parallel processing as the new trend in computing field, institutions and individuals may be held back by the misconception of necessity of expensive parallel computers. The author introduced two effective approaches to start parallel processing education and/or research with minimal costs. The first method is to practice multithreading programming on inexpensive multicore PCs for small scale parallelism. The second approach is to practice MPI programming on a cluster for large scale parallelism. Owing to the accomplishments of open source software developers, it is possible to construct clusters with inexpensive PCs. It is feasible for all institutions and individuals to start parallel processing research and education with the two apporaches. Parallel libraries for programming with threads and processes are identified with their features. However, the free software packages come with minimal manuals and no support at all. It still could be a barrier to build a working cluster. The author describes the practical tips and undocumented major dilemmas in this paper. Any instructor and researcher interested in building an inexpensive cluster will find the tips helpful and it easier to build a working cluster. 5. REFERENCES [1] Grama, Ananth et al., Introduction to Parallel Computing, Addison-Wesley, 2003 [2] Sloan, Joseph D., High Performance Linux Clusters, O Reilly, 2005 [3] Bookman, Charles, Linux Clustering, New Riders, 2003 [4] Vrenios, Alex, Linux Cluster Architecture, Sams, 2002 [5] Lucke, Robert, Building Clustered Linux Systems, 2005 [6] Gropp, William et al., Beowulf Cluster Computing with Linux, 2 nd Ed., MIT Press, 2003 [7] FAQ [8] Pacheco, Peter S., Parallel Programming with MPI, Morgan Kaufmann Publishers, Inc [9] Sobell, Mark, A Practical Guide to Red Hat Linux, Third Edition, Prentice-Hall, 2006 [10] [11] Tseng, Y. and Hargrove, C, Construction of Parallel Machines for Engineering Courses with Retired Personal Computers, ASEE Southeastern Section Annual Conference, Memphis, TN, 04/2008 [12] Hughes, C. and Hughes, T., Professional Multicore Programming, Wiley, 2008 [13] Hughes, C. and Hughes, T., Parallel and Distributed Programming Using C++, Addison- Wesley, 2004 [14] Reinders, James, Intel Threading Building Blocks, O Reilly, 2007 [15] Nichols, B. et al., Pthreads Programming, O Reilly, 1996 [16] Welling, A., Concurrent and Real-Time Programming in Java, Wiley, 2004 [17] Breshears, C., The Art of Concurrency, O Reilly, 2009 [18] Parhami, B., Introduction to Parallel Processing, Plenum, 1999 [19] Buyya, R., High Performance Cluster Computing, Vol. 2, Prentice-Hall, 1999 [20] Magee, J. and Kramer, J., Concurrency, Wiley, 2006 [21] Yang, L. and Guo, M., High-Performance Computing, Wiley, 2006 [22] Kepner, J., Parallel MATLAB, SIAM, 2009 [23] Arora, G. et al., Microsoft C# Professional Projects, Premier, 2002 [24] [25] Gropp, W. et al., Using MPI, MIT Press, 1999 [26] Gropp, W. et al., Using MPI-2, MIT Press, 1999