Comsics: the parallel computing facility in the school of physics, USM.

Transcription

1 Comsics: the parallel computing facility in the school of physics, USM. Yoon Tiem Leong Talk given at theory group weekly seminar, School of Physics, Universiti Sains Malaysia Tues, 19 October 2010

2 Abstract The comsics computer cluster ( has been set up in the computer lab, school of physics, recently, and is currently being tested for its stability. Comsics comprises of 20 ordinary PCs linked by a 1000MB/s ethernet switch, and has a total of 4 x 20 = 80 cpus. It serves both teaching and research-related computational purposes. The main feature of the cluster is that it allows the execution of parallel codes using MPI. Large computational codes which takes too long to run on a sequential machine can be parallelised to run in parallel mode in comsics to shorten run time, providing a very cost-effective way to accelerate large numerical computation such as molecular dynamic simulation, DFT and quantum Monte Carlo calculation. In this talk I will brief on the main features of the computer cluster, explain some simple technicalities of how to use the cluster by new users. Some simple examples of parallel computing codes using MPI will also be demonstrated.

3 The comsics cluster page

4 The comsics cluster (photo)

5 Specification of the computer cluster Linux Rocks clusters 20 Acer PCs Quadcore i5 processor 4 GB RAM per PC, 500 GB Hardisk Each with an extra network card (10/100/1000 MB/s) 24-port 1000 MB/s switch NO NVdia GPU (yet) - said to provide much faster numerical calculation speed. Total cost: RM 60,000.

6 The 1000 MB/s 24-port switch (and the LAN cable wiring)

7 Hardware requirement Hardware requirement: ordinary PCs, extra LAN card, switch, LAN cable Install Linux Rocks Clusters 5.3 (customised CENTOS 5 Linux designed by UCLA for clustering purpose.)

8 How to install the Rocks linux OS Put the Rocks Linux CD in a selected PC as frontend (a.k.a mother node ) The eth1 network card (built-in) of the frontend is connected to the USM LAN network The eth0 network card (slot-in, purchased saparately) of the frontend is connected to the switch. Set the IP, DNS, gateway during the installation. Wait for 30 mins Once the frontend is up, call up the terminal as root, type insert-ethers Put the Rocks installation CD into the compute nodes in turn, switch on, the compute node will be configured automatically. A total of 19 nodes take about 20 mins to be installed.

9 Post installation configuration Post-installation configuration is done using shell script for installing additional software such as display card driver, virtual box, virtual Windows XP, Mathematica. The scripts cpatch.conf and fpatch.conf are found in www2.fizik.usm.my/configrepo/. Run fpathc.conf in the frontend. When done, run cpatch.conf in the nodes. This will take 1 hour Overall, the process of setting up of the rocks clusters takes around 2 hours, thanks to the deployment of semi auto-cruise shell scripts developed by ourselves.

10 LAN wiring The wiring of LAN cables are all done by ourselves - hardwork

11 LAN wiring

12 LAN wiring

13 Purposes Used as ordinary stand-alone PC for courses ZCE 111, MAT 181 For running parallel computing at the background (for research) Research and teaching 2-in-1

14 shows the current state of cluster, e.g., number of hosts up, number of cpus, history of the computational resource used in the past few days. It can be accessed even from outside USM via the internet. ssh -X or ssh -X using cygwin, linux terminal, Putty or winscp (all are Windows application)

15 You can run Microsoft Windows in comsics Each node in comsics has VirtualBox installed, in which students can launch Windows XP. ALL softwares in comsics are LEGAL, no pirated copies of software are to be installed in it.

16 Who should use comsics If you have computationally expensive codes that take too long to run in sequential mode, plus you know how to parallalise your code using MPI, e.g., Monte Carlo simulation, quantum Monte Carlo simulation. Write your own MPI version fortran or c++ codes (e.g., the find eigen values of or to diagonalise a huge matrix; finding global minimum of a complicated function) and run it in comsics. A single machine can also rin MPI version code but you would not gain any significant speeding-up for your numerical calculation. If you want to run software packages that support MPI parallelism, e.g., ABINIT, Wien2k, VASP, CRYSTAL, Gaussian03, LAMMPS, etc.

17 What can a parallel machine do for you A computationally expensive code usually is caused by a few bottle neck sections (usually 1 or 2). In parallel computing, the bottle-neck section (e.g., random number generation esp. in Monte Carlo programmes, diagonalisation of a matrix, eigen-value finding, minimisation algorithm) is splitted into many branches to be processed separately by the cpus in the nodes. As a whole the code s running time can be reduced, sometimes up to 10 or more times.

18 Who don t need to use the parallel cluster If you only use computer for the usual stuff, e.g., internet browsing, word-processing, power-pointing. If your own PC is sufficient for your computational needs. If you are happy with the cracked software in your Windows. If you only use the PC for gaming purpose (Linux has bad display driver). If you don t want to remotely log in to your PC. If you simply don t use computers. If you don t run parallel codes. If you don t know how to convert your sequential codes into parallel versions.

19 Can I use the machines with my Windows PC? Yes, but you have to install cygwin (shipped with X-windows) Winscp (without X-windows, hence text-only terminal, no GUI) ssh -X userid@comsics.usm.my

20 The advantages Ease of maintainance Ease of installation Ganglia provide easy way to check the cluster s resource status (% of cpu used, momory used in each node; logging of user history). Computation resource management: Quing system with SGE allows users to qsub their code to run in turn. Access from anywhere in the world.

21 The problems Some nagging stability problem of the cluster due to some unknown reason, possibly due to the way of managing users. Have been installing-uninstalling for many times since the last two months. Cabling problem (e.g., loose contacts, students unplugging the LAN cables, etc.) Nodes re-install itself when booted, for unknown reasons.

22 Softwares and libraries for scientific computing purposes Intel fortran, Intel c++, Intel impi gfortran, gcc Mathematica Intel MKL libraries, lapack, blas, ATLAS

23 Three types of MPI OpenMPI MPICH2 IMPI (intel MPI)

24 Software packages for research purposes First-principle DFT software packages ABINIT (free) WIEN2k ($$) CRYSTAL ($$) VASP ($$) These software are to be installed in parallel mode, so that we can run large system (e.g., a few hundreds atoms in a supercell) Other visualisation software, e.g., xcrysdens, gnuplot, endeavor, etc. LAMMPS (free) - already installed in parallel

25 Demonstration: hello world with openmpi Create a file in your home directory named machines with following lines: compute-0-0 compute compute-0-17 Now launch the job from the comsics master node: /opt/openmpi/bin/mpirun -np 9 -machinefile machines /opt/mpi-tests/bin/mpi-ring

26 Demonstration: cpi.c with openmpi cd /mpitest cp /opt/mpi-tests/src/mpi-ring.c. /opt/mpich2/gnu/bin/mpicc -o mpi-ring mpi-ring.c -lm /opt/mpich2/gnu/bin/mpirun -np 10 /mpitest/mpi-ring

27 Demonstration: LAMMPS running sequantial and parallel codes To demostrate how parallel code shrinks the time required for a sample program melt.

28 Conclusion We have almost succesfully set up a 80-cpus, 20 nodes, parallel computing facility in the School of Physics. Trouble shooting is ever-lasting battle, but most troubled are already or being tamed. Still some more fine tunning and optimisation to do (Nvdia GPU computing?) Will use it to run real computational physics researches: molecular dynamics, first principle calculations, quantum Monte Carlo.