User Training Eclipse Cluster

Transcription

1 User Training Eclipse Cluster

2 Morning

3 Agenda (Morning) Understand the Eclipse Configuration both Hardware and Networking configuration List of bundle software Overview Cluster Concept High Performance Computing (HPC) High Throughput Computing (HTC) How to remote to Eclipse Cluster both command-line and GUI How to upload/download you input and output file User Account and contact point for problem solving information Review essential files & directory Parallel Command Using Ganglia Web Monitoring 3

4 Understand the Eclipse Configuration

5 Hardware Architecture 1st of HPC in Thailand 30 Nodes 704 core 5

6 Hardware Architecture 6

7 Eclipse Specifications. 30 Nodes consist of Compute Node Spec. 2 Login 704 Core of Compute nodes 1408 GB of Ram 2 Management 4 File servers 22 Compute nodes. 32 TB of Storage. 7

8 Networking Configuration 4 Network Configuration Public Network Internal Network Gigabit Network Infiniband Network ILOM Network 8

9 List of bundle software

10 Software Rocks Cluster 5.1 (base on Redhat Enterprise) Programming C/C++ Fortran 90/95 Python 2.5 Perl 5 Ruby 1.8 Java 1.6 Parallel programming {MPI, PVM, LAM-MPI} Scheduler Sun Grid Engine

11 Bioinfomatics Software clustalw EMBOSS fasta glimmer gmap gromacs hmmer mpiblast mrbayes ncbi phylip t_coffee 11

12 Overview Cluster Concept

13 What is Cluster? Cluster computing is a technology related to the building of high performance scalable computing system from a collection of small computing system and high speed interconnection network 13

14 Cluster Architecture Compute Node Interconnection Network Frontend Node 14

15 Cluster Components Computer (PC,Server) Network (Gigabit Ethernet, Infiniband) Software OS (Linux, Windows) Middleware 15

16 Goal of Clustering High-performance clustering High-availability clustering 16

17 High-performance clustering Link many computers together to team up and finish problem fasters by having multiple computer working on the same problem independently 17

18 High-availability clustering Make more reliable computer system by having many computers working together and takeover when any of them fail 18

19 Type of Job High Performance Computing (HPC) High Throughout Computing (HTC) 19

20 High Performance Computing Maximum performance, not maximum throughput Use of specialised codes, libraries MPI (Message Passing Interface) Solve large problem by breaking it in to a number of small problems (data or task partitioning), then solve them on distributed, multiple processors at the same time. Pros and Cons Difficult since a parallel program must be developed Good when Problem is larger than memory size of a single machines Speedup for a single instance of problem is needed 20

21 High Performance Computing Sub task 1 Sub task 2 Big job Sub task 3 Sub task 4 21

22 High Throughout Computing High throughout, not high performance Complete most number of jobs in shortest amount of time Serial, parametric (usually), non-parallelized code Solve them on multiple processors at the same time, varying input parameters Example BLAST, Monte Carlo simulation Use of Load Schedulers 22 Pros and Cons Easy to get started. Use the sequential code in C or Fortran. Excellence for many type of applications such as Parametric computing: Running the same computation with multiple data set Distributed application such as massive rendering in animation industry

23 High Throughout Computing Jobs 23

24 Benefit of Cluster Low startup cost Scalability Rapid response, technology tracking More user-driven configuration Order of magnitude price-performance advantage 24

25 HPC World and Trends

26 TOP500 by Architecture Cluster 373 (74.6%) Constellation 20 ( 4.0%) MPP 107 (21.4%) 26

27 How to remote to Eclipse Cluster

28 2 Ways to Remote Command-line using SSH SSH (Secure Shell) is a protocol for creating a secure connection between two computers. SSH provides authentication and encryption and compression. GUI using VNC VNC (Virtual Network Computing) is software that makes it possible to view and interact with a computer from any other computer or device connected to the internet. VNC is even cross-platform, so a person using a Windows-based computer can connect to and interact with a Unix system without any problems whatsoever. 28

29 Using SSH Install SSH to your computer Double Click SSH and Click Quick connect 29

30 Login Enter host name : eclipse.biotec.or.th Enter user name : Your username Click Connect button Enter password : Your password 30

31 Your Home Directory If you login to cluster successful that you see your home directory. You can work and submit job to scheduler. 31

32 Transfer File Click New File Transfer Window 32

33 Upload file Right click to file and selected upload 33

34 Upload file (cont d) File transferring to your home directory 34

35 Download file Right click to file and selected download Dialog 35

36 Download file (cont d) Select directory or folder in your computer Click download button. 36

37 Download file (cont d) File transferring to your computer 37

38 Logout from SSH Click Disconnect menu 38

39 Using VNC VNC Server Session Remote to Eclipse Cluster by SSH Start vnc server #vncserver 39

40 Configure VNC Server Uncomment xstartup #vi.vnc/xstartup 40

41 Restart VNC Server Session You kill vnc server process #vncserver kill :id Ex: vncserver kill :3 You start vnc server session again #vncserver 41

42 Using VNC Client Install VNC Client to your computer Selected TrghtVNC Viewer 42

43 Using VNC Client (cont d) Enter host name and session id Eclipse.biotec.or.th:id Enter password : XXXXXXXX 43

44 Your server desktop 44

45 Logout from VNC Exit TrightVNC Client Kill session id process #vncserver kill :id 45

46 User Account and contact point for problem solving information

47 User Account You can to request your account and ask the problem solving information. By contact Mr.Chumpol Ngamphiw Biostatistics and Informatics Laboratory, Genome Institute National Center for Genetic Engineering and Biotechnology (BIOTEC) Phone: (66-2) ext

48 Review essential files & directory

49 Linux System Directory A simplified Linux directory/file system: / etc... bin dev... date... cal... home std01 exam.txt tmp... work... std50 hobby.c 49 proj1...

50 Linux System Directory (cont d) /home user home directory /bin programs using at boot time and system maintenance. /lib contain libraries that are needed to execute the binaries in /bin and /sbin. Particularly important for booting the system and executing commands within the root file system. /mnt temporarily mounted file systems. /media mounted other device 50

51 Linux System Directory (cont d) /opt contains large and static application software packages /usr is for files that are shareable across a whole site. The /usr directory usu ally has its own partition, and it should be mountable read only. The following directo ries should be subdirectories of /usr /tmp contains temporary files 51

52 Directory of application Bioinformatics /opt/bio/clustalw /opt/bio/emboss /opt/bio/fasta /opt/bio/glimmer /opt/bio/gmap /opt/bio/gromacs /opt/bio/hmmer /opt/bio/mpiblast /opt/bio/mrbayes /opt/bio/ncbi /opt/bio/phylip /opt/bio/t_coffee 52

53 Directory of application (cont d) Bioinformatics /share/apps/ amber9 /share/apps/cns_solve_1.21 /share/apps/modeller9v6 /share/apps/namd_2.6 /share/apps/autodock4 /share/apps/ MGLTools /share/apps/pymol /share/apps/procheck GIS /share/apps/454 Scheduler /share/apps/gridengine 53

54 Directory of application (cont d) Sequential Programming /share/apps/sun /share/apps/ Python /usr/bin/python2.5 /usr/bin/gcc /usr/bin/f95 /usr/bin/perl /usr/bin/ruby /usr/java/latest/bin/java Parallel Programming /share/apps/mvapich2 /usr/mpi/gcc/mvapich /opt/mpich /opt/mpich2 54

55 Basic Linux Command ls - list directory contents cd - change directories mkdir- make directories rmdir remove directories rm remove files and directories cp copy file and directories mv move or rename file and directories cat - concatenate files and print on the standard output pwd - print name of current/working directory vi - program text editor 55

56 Lab 1: Using Linux Command mkdir dir1 mkdir dir2 cd dir1 vi file1 ls cat file1 pwd cp file1../dir2 rm file1 cd.. 56 cd dir2 ls mv file1../dir1 ls cd.. cd dir1 ls cd.. rmdir dir2

57 File System Utilities du disk usage du /etc df report file system disk space usage df 57

58 File Permission UNIX provides three kinds of permissions: Read - users with read permission may read the file or list the directory Write - users with write permission may write to the file or new files to the directory Execute - users with execute permission may execute the file or lookup a specific file within a directory 58

59 File Permission (cont d) The long version of a file listing ls l /etc head will display the file permissions: drwxr-xr-x drwxr-xr-x -rw-r--r--rw-r--r-drwxr-xr-x -rw-r--r-drwxr-xr-x -rw-r--r--rw-r----- Permissions root root root root root root root root root Owner root root root root root root root root smmsp Group 59 Oct Nov Feb Feb Nov Aug Feb Oct Aug 9 14: d security a2ps.cfg a2ps-site.cfg acpi 13 13:49 adjtime alchemist aliases 13 13:57 aliases.db

60 Interpreting File Permissions -rwxrwxrwx Other permissions Group permissions Owner permissions Directory flag (d=directory, l=link, etc.) 60

61 Change file permission : chmod Use chmod command to change a file permission Example: $ chmod 755 file Letter Permission Description R Read 4 W Write 2 X Execute 1 61

62 Archiving Tool gzip GNU zip gzip <filename> gzip d <filename> tar tape archive tar zcvf <filename> <input> tar jcvf <filename> <input> tar zxvf <filename> tar jxvf <filename> 62

63 Change user password Use passwd command #passwd 63

64 Lab 2 : Permission file 1. Create the file1.tar using tar command 2. Change permission the file1.tar by 1. Owner can read, write and execute 2. Group can read and execute 3. Other can read only 64

65 Parallel command

66 Remote command Rocks cluster consists of frontend nodes and compute nodes Each nodes has its own Linux OS Some commands can be use for remote execution between nodes in Rocks cluster Remote terminal ssh Remote file processing scp 66

67 ssh (Openssh SSH client) Remote execution program for logging into a remote machine executing commands on a remote machine Secure (Use encryption) ssh [-l login_name] hostname user@hostname [command] Example ssh compute-0-1 ssh compute-0-1 ls ssh compute-0-1 cat /etc/hosts 67

68 scp (Secure Copy) Remote file copy program copies files between hosts on a network uses ssh for data transfer same authentication same security scp [[user@]host1:]file1 [[user@]host2:]file2 Example scp file1 compute-0-1:/home/username/ scp username@compute-0-1:/home/username/file1 username@compute-0-2:/tmp/ 68

69 Lab 3 : Using scp command 1. Copy file1 from front-end node to compute-0-1 node at your home directory 2. Copy file1 from compute-0-1 node to compute-0-2 node at /tmp 69

70 Parallel command execution There are some cases that we want to execute some commands on every machine Get host name on each node List directories in each node Frontend command command command This can be done by loging into each node (ssh) and executing each command separately Compute Compute Slow Not practical Compute 70

71 Parallel command execution Use cluster-fork to execute parallel jobs consisting of standard UNIX commands same command runs on multiple nodes execute on all nodes at the same time with only one command report output on frontend used only in frontend node 71

72 cluster-fork command cluster-fork [node-list] command node-list can be omitted command will be executed on every compute node (except frontend node) Examples cluster-fork ls cluster-fork hostname 72

73 Using Ganglia Web Monitoring

74 Monitoring Management Using Web monitoring Frontend node of the cluster serves as a web server By default, web access is restricted to only the internal cluster network For security purpose Need to edit iptables firewall rule to allow web access from outside network 74

75 Cluster home page Running in frontend node Provide various cluster monitoring via Web interface Cluster status Process status Etc. Goto Web Monitoring : 75

76 Cluster home page 76

77 Cluster status monitoring Using Ganglia a graphical web interface to live cluster information Provide dynamic information on each nodes such as CPU load Free Memory Disk usage Network I/O OS version Etc. 77

78 Ganglia Web Monitoring 78

79 View information each node 79

80 Compute node information 80

81 Process Monitoring Using cluster top web presents process information from each node in the cluster Like standard top command,except Cluster top shows processes running on every nodes Shows only processes that occupy CPUs (%CPU MAX) 81

82 Cluster top 82

83 Cluster top information 83

84 Cluster top information (cont d) Process Columns TN The age of the information in this row, in seconds. HOST The node in the cluster on which this process is running. PID The Process ID. A non-negative integer, unique among all processes on this node. USER The username of this processes. Etc.. 84

85 Job queue 85

86 Job queue information 86

87 Job queue information (cont d) Job Columns ID The job ID. USER The username of this job. PROCESSORS The number of processors that running job. STATE The job state. Etc.. 87

88 Afternoon

89 Agenda Using SGE Job scheduling Understand Job Scheduler concept Prepared Job Script Using SGE essential commands Using MPI via Infiniband Quick Review Parallel Programming Concept How to compile & run mpi program Submit mpi job via SGE 89

90 Using SGE Job scheduling

91 Understand Job Scheduler concept There are 2 types of job execution in clustering system Interactive Choose the node and run job interactively Batch Let the scheduler choose the node for you NOTE: In ROCKS, every job MUST be run with ordinary user, not root or system account. 91

92 Running interactive job ssh to a compute node and run it manually (ssh cp0-0) You can view the load of cluster using Ganglia command ganglia cpu_idle view current cpu utilization ganglia load_five view current load average over five minute ganglia mem_free view current available memory (may not accurate) Using qrsh command of SGE (Sun Grid Engine) to run the job qrsh /path/to/my/program (NOTE: running in home directory) qrsh cwd /path/to/my/program (NOTE: change to current working directory the run the program) SGE will select one available node manually NOTE: qrsh command submit the job to the queue. Job will be deleted if it can not be executed immediately 92

93 Running batch job The job is submit through a scheduling server (Scheduler) Scheduler determine an appropriate node to run the job base on its scheduling policy The job is then submitted to one (or more) compute node Output and Error of the job appear in one ore more file(s) NOTE: batch job MUST not read interactive input from user! Every job must 93 program argument read input from file or

94 What is SGE? Sun Grid Engine (SGE) is an open source community effort to facilitate the adoption of distributed computing solutions. Sponsored by Sun Microsystems Features : Automatic computing resource selection Resource Accounting Support for parallel computing (mpi) Support for Grid Computing 94

95 SGE Job Management 95

96 SGE Components Host type Master Host Control all jobs Run at frontend node Execution Host Host that compute the job(s) Run at compute node Submit Host Where user log-in and submit their job In ROCKS, frontend is also Submit Host Administrative Host Where admin log-in and do administrative task over SGE Also frontend in ROCKS. 96

97 SGE Components (cont d) SGE Software Components sge_commd - Communication daemon. Centralizing all communication. Run on all nodes sge_qmaster - Entry point for all command (qsub, qstat, etc ). Run at Master Host (frontend) sge_execd - Execution daemon. Run only on remote computing resource. Run at Execution Host (compute node) SGE Utility (qsub, qdel, qstat, etc ) - Utility command for user job submission and statistics. Install on Submit Host and Administrative Host only. 97

98 SGE Components (cont d) Queue A container for a class of jobs allowed to execute on a host concurrently A queue determines jobs types Cpu (itanium.q, xeon.q) Mem (himem.q) Time (short.q, long.q) Licences (Fluent.q) No need to submit job to a particular queue! Only need to specify your job requirements OS, software, mem SGE will dispatch to suitable queue on a low-loaded host ROCKS automatically setup queue for you! 98

99 Job management in SGE 1. Each user submit their job into SGE scheduler. No need to wait for the job to finish. 2. SGE choose node(s) to run the job. 3. Output and error of the job will be placed in output and error file 99

100 SGE Architecture 100

101 Basic SGE Command qsub - Job submission qstat - View job statistics qdel - Delete a job from queue qhost - show current online host 101

102 What is Job Script? Job script is a shell script that describe the job The program command Some job parameter (aka. qsub option) May include the command to start parallel job (such as mpirun ) 102

103 Prepared Job Script Before you submit job to SGE. you want create the job script. Example : Create a simple Job Script to submit the job #!/bin/sh date echo Hello world Save it to a file named simplejob and submit job by #qsub103simplejob

104 Basic job submission The job id will be shown after job submited After job finished, output will be placed in simplejob.o<job id> and error in simplejob.e<job id> 104

105 Job statistics Now create another job script called simplejob2 with the following content #!/bin/sh date echo sleep seconds sleep 1000 Submit the job qsub simplejob2 105

106 Job statistics (con t) Now, let s see the status of our job with qstat state qw means job is waiting in the queue (SGE is allocating a node for the job). Now try qstat again state t means job is starting. r means job is running 106

107 Job deletion Use qstat to see the job id of simplejob2 Now, let s delete the job with qdel <job id> 107

108 Job deletion (con t) Job output and error (until the job was killed) will be placed in simplejob2.o<job id>. 108

109 More on job submission Let s see what we can do on job submission Create a directory named myproject then cd to that directory mkdir myproject cd myproject Then, create a program myprog with the following content Compile this program into myprog gcc myprog.c -o myprog 109

110 More on job submission (con t) Now let s create a job script advancejob Note the./myprog line 110

111 More on job submission (con t) Now, try submiiting the job with the same command qsub advancejob Now, let s see the output 111

112 Option of SGE -cwd : specify the current working directories -o : specify the directory of output file -e : specify the directory of error file -S : specify the shell interpreter for the job script -N : specify job name -a : specify job start date ([YY]MMDDHHMM[.ss]) -j y : merge standard error to output file (advance.out) in this case -m be and M [Your address] : specify send status of job to your 112

113 Placing job option in the script You can specify the job option in job script, by prefix the line with #$ 113

114 The qhost command You can use qhost command to see the online node in SGE qhost Try supplying -j option and see what s happened (try it after submit some job) 114

115 Lab 4 : Using SGE Give you submit sequential matrix multiplication to SGE 115

116 Using MPI via Infiniband

117 The demand for computational speed The various class of problems often need huge quantities of repetitive calculations on large amounts of data to give valid results Computations must be completed within a reasonable time period 117

118 Limitation of a Uniprocessor System These problems cannot be solved in a reasonable amount of time with uniprocessor systems A uniprocessor speed is limited by Speed of light Physical properties of Semiconductor Scalability Memory expansion Input/output Storages Thus, we need the parallel processing to solve these problems 118

119 What is parallel programming? Parallel programming is writing program to split the overall problem into parts, each of which is performed by a separate processor in parallel. 119

120 Parallel Programming using Cluster Parallel programming usually done using message passing Message passing model Work are distributed among set of processes These processes communicate by passing network message to exchange data Standard PVM old message passing library, still popular MPI, de facto standard message passing 120

121 Parallel Processing Solving large problem by breaking it into a number of small problems, then solve them on multiple processors at the sametime Real life example building a house using many workers Assembly line in factory 121

122 Parallel Processing (cont d) LARGE SCALE PROBLEM Partitioning SUB TASK SUB TASKSUB TASKSUB TASK SUB TASK Mapping PROCESSOR PROCESSOR 122 PROCESSOR PROCESSOR

123 Parallelism in Application Parallelism in application is the potential of algorithm to be partitioned into a set of small tasks that run concurrently Important to the performance gained from parallel processing There are two types of parallelism Functional Parallelism Data Parallelism 123

124 Steps in Developing Parallel Application Develop sequential application first Identify Most time consuming part using profiling tool Parallelism inherent in that part Choose strategy for data partitioning and task partitioning Choose the development tools on target parallel machine Add code for task control and communication Compile, test, debug, measure performance, improve 124

125 How to Program a Parallel Computer Using Parallelizing Compiler Building applications yourself using parallel message passing library such as PVM and MPI Using parallel language such as High Performance Fortran and OpenMP Using public math library 125

126 Programming using Message Passing Partition task into multiple concurrent tasks that communicate by passing message MPI (message passing interface) a de facto standard that is now supported by all platform. Free implementations are also available (MPICH, LAM) Advantages Standard and portable High performance Disadvantage Very difficult to program126 Task1 Task2 Task3

127 What is MPI? MPI = Message Passing Interface It is a library specification for messagepassing, proposed as a standard by a broadly based committee of vendors, implementors, and users. Implementation e.g. MPICH, LAM For parallel computers, clusters, and heterogeneous networks Portable and efficient Run on SGI, CRAY, Linux Cluster 127

128 MPI Basic Functions MPI is complex (129 Functions) but a beginner, only 6 functions are enough to implement a simple algorithms. MPI_Init(int *argc, char **argv); MPI_Finalize(); MPI_Comm_size(comm,size); MPI_Comm_rank(comm,pid); MPI_Send(buf,count,datatype,dest,tag,comm); MPI_Recv(buf,count,datatype,source,tag,comm,status); 128

129 Basic Program (C) #include <stdio.h> int main( int argc, char *argv[] ) { printf( "Hello, world!\n" ); return 0; } 129

130 A Minimal MPI Program (C) #include "mpi.h" #include <stdio.h> int main( int argc, char *argv[] ) { MPI_Init( &argc, &argv ); printf( "Hello, world!\n" ); MPI_Finalize(); return 0; } 130

131 How to run this program 1. Compile Program /opt/mpich/gnu/bin/mpicc -o simple simple.c 2. Run Program /opt/mpich/gnu/bin/mpirun -np 4 simple 131

132 Result of Parallel Hello World Process 0 Process 3 Hello,World! Hello,World! 132 Process 1 Process 2 Hello,World! Hello,World!

133 How does program work? Process 0 Process 1 Process 2 Process 3 #include "mpi.h" #include <stdio.h> #include "mpi.h" #include <stdio.h> #include "mpi.h" #include <stdio.h> #include "mpi.h" #include <stdio.h> int main( int argc, char *argv[] ) { MPI_Init( &argc, &argv ); printf( "Hello, world!\n" ); MPI_Finalize(); return 0; } int main( int argc, char *argv[] ) { MPI_Init( &argc, &argv ); printf( "Hello, world!\n" ); MPI_Finalize(); return 0; } int main( int argc, char *argv[] ) { MPI_Init( &argc, &argv ); printf( "Hello, world!\n" ); MPI_Finalize(); return 0; } int main( int argc, char *argv[] ) { MPI_Init( &argc, &argv ); printf( "Hello, world!\n" ); MPI_Finalize(); return 0; } 133

134 MVAPICH on Infiniband network MVAPICH is compiler to running parallel program on infiniband network. install at /usr/mpi/gcc/mvapich-1.0.1/ 134

135 Using MVAPICH Compile program /usr/mpi/gcc/mvapich-1.0.1/bin/mpicc cpi.c o cpi Run program /usr/mpi/gcc/mvapich-1.0.1/bin/mpirun np 4 hostfile hosts cpi Note Before run program. You want create host file hosts are the contents : compute-0-1 compute-0-1 compute-0-1 compute

136 Submit MPICH job via SGE Create script file mpijob.sh #!/bin/sh #$ -cwd #$ -pe mpich 4 /opt/mpich/gnu/bin/mpirun -np $NSLOTS -machinefile $TMPDIR/machines hello Submit job #qsub mpijob.sh 136

137 Submit MVAPICH job via SGE Create script file mvapijob.sh #!/bin/sh #$ -cwd #$ -pe mpich 8 /usr/mpi/gcc/mvapich-1.0.1/bin/mpirun -np $NSLOTS -hostfile $TMPDIR/machines hello Submit job #qsub mvapijob.sh 137

138 Lab 5 : Run Parallel Program Running sequential matrix multiplication at size 200, 400, 800, 1000, 2000 and time the runtime using time function. Running parallel matrix multiplication at size 200, 400, 800, 1000, 2000 on 2,4,8, 16 processors Compare the runtime 138

139 The End. Question & Answer