Introduction to Sheffield University High Performance Computing Facilities

Transcription

1 Introduction to Sheffield University High Performance Computing Facilities Deniz Savas dsavas.staff.shef.ac.uk/teaching June 2017

2 Objectives 1. Understand what High Performance Computing is 2. Be able to access remote HPC Systems by different methods 3. Transfer data to and from the HPC cluster 4. Know how to use the different kinds of file storage systems

3 What is High Performance Computing National HPC Support Tier1: Provided by EPSRC for peer reviewed research on ARCHER in Edinburg Tier 2: Provided in six different locations for less intensive but more diverse usage University local HPC Clusters Such as provided at Sheffield University by iceberg and sharc clusters. Dedicated Grid Computing Tacking compute problems for a specific research field by distributed working such as Particle Physics Data Grid Commercial Cloud Computing Access resources over internet on demand, Amazon EC2, Microsoft Azure etc

4 Supercomputing Capability Computing e.g. Parallel Fluent, Molecular dynamics, MHD Capacity Computing Throughput computing blast searches, pattern searches, data searching Grid computing Heterogeneous resources e.g. ppgrid, DAME, cosmogrid (large scale capability) At home projects, boinc, Condor etc. (capacity computing)

5 Grid Computing : Data Grid Engine flight data Particle physics data grid, Distributed Aircraft Maintenance Environment Computing Network stores large volume of data across network Heterogeneous Multiple data sources London Airport Airline Maintenance Centre Grid New York Airport Diagnostics centre American data center European data center

6 High Performance Computing Tiers Tier 3 Computing Iceberg, ShARC Gateway to High Performance Computing Tier 2 Computing Polaris larger scale and more challenging problems Tier 1 computing Archer large scale grand challenge problems

7 Projects Atomistic & Mesoscale Simulations for Radioactive Waste Disposal Biomineralization and Biomimetics Computational Biology Computational Biology INSIGNEO - Institute for in silico medicine Designing Nanomaterials for Energy Applications Large Scale Calculations for Aerospace & Energy Applications Molecular Ecology Group, Genomics MHD Wave Coupling in the Solar Atmosphere

8 Iceberg: Summary of Facts & Figures Processor cores: 3440 cores + 16 GPUs Total Main Memory: 31.8 TB Filestore: 45 TB Temporary Disk Space: 260 TB Physical size: 8 racks Power consumption: 83.7 KW

9 iceberg cluster specifications Intel Ivy Bridge nodes all infiniband connected, containing; 92 nodes each with 16 cores and 64 GB of memory ( i.e. 2 * 5-core Intel E v2 ) 4 nodes each with 16 cores and 256 GB of memory TOTAL INTEL CPU CORES = 1152, TOTAL MEMORY = 2400 GB Scratch space on each node is 400 GB Intel Westmere based nodes all infiniband connected, containing; 103 nodes each with 12 cores and 24 GB of memory ( i.e. 2 * 6-core Intel X5650 ) 4 nodes each with 12 cores and 48 GB of memory TOTAL INTEL CPU CORES = 1152, TOTAL MEMORY = 2400 GB Scratch space on each node is 400 GB

10 GPU computing hardware attached to iceberg Kepler K40M all infiniband connected, containing; 8 Nvidia Tesla Kepler K40Ms GPU units for GPU programming 2880 thread processor cores per GPU 12GB GDR memory per GPU Total GPU memory = 96 GB Each GPU unit is capable of about 4.3TFlop of single precision floating point performance, or 1.4TFlops at double precision. Hence yielding maximum GPU processing power of upto 11.2 TFlops in total. Fermi M2070 all infiniband connected, containing; 8 Nvidia Tesla Fermi M2070s GPU units for GPU programming 448 thread processor cores per GPU 6GB GDR memory per GPU Total GPU memory = 48 GB Each GPU unit is capable of about 1TFlop of single precision floating point performance, or 0.5TFlops at double precision. Hence yielding maximum GPU processing power of upto 8 TFlops in total.

11 Iceberg Cluster There are two head-nodes for the iceberg cluster login HEAD NODE1 HEAD NODE2 login qsh,qsub,qrsh Iceberg(1) Iceberg(2) qsh,qsub,qrsh Worker node Worker node There are 232 worker machines in the cluster Worker node Worker node Worker node All workers share the same user filestore Worker node Worker node Worker node Worker node Worker node Worker node Worker node

12 SHARC Cluster Sheffield Advanced Research Computer 98 Standard memory Nodes with following specs Dell PowerEdge C6320 computers, with dual-processor Intel Xeon E v3 CPU containing 16 cores in total at 2.40 Ghz clock-rate. Total RAM per node: 64 GB (i.e. 4 GB / core) Local storage per node: 1 TB SATA III HDD for /scratch and /tmp 4 Big memory nodes: 64 cores, 16GB/core 8 NVIDIA Tesla K80 GPUs each with 24GB memory for computation Four Large memory nodes same as above but with 128 GB memory each Two Login Nodes ( for resilience ) Two GPU nodes The GPU nodes are the same as standard nodes but are fitted with 8 NVIDIA Tesla K80 GPU cards. One Hardware Accelerated GPU node for Visualisation node Fitted with two Quadro K4200 graphics cards. Filestore Fast network filesystem (Lustre): 669 TiB (/home and /data are shared with Iceberg)

13 Registering to Use HPC at the University of Sheffield All staff and research students are entitled to use iceberg See: Staff can have an account by simply ing

14 Passwords In normal linux environment the passwd command can be used to change the user passwords. However, because we manage passwords centrally this command will not work on iceberg. If you wish to change your iceberg password you will have to do this via a web interface at the following URL:

15 Demonstration 1 Research computing website Iceberg documentation HPC Discussion group Checking and Changing your password

16 Accessing iceberg and sharc by remote logging in Terminal access is described at: Recommended access is via any browser at: This uses Sun Global Desktop ( All platforms, Graphics-capable) Also possible: Using an X-Windows client ( MS Windows, Graphics-capable) Exceed 3D Cygwin Various ssh clients (MS Windows, Terminal-only ) putty, SSH Note: ssh clients can also be used in combination with Exceed or Cygwin to enable graphics capability. Above web page describes how this can be achieved.

17 Web browser method of access: Sun Global Desktop Start session on headnode Start an interactive session on a worker Start Application on a Worker Help

18 SSH method of access from MAC or Linux platforms You only need the SSH client. The server is unnecessary, unless you wish to connect back to your home machine via the Internet using SSH. Connecting to Iceberg or sharc - ssh -X yourusername@iceberg.shef.ac.uk ssh -Y yourusername@iceberg.shef.ac.uk ( on MACs ) NOTE: substitude sharc for iceberg above for logging into sharc -X flag is optional, it allows the usage of graphical applications on the remote computer. ssh has large number of options for changing ports, specifying authentication files, encryption algorithms etc. On the remote computer type: man ssh for further info Note1: -X parameter is needed to make sure that you can use the graphics or gui capabilities of the software on iceberg. Note2: Depending on the configuration of your workstation you may also have to issue the command : xhost + before the ssh command.

19 Basic X Concepts X Server runs on local machine PC Exceed, Cygwin, Xming UNIX Workstation Included in OS Apple Mac Exodus X Client runs on remote machine Graphical Application xterm xcalc Modelling and visualisation packages etc.

20 Your HPC filestore Four areas of filestore always available on iceberg and sharc These areas are available from all headnode and worker nodes. 1. home directory: This is same whether you are on iceberg or sharc /home/username 10 GBytes allocations Permanent, secure, backed up area ( deleted files can be recovered ) 2. data directory: This is same whether you are on iceberg or sharc /data/username 100 GBytes of ollocation Not backed but mirrored on another server 3. /fastdata area ( this is different on sharc and iceberg ) /fastdata Much faster access from MPI jobs No storage limits but no backup, or mirroring Files older than 90 days gets deleted automatically Always make a directory under /fastdata and work there. 4. If you are in a research group you may also have access to a shared area Please see this link for further information.

21 Storage Allocations Storage allocations for each area are as follows: On /home 10 GBytes On /data 100 GBytes /fastdata no-limits ( but not permanent ) Check your usage and allocation often to avoid exceeding the quota by typing quota If you exceed your quota, you get frozen and the only way out of it is by reducing your filestorage usage by deleting unwanted files via the RM command ( note this is in CAPITALS ). Each worker has its own unique /scratch area. Therefore any data on /scratch will not be available from other workers on the cluster.

22 Transferring files to/from iceberg and sharc Summary of file transfer methods as well as links to downloadable tools for file transfers to iceberg are published at: Command line tools such as scp, sftp and gftp are available on most platforms. You can not use ftp to iceberg and sharc Use sftp instead. Graphical tools that transfer files by dragging and dropping files between windows are available winscp, coreftp, filezilla, cyberduck

23 Secure ftp (sftp) Establishes an FTP-style file transfer session between the Unix systems sftp command always used in the form: sftp e.g.from iceberg: sftp

24 Transferring Files Using sftp From the SFTP prompt (sftp>) can do the following: get command to retrieve a file from the remote Unix server. get test.txt put command to transfer a file from your Unix system to the remote Unix system you are connected to. put file2.txt

25 Navigating file systems using sftp From the SFTP prompt (sftp>) can do the following: ls command to display the contents of a directory on the remote Unix system you are connected to. ls /home/user. Will display the contents of the directory /home/user on the remote Unix system. cd and lcd commands change current remote directory, or current local directory. e.g. cd /home/user. Will change the current remote directory to /home/user.

26 Summary of sftp commands get : Transfer a file from remote location to local area Example: get myremote.fil put : Transfer a file from this computer to remote computer Example: put mylocal.fil mget : Retrieve multiple files from server Example: mget *.dat mput : Transfer multiple files to server Example: mput file?.log pwd : Display remote working directory quit or exit : Quit sftp and exit to shell rename : Rename remote file Example: rename myold.fil mynew.fil rmdir path: Remove remote directory rm path : Delete remote file version : Show SFTP version? : Synonym for help

27 Summary Listing of SFTP commands continued ascii : Use text transfer mode binary : Use binary transfer mode Image : Use binary transfer mode help :Display the help text lls [ls-options [path]] Display local directory listing lmkdir path : Create local directory ls [path] : Display remote directory listing mkdir path : Create remote directory put local-path [remote-path] Upload file

28 Secure copy - scp Using SCP Fast, easy method to copy single files from your Unix system to a remote Unix system.

29 Retrieving a file using SCP To retrieve a file from a remote Unix system, the syntax is: scp username@server:file local-file username= username on the remote system server= the name of the remote Unix system file= the file to retrieve from the remote system local-file= the location you wish to save the file to on your local Unix system

30 Transferring a file to a remote Unix system using SCP The syntax is: scp local-file username@server:file local-file= the file to transfer from the local system username= username on the remote system server= the name of the remote Unix system file= the location you wish to save the file to on the remote Unix system

31 Transferring Multiple Files Use mget and mput with sftp OR Use tar to compress all files into an archive and transfer that tar archive and at the other end uncompress it using tar at the other end EXAMPLE: On your local Linux workstation: tar cvf myfiles.tar na* tar tvf myfiles.tar : create and archieve : check contents of archieve scp myfiles.tar your_username@iceberg.shef.ac.uk On iceberg after logging in: tar xvf myfiles.tar : extract the archieve :transfer it

32 Pitfalls when transferring files ftp is not allowed to by iceberg. Only sftp is accepted. Do not use spaces in filenames. Linux do not like it. Secure file transfer programs sftp classify all files to be transferred as either ASCII_TEXT or BINARY. All SFTP clients attempt to detect the type of a file automatically before a transfer starts but also provide advanced options to manually declare the type of the file to be transferred. Wrong classification can cause problems particularly when transfers take place between different operating systems such as between Linux and Windows. If you are transferring ASCII_TEXT files to/from windows/linux, to check that transfers worked correctly while on iceberg, type; cat v journal_file If you see a ^M at the end of each line you are in trouble!!! CURE: dos2unix wrong_file on iceberg

33 What is shell? Provides an Interface to the UNIX Operating System It is a command interpreter Built on top of the kernel Enables users to run services provided by the Unix OS After login in, your interactions takes place with the shell What is a shell script? In its simplest form a series of commands in a file is a shell script. By simply invoking the name of the shell script we can execute all the commands it contains in the correct order. Thus creating shell scripts saves us from having to retype commands to perform a common set of tasks. Shell provides a secure interface between the user and the kernel of the operating system.

34 Secure Shell Program to log into another computer over a network Execute commands on a remote machine Provides strong authentication and secure communications over insecure channels. Intended as a replacement for legacy unix utilities such as rlogin, rsh, rcp, and rdist.

35 Few Basic Rules Unix is case sensitive. Commands are in lower case. Backspace and/or Del Keys correct typing errors. If the terminal parameters are not correctly set; try Ctrl+H Ctrl+C Aborts a program or command. You can use the arrow keys to recall previous commands, optionally edit and execute them.

36 Key combinations in bash Ctrl+A Ctrl+C Ctrl+D Ctrl+E Ctrl+H Ctrl+Z Arrow left/right Arrow up/down Tab Tab tab Move cursor to beginning of line <Break> Stop running program <Logout> Move cursor to end of line Delete character to the left Suspend the program Move pointer left/right to insert chars etc. Recall previous commands so as to edit them andsubmit them by pressing ENTER Command or filename completion. Auto-complete the command line (to save typing) If tab does not work tab again to show possibilities

37 Demonstration 2 Logging into iceberg (start an interactive session) Sunglobal desktop Putty Exceed Transferring files to and from Iceberg For the practice sessions you can use the linux cheat sheet at

38 Practice Session Problem 1-2 on the worksheet Start an interactive session on iceberg using the Sun Global Desktop Start a file transfer client and move files to and from iceberg Extract the course examples, change directory to a directory you will be using for the course and type mkdir ddp cd ddp tar zxvf /usr/local/courses/hpc_intro_long.tgz

39 Getting help Research Computing Support Home Page HPC at Sheffield Home Page Training (also uses the learning management system) Discussion Group (based on google groups) the group Contacts