New User Tutorial. OSU High Performance Computing Center

New User Tutorial OSU High Performance Computing Center

TABLE OF CONTENTS Logging In... 3-5 Windows... 3-4 Linux... 4 Mac... 4-5 Changing Password... 5 Using Linux Commands... 6 File Systems... 7 File Transfers (between your computer and Cowboy)... 8-13 Globus (large data transfers)... 8 WinSCP (Windows)... 9-10 Cyberduck (Windows and Mac)... 10-11 Filezilla (Windows, Linux and Mac)... 11-12 Command-line transfers (Linux & Mac)... 13 Batch Scheduler on Cowboy... 14 Submission Scripts... 15-16 Submitting Jobs... 16-17 Editing with Nano... 18 Cleaning Up... 19 Additional Job Submission Script Information... 19 Job Queues... 19 Processors Per Node (PPN)... 20 For applications that run within a single compute node... 20 For applications using MPI (more than one compute node)... 21 HPCC New User Tutorial - 2

Using a Terminal Most computer users are familiar with a graphical user interface (GUI). A GUI consists of visual icons and directories that are navigated through using a mouse. However, most high performance computing systems use a command-line interface through a terminal application because of its efficiency with certain types of computing. While there is a steep learning curve associated with using a terminal, it has a nice set of features that allow researchers to automate tasks so that the computer does most of the work, which saves time and money. Logging in to Cowboy Windows OS Most Windows operating systems (OS) don t have a terminal application preinstalled. Putty is a free and open-source terminal emulator that supports network protocols such as Secured Shell (SSH), which is what is used on Cowboy. If there isn't already an SSH client installed on the machine to be used, here are the instructions for using Putty: First download Putty from http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Download putty.exe, save it to your desktop and then run it. Start Putty. A window will open that looks like the image below HPCC New User Tutorial - 3

In the box labelled Host Name, enter the name of the system you want to connect to: cowboy.hpc.okstate.edu Be sure the Connection type is SSH and the Port number is 22. In the Saved Sessions box, you can give a name to these login settings, i.e. Cowboy, and click save. The next time you open Putty, you can double-click this saved session and it will open a terminal window that is ready for you to login to Cowboy. After saving your settings and clicking Open, you'll be prompted for your Cowboy username. Hit Enter after you ve typed your username. Next you ll be prompted to enter your password followed by Enter NOTE: your password will not show on the screen as you type If you entered your password correctly, you should get a command-line prompt. If it says something like permission denied or login invalid, you might have entered your password incorrectly. Remember passwords are case sensitive. To terminate your session on Cowboy after logging in, type exit or logout on the command-line prompt. Linux OS Linux OS will have some type of terminal application preinstalled. You should be able to launch a terminal from your desktop s application menu. In a terminal type: ssh username@cowboy.hpc.okstate.edu Where 'username' is your login name. For Example: If your username was 'pete', you would enter: ssh pete@cowboy.hpc.okstate.edu Next you ll be prompted to enter your password followed by Enter NOTE: your password will not show up on the screen as you type If you entered your password correctly, you should get a command-line prompt. If it says something like permission denied or login invalid, you might have entered your password incorrectly. Remember passwords are case sensitive. To terminate your session on Cowboy after logging in, type exit or logout on the command-line prompt. Mac OS Mac OS operates similarly to a Linux OS and will also have a preinstalled terminal application. At the top of the next page are directions for locating the Mac OS terminal. Open your Finder (hard-drive) window HPCC New User Tutorial - 4

Double click on the Applications folder Utilities folder Terminal icon When the terminal window opens, type: ssh username@cowboy.hpc.okstate.edu Where 'username' is your login name. For Example: If your username was 'pete', you would enter: ssh pete@cowboy.hpc.okstate.edu Next you ll be prompted to enter your password followed by Enter NOTE: your password will not show up on the screen as you type If you entered your password correctly, you should get a command-line prompt. If it says something like permission denied or login invalid, you might have entered your password incorrectly. Remember passwords are case sensitive. To terminate your session on Cowboy after logging in, type exit or logout on the command-line prompt. After logging in, you will be in one of Cowboy s login nodes. "Node" refers to a server or computer. Cowboy also includes 254 compute nodes. All these nodes are connected by a very fast private network. A cluster consists of these nodes, the network and a file system. **Using a mouse in a terminal: In Putty and linux s xterm, highlighting (with mouse) = copy Right clicking (Putty) = paste; Middle clicking (xterm) = paste In the Mac terminal, highlight (with mouse) + [Command]-C = copy Then use [Command]-V to paste In the linux GNOME terminal, [Ctrl][Shift]C = copy [Ctrl][Shift]V = paste Other than the functions listed above, your mouse has no other function. To move the cursor around, you use the keyboard s arrow keys. Changing Password Once you're logged in, on the command-line prompt type: passwd You will be prompted to enter your old password and then your new password twice. REMEMBER: Your passwords will not appear as you type) Please use a STRONG password Passwords should be at least eight characters long and include numbers, letters and other characters To log out type: exit or logout HPCC New User Tutorial - 5

Using Linux Commands A command-line interface (CLI) works by taking commands you type in and giving those commands to the operating system to perform. A few decades ago, computer systems were dominated by Unix code using CLI. In the 1990s, Linux, a Unix-like operating system, was developed, and it drives most of the CLI code today. Below are some common Linux commands that you might use frequently on Cowboy: pwd (print working directory) displays your current location on system ls lists content of directory cd (change directory) changes to a different directory cp copies a file to another name or location mv moves or renames a file "rm" removes/deletes files permanently grep - searches for specific words or strings nano -w nano is a text editor The -w tells nano to not put unintended line breaks into your file diff - compares two files to show differences tar - archive and unarchive source files man - help/usage for all system commands exit - terminate an SSH session Also, some important keyboard shortcuts: TAB - autocomplete commands and filenames The up arrow will cycle through your previously typed commands CTRL-D exits a terminal, same as typing `exit` CTRL-C breaks/cancels an ongoing operation, i.e. like the escape key The OSU HPCC website has a page of links to more Linux tutorials at https://hpcc.okstate.edu/content/hpc-education-resources Some of them are reference guides, but here's a good beginner tutorial for Linux: http://www.ee.surrey.ac.uk/teaching/unix/ HPCC New User Tutorial - 6

File Systems Each user will have access to two basic directories: the home and the scratch When you first log in, your current working directory is /home/<username>, where <username> is your Cowboy username. o /home/<username> has a quota of 25GB. Your source code and executables will be stored here. This filesystem is NOT BACKED UP, but it does have snapshots. To check your current usage, type du sh from your home directory. o /scratch/<username> is for large files and large collections of files. This filesystem is NOT BACKED UP. Users will also have access to a shared /opt directory. The /opt folder primarily contains applications needed by users. To request that an application be installed, please email hpcc@okstate.edu. NOTE: The first / in a file system path represents the root or the first or top-most directory in your file system. When you log into Cowboy, you are automatically in your home folder which stems from the root. Your current working directory would look like /home/<username>, with the first / representing the root. All proceeding / in a file path serve as dividers between any subsequent directories and files. HPCC New User Tutorial - 7

File Transfers between your computer & Cowboy Many people prefer to use a graphical user interface (GUI) application to transfer files. Globus is a file transfer application that handles large data transfers. For smaller data transfers, you can use one of the three recommended options (p. 7-8). If you d like to use a command-line interface (Linux and Mac OS only), please see the instructions on p. 9. Globus (large data transfers) Globus is an online service to facilitate fault tolerant large data transfers, Cowboy is setup as an 'endpoint' with name 'okstate#cowboy1'. To use Globus go to https://www.globus.org/ You can set up your computer as the other endpoint by downloading the Globus personal client. The Globus website also has several quick start guides. Globus can be utilized easily by following these steps: Create an account with Globus Online. Verify the account and login to Globus. Find the "Manage Data" drop down menu and click "Transfer Files". You should find yourself on a screen similar to this one: The endpoint for files on Cowboy is: o okstate#cowboy1 Once both endpoints are entered correctly, clicking one of the arrows will commence file transfer. For more information or to get started, visit https://www.globus.org/ HPCC New User Tutorial - 8

WinSCP (Windows) smaller transfers Download the 'Portable executables' from http://winscp.net/eng/download.php. This will be a zip file that contains the fine WinSCP.exe. You can place the WinSCP.exe on your desktop, this is the only file you need. To run WinSCP, double-click the WinSCP icon on your desktop. A window like the one below will open: In the 'Host name' box enter the hostname of the hpc system you wish to use. Our newest supercomputer hostname is: cowboy.hpc.okstate.edu In the 'User name' box enter your username (or leave it blank and it will prompt you for it later.) If you would like to save the above information for future use: Click the 'Save' button underneath the User name box. Double-click on the session you just saved to login, or highlight it and click 'Login' this is all you have to do in the future to connect to Cowboy using WinSCP. If you choose not to save the session information, press enter and it will begin connecting. Using WinSCP: Once you are connected, you should be greeted with a window similar to the one at the top of the next page. On the left is your local disk and on the right is your home directory on the Cowboy system. From here you can drag and drop to copy files in either direction. **You may be asked to confirm that you want to copy files. HPCC New User Tutorial - 9

Cyberduck (Windows and Mac OS) Download Cyberduck: https://cyberduck.io/ Click the Download icons below the duck the Start Download icon to the right of the duck is an ad. After downloading and opening Cyberduck, you should have a window that looks like this (this is the Mac OS X version the Windows version may look slightly different): **Cyberduck instructions continue on next page** HPCC New User Tutorial - 10

Click on the Open Connection icon at the top left. In the popup window, you will edit the following: o Choose the SFTP (SSH File Transfer Protocol) option in the drop down menu at the top. o For Server : cowboy.hpc.okstate.edu o Change Port to 22 o Enter your Cowboy username and password o Click Connect After connecting, your Cyberduck window should now look like the window below listing the contents of your home directory in Cowboy. You can drag and drop files on Cowboy from this window onto your Desktop and vice versa: Filezilla (Windows, Linux and Mac OS) Download FileZilla: https://filezilla-project.org/ Click the Download FileZilla Client All platforms icon After downloading and opening Filezilla, your window should look like the one at the top of the next page listing the contents of your computer s directory on the left. This is the Mac OS X version the Windows or Linux versions may look slightly different): HPCC New User Tutorial - 11

In this window, you will edit the following: o For Host type cowboy.hpc.okstate.edu It should automatically change to sftp://cowboy.hpc.okstate.edu o Enter your Cowboy username and password o Change Port to 22 o Click Quickconnect After connecting, your FileZilla window should now look like the window below listing the contents of your computer on the left and the contents of your home directory in Cowboy on the right: From here you can drag and drop to copy files in either direction. HPCC New User Tutorial - 12

Command-line file transfers (Linux and Mac OS only) To copy a file from Cowboy to your local machine: Open a terminal on your local machine, navigate to the directory you want the file to go, and type the following on the command line: o scp yourusername@cowboy.hpc.okstate.edu: directory_where_your_file_is_stored/yourfilename. o For example: scp pete@cowboy.hpc.okstate.edu: /scratch/pete/data/examplefile.txt. Be sure to include the space and the period after the filename, like it is highlighted above. You will be prompted to enter your Cowboy account password Once your file is copied from Cowboy to your local machine, in the terminal, you ll see the filename and 100% (along with some other information about the file) You can now find the file in the folder you sent it to on your local machine. To copy a file from your local machine to Cowboy: Open a terminal on your local machine, log in to Cowboy and navigate to the directory where your file is located Once there, type the following on the command line: o scp localfilename yourusername@cowboy.hpc.okstate.edu:/directory/ o For example: scp examplefile.txt pete@cowboy.hpc.okstate.edu: /scratch/pete/data/ You will be prompted to enter your Cowboy account password. Once your file is copied from your local machine to Cowboy, you ll see the filename and 100% (along with some other information about the file) Login to Cowboy and navigate to the directory you sent the file to and you will find the file. HPCC New User Tutorial - 13

Batch Scheduler on Cowboy Remember, when you log in to Cowboy, you are automatically located on one of Cowboy s login nodes. However, on Cowboy (or any HPC cluster) you don't actually run your jobs on the login node. You can edit files here, but when it's time to run your jobs, you will need to use the batch scheduler command to move you to a compute node. The job scheduler takes some information from you, finds the best compute node(s) to use, and runs your job there. This is the most efficient way to use the cluster. Login Set up jobs with submission scripts Submit them Log out and wait for your job to complete. You can even have the scheduler email you when your jobs are finished! We will cover this later. HPCC New User Tutorial - 14

Submission Scripts Below is an exercise you can complete to be able to view your own copy of a submission script. Remember, your username will be used in place of pete. 1. From the login node, change to the scratch directory (cd /scratch/pete). Then make a new directory called 'sample' (mkdir sample) and change to this new directory (cd sample): [pete@login2 ~]$ cd /scratch/pete [pete@login2 ~]$ mkdir sample [pete@login2 ~]$ cd sample [pete@login2 sample]$ pwd /home/pete/sample NOTE: The command pwd shows you what directory you are in, which is why the output at the bottom shows /home/pete/sample. At any point, you can use pwd to confirm where you are. 2. Now copy a sample submission script from our shared directory to your sample directory (cp /opt/examples/helloworld.pbs.) NOTE: the first command has a space after pbs followed by a period. The period represents current directory and specifies to where you'd like the file copied, which in this case should still be your sample directory. [pete@login2 sample]$ cp /opt/examples/helloworld.pbs. [pete@login2 sample]$ ls helloworld.pbs NOTE: The ls command lists the content of the directory, which is why the output below it will now show the file helloworld.pbs. At any point, you can use ls to see the contents of your current directory. **LINUX TIP: To reduce typing errors, remember your friends, the TAB (autocomplete) and up arrow (previous commands) functions!! 3. After typing the cat helloworld.pbs command, your sample submission script should look like the one below: [pete@login2 sample]$ cat helloworld.pbs #!/bin/bash #PBS -q express # specify the queue batch, express or bigmem #PBS -l nodes=1:ppn=1 # request 1 processor on 1 node #PBS -l walltime=10:00 HPCC New User Tutorial - 15

#PBS -j oe cd $PBS_O_WORKDIR module load examples helloworld NOTE: The following three lines should be checked when using a submission script: 1. #PBS -q express - This line determines what queue your job will run. 'Express' is one of four job queues. Learn more about job queues. 2. #PBS -l nodes=1:ppn=1 - This line determines how many nodes and processors per node (ppn) your job will need. Most jobs are fine using 1 node and 1 ppn. Learn more about nodes/ppn here. 3. #PBS -l walltime=10:00 - This line determines your job's time limit. Please try to accurately predict how long your job will take. a. Entering too long of a walltime may delay the start of your job as it waits for an open node. b. If your job is taking longer than originally expected, please email hpcc@okstate.edu to request a walltime extension as soon as possible. Include the lines #PBS -j oe and cd $PBS_O_WORKDIR in your script. module load <module name> will direct what module (or application/program) you'd like to run your job with. Visit the modules page to learn how to see what modules are available on Cowboy. If you have any questions about job queues, nodes and processors, or walltime, please email hpcc@okstate.edu. Submitting Jobs Since all our systems are shared resources, all jobs must be submitted to the queue and will be run as soon as the resources are available. **Do not run jobs on the login nodes. Anything more than file editing and very short compiles should be done through the scheduler! To submit the job, use the qsub command followed by your job name: [pete@login2 sample]$ qsub helloworld.pbs 414.mgmt1 This command submits the job to the batch scheduler, and returns a jobidnumber (in this case, it is 414, but it will be different when you run it). HPCC New User Tutorial - 16

To see all the running jobs type: $ showq NOTE: It can take several seconds for your job to launch, so if you don't see it at first, please wait a bit and try again. If you are really impatient, you can use the 'qstat' command. To view only your jobs, add the grep command and substitute your username for pete: [pete@login2 sample]$ showq grep pete 414 pete Running 1 00:59:28 Tue Sep 18 11:02:23 To see the output of your job while it is running, use the qpeek command and your job number: [pete@login2 sample]$ qpeek 414 Hello pete I am running on node n245.cluster You can stop (kill) your job using the qdel command and your job number: [pete@login2 sample]$ qdel 414 More information about checking job statuses or stopping jobs. Once your job finishes, you can view/list (ls) the output files created by the scheduler. The file will be listed as jobname.ojobidnumber **NOTE: The original helloworld.pbs file will also still be in the directory. [pete@login2 sample]$ ls helloworld.pbs.o414 helloworld.pbs [pete @login2 sample]$ cat helloworld.pbs.o414 Hello pete I am running on node n245.cluster waiting for 10 seconds successfully finished! Later, when you look at much bigger output files, use the less command: $less helloworld.pbs.o414 (**NOTE: Type q to exit the less viewer. Use arrow keys to move around, use '/' and a word to search.) For a more interesting example with less, try this: $less /opt/examples/pg2701.txt Within the less viewer, you can search for text. First type the forward slash character (/), then type 'Ahab' without the quotes and press enter. HPCC New User Tutorial - 17

Editing with Nano Now that you ve created and submitted the exercise submission script, you may now want to edit the script to fit your job s needs (i.e. which queue you want to use, the walltime amount, etc.). This can be done in the terminal using a text editor. Nano is a text editor and is good for editing small files. If you are going to edit larger, more complex files, you can learn other text editors such as vi or emacs. There are a variety of text editors out there. Many people have strong opinions as to which is best, but you should find the one you re most comfortable using. Let's edit this script with nano by using the nano command followed by the file to edit: [pete@login2 sample]$ nano helloworld.pbs You should see the contents of the helloworld.pbs script we copied earlier. To practice editing a script and to learn how to set email notifications for your jobs, follow the directions below: Use the arrow keys to go down to this line: cd $PBS_O_WORKDIR Press enter to add a blank line above. On that line type the following to make the scheduler email you when your job begins, ends or aborts: #PBS -m abe -M <youremailaddress> At the bottom of the screen you should see menu options. To exit nano, it says ^X. The ^ symbol stands for CTRL, so to exit you d use "CTRL-X". o If you have not made any edits, you will be returned to the command line. Because we have made edits, when you press CTRL-X, you'll see this question at the bottom of the screen Save modified buffer (ANSWERING "No" WILL DESTROY CHANGES)? Y Yes N No ^C Cancel Press 'y' to save. Then just press 'Enter' to accept the filename helloworld.pbs. If you submitted the script again by following the steps in the Submitting Jobs section, you should now receive an email to the email address you entered when the job is completed. HPCC New User Tutorial - 18

Cleaning Up After your job runs successfully you can clean up your directory by deleting unneeded files. WARNING: There is no way to undelete a file in linux, so be absolutely sure you do not need a file before removing it! In the example from above, you will have at least one output file from running the job (i.e. helloworld.pbs.o414), possibly more. [pete@login2 sample]$ ls helloworld.pbs.o414 helloworld.pbs To FOREVER delete the file helloworld.pbs.o414, use the rm command: [pete@login2 sample]$ rm helloworld.pbs.o414 [pete@login2 sample]$ ls helloworld.pbs NOTE: No output is produced after typing rm helloworld.pbs.o414, but if you type the list command ls, now you will see helloworld.pbs.o414 is no longer contained within your sample directory. You have successfully removed it. Additional Job Submission Script Information This section will explain in more detail about Job Queues and Processors Per Node. Job Queues Cowboy s job queues facilitate fair share scheduling of its resources. You can edit your script to use one of four queues: batch this is the default queue and has a walltime limit of 120 hours (120:00:00). If your job needs to run longer than this and your software does not have checkpoint/restart capabilities, please email hpcc@okstate.edu as far in advance as possible to extend your job s walltime. express this is for short jobs and debugging/testing scripts. It contains two compute nodes and has a walltime limit of one hour (1:00:00). bigmem this queue directs jobs to one of two compute nodes that have 256 GB RAM and a NVIDIA Tesla C2075 GPU card. The walltime limit is 120 hours. killable this queue is for long running jobs that are unable to use a checkpoint/restart feature. The walltime limit is 504 hours (504:00:00). Jobs in this queue are subject to being killed, at the discretion of HPCC administrators, for hardware and software issues. HPCC New User Tutorial - 19

Additional Job Submission Script Information Processors Per Node Each compute node on Cowboy has 12 processors (or cores) and 32 GB of RAM. For applications that run within a single compute node: If your code only utilizes a single core, it is better to only request a single core: #PBS -l nodes=1:ppn=1 However, if your code can use all 12 cores efficiently, request all 12 cores in your job script: #PBS -l nodes=1:ppn=12 In general, request 1 core for each 2.5 GB RAM that your application needs. For example, if your code needs all 32 GB RAM you want to request all the cores of the node so that your job has exclusive use of the node: o #PBS -l nodes=1:ppn=12. If your code needs 16 GB RAM, you can request half the node (6 cores): o #PBS -l nodes=1:ppn=6 o NOTE: In this situation, if you submit 2 jobs, then both of them may run on the same node. If your application needs more than 32GB (shared) RAM, please use the bigmem queue. The bigmem nodes have 256 GB RAM and 12 cores. Request 1 core for each 20 GB RAM your code requires. #!/bin/bash #PBS -q batch #PBS -l nodes=1:ppn=12 #PBS -l walltime=24:00:00 #PBS -j oe Example submission script using a single node: module load <software_module_name> cd $PBS_O_WORKDIR <put the commands to run your application here> Continued next page HPCC New User Tutorial - 20

For applications using MPI (applications that can use more than one compute node) #!/bin/bash #PBS -q batch #PBS -l nodes=16:ppn=12 #PBS -l walltime=24:00:00 #PBS -j oe Example submission script using multiple nodes module load mvapich2-1.8/intel cd $PBS_O_WORKDIR NP=`cat $PBS_NODEFILE wc -l` mpirun -np ${NP}./yourexecutable NOTE: The ` (back-tick/accent grave) character in the NP line is not an apostrophe. The key for this character is in the upper left corner of the keyboard that has the tilde ~ character. HPCC New User Tutorial - 21