Welcome to the Introduc/on to BioHPC training session. My name is David Trudgian, and I m one of the Computa/onal Scien/sts in the BioHPC team.

Transcription

1 Welcome to the Introduc/on to BioHPC training session. My name is David Trudgian, and I m one of the Computa/onal Scien/sts in the BioHPC team. We ll introduce the other staff in a minute. This is the only mandatory training session for BioHPC users, but we hope you ll come along to many of our other sessions. We hold this introduc/on on the 1 st Wednesday of the month, further basic training on the 2 nd Wednesday, advanced topics on the 3 rd Wednesday, and a drop- in coffee session on the 4 th Wednesday. All the details are on our web portal. The slides for this session will be posted on the portal, in the training sec/on. There ll be notes with the slides, covering roughly what I tell you here today. 1

2 This session is our quick introduc/on to BioHPC. We ll introduce the team, tell you about the systems and services we provide, and give you instruc/ons for the basics of accessing our resources. We re going to concentrate on the simplest way to use our systems, via our web- based cloud services. If you re a more advanced user you ll find informa/on about how to connect via the command line on our portal and in later training sessions. Less fun, but s/ll important, we ll cover a few rules to follow when using the systems, and some things we want to encourage you to do to get the most out of BioHPC. If you remember one thing from today, let that thing be the address biohpc- help@utsouthwestern.edu The biohpc- help address goes into our support /cket system, where the best person for the job is going to answer your ques/ons. If you have *any* ques/on please ask us. Our whole reason for being here at UT Southwestern is to help you to effec/vely carry out computa/onal research. 2

3 So let s start with a brief introduc/on to the BioHPC team. BioHPC was established a bit over a year ago when the director, Liqiang Wang, was recruited to setup an HPC facility for a subset of departments at UTSW. Liqiang has extensive experience in IT infrastructure and HPC, and now leads a team of 3 others. Yi Du is a Computa/onal Scien/st, with a background in parallel sowware design and large- scale data analysis. She is experienced in the more mathema/cal applica/ons of HPC, coming from an oceanography background. My exper/se is in more general bioinforma/cs work. I spent the last 6 years working as a bioinforma/cian in computa/onal mass- spectrometry proteomics, first at Oxford, then at UT Southwestern. Last but not least, Ross Bateman is a technical support specialist who will be looking awer the hardware you are using, suppor/ng our desktop clients, and providing general system maintenance and user support. We all sit behind the biohpc- help address, so please use that to reach us, instead of ing our individual accounts. 3

4 Let s briefly touch on what High Performance Compu/ng is, and why we need it at UTSW. Tradi/onally, HPC has had quite a narrow defini/on in the minds of many people. You ll owen see it defined as using parallel programs across a cluster of computers, to solve very complex problems. At the majority of very large HPC centers most HPC tasks do fall into this defini/on, par/cularly large computa/onal models in field such ass meteorology, physics, geology etc. In the biomedical sciences we have different needs from HPC, and we re going to define High Performance Compu/ng as compu/ng that cannot be carried out on standard systems whatever the reason. We owen have huge datasets from imaging or sequencing experiments, that need large amounts of storage. OWen these datasets aren t prac/cal to process on even a powerful desktop computer. For example, genome assembly sowware requires many GB of RAM. Processing of a large series of high- resolu/on microscope images may take an excessive amount of /me on a single system. The solu/on isn t always parallel sowware running across many machines. Some/mes we have modest requirements, except a need for fast, huge storage. Some/mes we need 256GB RAM on a single machine. We need to use a wide range of sowware, some of which is parallelize, some which isn t. Op/miza/on of home- built and download sowware is frequently necessary for large projects to become tractable. 4

5 Back to us - As well as a team of staff, BioHPC is computer hardware, sowware, and services. Over at the IR data hall in the Bass Center we have a 41- node compute cluster, and over 1 petabyte of storage. The compute cluster is the center of the BioHPC world, and we ll get back to it later. We also provide access to wide array of sowware, on the cluster and via thin- client and worksta/on machines that we administer. A big push at present is to provide web- based cloud services, so you can use the cluster, storage, sowware from a web browser. This opens up our facili/es to users who aren t yet comfortable with the Linux command line. We ll be concentra/ng on these web- based services in this session. 5

6 Our compute cluster is called Nucleus, and has 41 nodes right now. It s a heterogeneous cluster, where the nodes have different specifica/ons. At present there s a mix of 64, 128, 256, 384GB nodes, plus 2 nodes with GPU cards. Very soon the size of the cluster is going to double, and the number of GPU node will more than double. We ll also be upgrading the amount of RAM in the 64GB nodes, so that all nodes have a minimum of 128GB. The cluster is running RedHat Enterprise Linux 6, and uses the SLURM job scheduling sowware. Not by accident this is the same basic setup as the TACC Stampede supercomputer in Aus/n, which UTSW members can obtain /me on. The tradi/onal way to use the cluster is via a command- line session, logging in to the login- node using an SSH session. The login node is accessible on campus or when connected to the VPN via SSH to nucleus.biohpc.swmed.edu You can also run jobs on the cluster from our web portal, which we ll cover today. 6

7 For the size of our cluster we have a very large amount of storage, reflec/ng the type of HPC work that goes on in the biomedical sciences, where large datasets but rela/vely small processing jobs (compared to other fields) are quite common. Two storage systems are used to provide private, per- user home directories, as well as user and lab shared space on project and work file systems. The lysosome storage system is our high- capacity, lower- speed system. It provides your home directory, and project space. The endosome storage system is a higher- speed parallel storage system, which provides the work file system. Every user has at least a 50GB quota for their home directory, which has a path / home2/username Project and work quotas are arranged separately, and organized by department and lab groups under the /project and /work paths. 7

8 As well as the central compute facili/es, we provide and look awer desktop thin- client and worksta/on computers. These thin- clients and worksta/ons run the same version of Linux as the cluster but with a graphical desktop. They are directly connected to the storage systems, so you can access all you BioHPC files as on the cluster. All of the cluster sowware modules are available, so it s very easy to develop and test your jobs on a client before submiing them to the cluster. We ll also be using these systems in future as a distributed compute resource, where jobs can be run using up to 50% of the CPU of the machine. This will harness the power of many small but idle machines for work that doesn t require the advanced features of the cluster The thin clients are /ny, but reasonably powerful desktop boxes, without usable local- storage. They are useful for accessing our systems, developing and tes/ng code etc. Worksta/ons are higher- power machines with some local storage, and a CPU which is capable of running quite large tasks in a reasonable amount of /me. The extra power makes them useful for running urgent smaller jobs locally if the cluster is busy. If you don t have one of these clients, and you would like one, speak to your PI who 8

9 A lot of different sowware is required by the groups who are members of BioHPC, and different users might want different versions etc. We use a system of modules to provide a wide range of packages. On the cluster or clients you can load modules of sowware that you need. We ll look at this in a bit too. If you need addi/onal sowware, or updated versions of exis/ng sowware then please us. If you are trying things out, and know how to, you can also install sowware into your home directory for your sole use. As always, us if you have any sowware ques/ons. 9

10 Last up we want to highlight our cloud services, web- based tools available via our portal at: portal.biohpc.swmed.edu We have a tool to generate and submit cluster jobs, tools to access a command- line or graphical session on the cluster via the web, cloud storage systems that work a bit like dropbox, and various other things. Please explore the portal, and keep checking back as we add new services. A next- genera/on sequencing pipeline, and job- submission helpers for common sowware are in development right now. 10

11 Now we ve covered what BioHPC provides, we need to look at how you can access our systems and services. We only have /me to cover the basics here, so make sure you look through the rest of the training calendar, and block of some /me to experiment and try things out. 11

12 Because of the lack of /me this isn t going to be an introduc/on to the Linux command line. There are many guides online, great books etc. to learn about using Linux. I m going to highlight par/cular things you need to know to use BioHPC, but I can t teach the command line from scratch. We ll use our web- based cloud services to access the cluster without needing a great deal of command line knowledge. There is a session this month on Wednesday 18 th that will teach some useful command line and scrip/ng topics. If you re not very experienced with Linux I strongly suggest working through the web tutorial on the slide here, trying to get some prac/ce, and then coming along to the February 18 th session as well. 12

13 So let s begin using BioHPC. We re going to take a simple piece of matlab code, which just says Hello World, put it on BioHPC storage, and run it on the cluster. We ll use the web- based tools to do all of this. To upload the file we re going to use our internal lamella web- based storage app. We actually have 2 separate systems here. lamella is for internal use, accessing any of your BioHPC storage. cloud.biohpc.swmed.edu is accessible outside of UTSW over the internet for file exchange with collaborators. IR informa/on security rules mean that the file exchange storage is separate from your main BioHPC space. We re going to use lamella now to access our BioHPC home and project space, and upload the matlab script I men/oned. ** DEMO ** 13

14 ** REFERENCE FROM DEMO ** To mount home, project storage within the lamella web system choose the Personal op/on from the user menu, then add External Storage lines as shown. If the storage is mounted correctly you will get a green circle on the lew, and it ll be visible when you go back to the lamella file browser. 14

15 Okay, via lamella our matlab script hello.m is now in the home directory, in a folder called intro. We will submit this script as a cluster job, to run on a nucleus cluster node using the web job submission tool. ** Demo ** Must add module matlab/2014b The command is: matlab nodisplay < ~/intro/hello.m This means, run matlab, without a graphical interface, and read in the file, which is in our home directory (~/) intro folder, and is called hello.m 15

16 Another way to use the cluster is to access a GUI session, running on a cluster node. The webgui tool in the portal submits a job to the cluster. When the job runs it runs a GUI session on a single cluster node, and we then connect to that session via the portal in order to use it. The GUI session is minimal there isn t a complete desktop. It s good for tes/ng code, or running jobs where a GUI is required. It s not for use as a desktop replacement. You can t browse the web, check your etc - that would be inefficient use of the very expensive cluster node. Let s use a webgui session to run our matlab code. ** DEMO ** 16

17 ** SoWware modules reference from GUI DEMO ** In a webgui session, command line session, or job script we make sowware available using the module command. 17

18 ** SoWware modules reference from GUI DEMO ** Here s an example. We heave a command line on the head node, and run module list to see sowware modules currently in- use. If we try to run the cufflinks command it fails, because the relevant module is not loaded. We can search for it with module avail cufflinks, and load it with module load cufflinks or module load cufflinks/2.1.1 if we want a specific version. Now module list shows the cufflinks modules, as well as boost which cufflinks depends on. Running the cufflinks command works this /me. 18

19 Now we ve done a web job submission, and looked at the webgui session let s spend a lirle /me at a command line session on the cluster login node. We can use the Nucleus Web Terminal service in the portal to login to the cluster login node without needing an SSH client on our computer. ** DEMO ** 19

20 ** Command Line Reference ** Useful commands related to storage usage and job scheduling. 20

21 ** Command Line Reference ** Useful commands for looking at, or edi/ng files via the command line. 21

22 ** Command Line Reference ** The sinfo command shows the status of the cluster. For each par//ons it lists the nodes which are allocated (in- use) or idle. Nodes are in more than one par//ons. All nodes are in the super par//on and at least one more specific par//on, e.g. 64GB. Look at the lines for the super par//on for an overview of how many nodes total are allocated or idle. 22

23 ** Command Line Reference ** The squeue command shows a list of jobs running or wai/ng for the cluster. This is the same informa/on shown in the job queue on the web portal. 23

24 ** Command Line Reference ** Running squeue with the l op/on squeue l` shows some addi/onal informa/on, such as the /me limit for each job. 24

25 ** Command Line Reference ** If we have submired a job, but want to cancel it we can using the scancel command with the job id, e.g. scancel

26 ** Command Line Reference ** When you re comfortable with the web job submission you can submit jobs from the command line using the sbatch command. You write a batch script (like the one you see in the web job submission tool), and then submit that script with sbatch myjob.sh where myjob.sh is the filename for your job script. 26

27 ** Command Line Reference ** Job output goes into named files, depending on the seings in the job script. By default the output file will include the job id number in its name. You can view the file directly with the cat or less commands in the terminal, or you can download the file to your PC, open it directly on a BioHPC client etc. 27

28 You may want to access BioHPC systems from home. For informa/on security reasons most of our services only work via the UTSW network, or VPN connec/on. The UTSW VPN is now more complicated to use, as it requires 2- factor authen/ca/on with a smartphone app or text message codes. See the IR website for instruc/ons on how to get connected. Here are some quick notes on the VPN specifically for Mac users a way of geing a VPN client app if the web- based access doesn t work for you. It s quite common to have problems on Macs, especially ones running newer version of OSX, due to issues with Java security on these systems. Geing the Juniper VPN client app from the IR campus file server is a way around these issues. 28

29 Right, now we re on to the last sec/on today, which is less fun, but important to make sure that you and everyone else is able to use BioHPC without problems, arguments and fric/on. All HPC clusters are crowded, shared resources. They are extremely expensive to purchase, maintain and support. Because of the special equipment in each node they cost many, many /mes the price of even a very powerful desktop PC. To make the most of the investment we have to use the systems efficiently. The BioHPC team has a difficult job to do, balancing the needs of users in different departments who want to run very different types of jobs. To make sure everything runs smoothly, and all our users get along well we need your help. 29

30 The most important thing we ask, is that if you have any ques/on, please ask us! We are here to help you get your work done as quickly and well as possible. We might not understand all of the science behind your work, but we can probably tell you how to get it done faster or berer on BioHPC. An important principle to follow when using a shared resource, is to request the least amount of resource you know you need. We only know your requirements based on the type of node, the run /me etc. you ask for. The job scheduler can only schedule things accurately if you give it good es/mated of what you need. A safety margin is fine, but don t ask for a 384GB node for 20 hrs to run a 5 minute task that could easily run on a client system. This leads on to making all reasonable arempts to use the resources efficiently. Our nodes have 32 cpu cores, so if you have many single- process jobs run them concurrently. Cancel or close jobs or sessions you don t need any more. Please arend the SLURM job scheduler training, which is going to help you a lot with this kind of stuff. Finally, try and approach HPC work like wet- lab work. Keep notes. Keep old versions of job files so we can look through changes when helping you solve a problem. 30

31 Here are the few absolute rules we have right now: Please don t run complex jobs on the login node. Long running, CPU intensive work must be submired to the cluster through SLURM. We have a limit of 16 nodes in use by a single- user at any one /me. This is enforced automa/cally. We don t allow interac/ve use of compute nodes, except via or remotegui script, or the webgui. These are limited to 20hr sessions. We don t want expensive cluster nodes idle because they are being used like a powerful desktop. Because of the privacy and security implica/ons, any arempt to login to a cluster node that is not allocated to one of your own jobs will result in your account being disabled, un/l you contact us. That s it as long as people are considerate we can keep the number of enforced rules to a minimum. 31

32 Back to friendly things we want to help you with your problems quickly, and effec/vely. Because we are computa/onal scien/sts from different backgrounds we might not understand everything about the jobs you are running. To help us out, let us know: Any and all error messages and ouput you get if you have a problem. When and where you had the problem if we know when and where we can owen find solu/ons in log files. What were you running and how? Which piece of sowware, what parameters, what files? Show us you job script etc. Anything complex or strange about your setup? Please tell us if you re happy for us to take a look at your scripts and files, and if so, where they are. If you re happy for us to look directly it can save a lot of /me. We do, of course, realize that you may be working with data that might be confiden/al for various reasons. 32

33 33