Computational Skills Primer. Lecture 2

Transcription

1 Computational Skills Primer Lecture 2

2 Your Background Who has used SCC before? Who has worked on any other cluster? Do you have previous experience working with basic linux and command line usage (CLI)? Who has gone through the tutorial assigned on basic linux and command line usage?

3 Computer was born in the mind of man, not the other way around!! Goal of this lecture: - Overcome the fear of black screen (if you have one!!) - Learn techniques for working on SCC which you will need for your upcoming projects. - Unleash the power of shared computing and learn to use it efficiently.

4 Prerequisites Patience with self and with your group mates Keep an open mind It s more about learning and less about grades. Attitude of collaboration It s OK to not know - we can learn together Rome ne s'est pas faite en un jour!!!

5 What is SCC? Shared Computing Cluster (SCC) Shared: Multi-user, Multi-tasking environment. Computing: Interactive jobs, Single processor and parallel jobs, Graphics job etc. Cluster: Nexus of computers connected by a fast local area network which coordinated the computational workload via job scheduler

6 Massachusetts Green High Performance Computing Center

7 A computer cluster and node A computer cluster is a set of loosely or tightly connected computers that work together so that they can be viewed as a single system. Computer clusters have each node set to perform the same task, controlled and scheduled by software. The components of a cluster are usually connected to each other through fast local area networks, with each node (computer used as a server) running its own instance of an operating system.

8 SCC Architecture

9 Why use SCC when we can run jobs on our local system?? Collaborate on projects Run code that exceeds workstation capability Secured Network Fast and easy data share Access restricted data like (dbgap) Run code that runs for long periods of time (days, weeks, months) Run code in highly parallelized formats (use 100 machines simultaneously).

10 Software to Access SCC

11 How to connect with SCC

12 SCC Part I: Navigating through files Essential navigation commands: pwd ls cd print current directory list files change directory We use pathnames to refer to files and directories in the Linux file system. There are two types of pathnames: Absolute the full path to a directory or file; begins with / Relative a partial path that is relative to the current working directory; does not begin with / Special characters interpreted by the shell for filename expansion: ~... *? TAB your home directory current directory parent directory wildcard matching any filename wildcard matching any character try to complete (partially typed) file or directory name

13 List of useful commands - Part I Useful options for the ls command: ls -a List all files, including hidden files beginning with a period. ls -ld * List details about a directory and not its contents ls -F Put an indicator character at the end of each name ls l Simple long listing ls lr Recursive long listing ls lh Give human readable file sizes ls ls Sort files by file size ls lt Sort files by modification time (very useful!)

14 List of useful commands - Part II cp [file1] [file2] copy file mkdir [name] make directory rmdir [name] remove (empty) directory mv [file] [destination] move/rename file rm [file] remove (-r for recursive) file [file] identify file type less [file] page through file head -n [file] display first n lines tail -n [file] display last n lines ln s [file] [new] create symbolic link cat [file] [file2 ] display file(s) tac [file] [file2 ] display file in reverse order

15 Word Count Count everything ~]$ wc ncrna_pfam.output ncrna_pfam.output Count lines ~]$ wc -l ncrna_pfam.output ncrna_pfam.output Count words ~]$ wc -w ncrna_pfam.output ncrna_pfam.output

16 Needle in the haystack Find command can be used to locate a file or directory using following options: find. name my-file.txt # search for my-file.txt in. find ~ -name bu type d # search for bu directories in ~ find ~ -name *.txt # search for *.txt in ~ find./directory from current -name.*jpg #search for all jpg file in directory path from current directory

17 Hands-on Terminal Session I Access your home directory and create a directory named work. Copy all the DiffExp*.txt files from /project/bf528/kkarri/ to your work directory Rename the file names as file1.txt, file2.txt and so on.. Count the number of lines in all these files. There is a hidden R script file (.R extension) in /project/bf528/ Find the file and copy it to your work directory. Rename the file to pearson_script.r

18 SCC Part II: Working and Managing Files and Directories File Editors Vim : A better version of vi (an early full-screen editor). Nano: Gedit: Notepad-like editor with some programming features. Requires Xwindows. Advantages of Vim and Nano Nano: Vim: Very powerful editor Easy to use and master. Session recovery Only includes basic text editing Split screen Tab expansion Search function Completion commands Search and replace Syntax coloring "Goto line" command May be challenging for beginners Automatic indentation functions

19 Permissions Files Access Control: Every file has an owner. Every file belongs to a group. Every file has permissions controlling access to it. [kkarri@scc4 ~]$ ls -l newdir drwxr-xr-x 3 kkarri waxmanlab 512 Jan 21 16:03 newdir

20 chmod Change the permissions on the directory newdir so that members of your group can write to it: ~]$ ls -l newdir drwxr-xr-- 3 kkarri waxmanlab 512 Jan 21 16:03 newdir [kkarri@scc4 ~]$ chmod g+w newdir [kkarri@scc4 ~]$ ls -l newdir drwxrwxr-- 3 kkarri waxmanlab 512 Jan 21 16:03 newdir

21 Decoding chmod The chmod command also works with the following mappings, readable=4, writable=2, executable=1, which are combined like so: ~]$ ls l newdir drwxrwxr-x 3 kkarri waxmanlab 512 Jan 21 16:03 newdir [kkarri@scc4 ~]$ chmod 750 newdir [kkarri@scc4 ~]$ ls -l newdir drwxr-x--- 3 kkarri waxmanlab 512

22 Compressing and decompressing files tar (Tape ARchiver) : To create a disk file tar archive. Here are the options we are using: -z: Write the archive through gzip -c: Create a new tar archive -v: Verbose, show the files being worked on as tar is running -f: Specify the name of an archive file $ tar -zcvf moe.tar.gz /home/moe To restore files from a tar archive, use $ tar -zxvf archivename gzip is a utility for compressing and decompressing individual files. To compress files, use: $ gzip filename The filename will be deleted and replaced by a compressed file called filename.z or filename.gz. To reverse the compression process, use: $ gzip -d filename or $ gunzip filename viewing compressed text files with zcat $ zcat genelist.gz, $ zcat genelist.gz head

23 Executing a script Shell Script : sh script_name.sh Rscript : Rscript script_name.r Python : python script_name.py

24 Hands-on Terminal Session II Open the pearson_script.r and try to edit the script. Can you edit the file? What is the permission for your R script? Change the permission for user to be able to write and execute. In each of your text files (.txt), substitute Con with Control and save the changes. Execute your pearson_script.r Create a pdf folder and copy all the pdf files (*.pdf) and compress them as.tar.gz

25 Storage (GB) In general Home Directory Personal files, custom scripts. /project Source code, files you can t replace. /projectnb Output files, downloaded data sets. Large quantities of data that you could recreate in the incredibly unlikely event of a disastrous data loss. Available from all head nodes (scc1, scc2, etc) Restricted data (dbgap) /restricted/project/projname backed up space for dbgap data /restricted/projectnb/projname not backed up space for dbgap data Only accessible through scc4.bu.edu and compute nodes.

26 Scratch Space Each node (login or compute) has a directory called /scratch stored on a local hard drive. This can be used by batch jobs to quickly write temporary files. If you wish to keep these files, you should copy them to your own space when the job completes. Scratch files are kept for 30 days, with no guarantees.

27 SCC Part III: Environment configuration and executing jobs Modules Used to load applications not automatically loaded by the system, including alternative versions of applications. - Check the available modules [kkarri@scc4 newdir]$ module avail R - Load a module in current environment [kkarri@scc4 newdir]$ module load R/ Unload a module [kkarri@scc4 newdir]$ module unload R/3.4.0 To check the version of a tool or software [kkarri@scc4 newdir]$ which R

28 Running Jobs A job is a unit of computation, e.g. execute a single program Three types of jobs: Interactive job running interactive shell: run GUI applications, code debugging, benchmarking of serial and parallel code performance; Interactive graphics job - for running interactive software with advanced graphics, e.g. windows and buttons Batch job job command specified in a script and run on a cluster node with no user interaction Most of your jobs will be batch jobs

29

30 Batch Jobs qsub and qstat Use the Open Grid Scheduler (OGS) command qsub to submit the compiled program to the batch system: stranded]$ qsub -P waxmanlab stranded_transcriptome.qsub NB: -P <project name> is a required argument! Check the status of your job with qstat [kkarri@scc4 stranded]$ qstat -u kkarri job-id prior name user state submit/start at queue slots ja-task-id QLOGIN kkarri r 01/20/ :23:05 linga@scc-ka8.scc.bu.edu new_cuffme kkarri r 01/21/ :09:13 mem512@scc-wj3.scc.bu.edu 28

31 Understanding qstat output

32

33 Signal Kill (9) error What happens if you use more slots than requested? We kill it to preserve other jobs running on that node. If you have notifications enabled, you will receive a notice that the job was aborted. Note that it ran for 9 minutes and the CPU ran for 22. You will also receive an explanation .

34 Customizing parameters based on your job requirement More information available on:

35 CPU Parallelization OpenMP: Single node using multiple processes OpenMP: Single node threading a single process Common with scripts when the user only wants a single job. Commonly built into applications. OpenMPI: Multi-node, many CPU, shared memory processing Very powerful computation, not used much on BUMC. More information available on:

36 MPI Jobs

37

38 Delete single or multiple jobs Using qdel command and Job id you can request to delete a job [kkarri@scc4 newdir]$ qdel kkarri has deleted job Delete all of your running and queued jobs [kkarri@scc4 newdir]$ qdel -u kkarri kkarri has deleted job kkarri has deleted job

39 qsh interactive session Request an interactive session using qsh stranded]$ qsh -P waxmanlab Your job ("INTERACTIVE") has been submitted waiting for interactive job to be scheduled Request an interactive session using qlogin #asking for 16 cores stranded]$ qlogin -P waxmanlab -pe omp 16 -l h_rt=12:00:00 More requested cores, more time to get access to the session!!!!

40

41 Virtual Private Network Boston University s Virtual Private Network (VPN) creates a tunnel between your computer and the campus network that encrypts your transmissions to BU. Use of the VPN also identifies you as a member of the Boston University community when you are not connected directly to the campus network, allowing you access to restricted networked resources. Gain access to restricted resources when you are away from BU, including departmental servers (such as printers and shared drives). Protect data being sent across the Internet through VPN encryption, including sensitive information such as your BU login name and Kerberos password. Increase security when connecting to the Internet through an open wireless network (such as in a cafe or at the airport) by using the BU VPN software.

42 Hands-on Terminal Session III fastqc A quality control tool for high throughput sequence data(will discuss in detail in coming lectures) The input for this tool is a.fastq.gz file and the command to run is fastqc name.fastq.gz Copy the test.qsub script from /project/bf528/kkarri Check the availability of module fastqc Open the script in vim or gedit and edit the script by specifying incomplete parameters ( In CAPITALS) Add the fastqc command using the SRR _R1.fastq.gz file located in /project/bf528/kkarri folder (hint: use pwd to get the file path) Submit test.qsub as batch job using the project bf528 and check the status of your job.

43 Discussion Questions For the following jobs, what according to you would be a suitable mode of job run on scc- an interactive session (qsh,qlogin) or batch job (qsub) Alignment of ~50 millions raw sequencing reads to a large reference genome. Run a compute process > 15 min Run a job > 12hrs

44 Additional Reading For in-depth understanding of these concepts go through the following modules on cluster computing and advance command line text editors: 06_cluster_computing/06_cluster_computing.html 03_advanced_cli/03_advanced_cli.html