Lecture 3. Essential skills for bioinformatics: Unix/Linux

Size: px

Start display at page:

Download "Lecture 3. Essential skills for bioinformatics: Unix/Linux"

Cassandra Daniel
5 years ago
Views:

1 Lecture 3 Essential skills for bioinformatics: Unix/Linux

2 RETRIEVING DATA

3 Overview Whether downloading large sequencing datasets or accessing a web application hundreds of times to download specific files, retrieving data in bioinformatics can require special tools and skills. Downloading a large scale of sequencing data through your web browser is not feasible. Web browsers are not designed to download such large datasets. Additionally, you d need to download the sequencing data to your server, not your local workstation. To do this, you d need to SSH into your data-crunching server and download your data directly to this machine using command-line tools. We will take a look at some of these tools.

4 wget wget is useful for quickly downloading a file from the command line. For example, human gene annotation GTF file from Ensembl:

5 wget wget downloads this file to your current directory and provides a useful progress indicator. It can handle both http and ftp links. In general, FTP is preferable to HTTP for large files. For simple HTTP or FTP authentication with a user name and password, you can authenticate using --user= and -- ask-password options.

6 wget: useful options Option Values Use -A, --accept Either a suffix like.gtf or a pattern with *,?, or [ and ]. Only download files matching this criteria -R, --reject Same as with --accept Don t download files matching this -nd, --no-directory No value Don t place locally downloaded files in the same directory hierarchy as remote files. -r, --recursive No value Follow and download links on a page -np, --no-parent No value Don t move above the parent directory --limit-rate Number of bytes per second Maximum download bandwidth --user=user User name User name --ask-password No value Prompt for password -O Output file name Download file to filename specified

7 DATA INTEGRITY

8 Overview Data we download is the starting point of all future analyses. The risk of data corruption during transfers is a concern when transferring large datasets. In addition to verifying your transfer finished without error, it s also important to explicitly check the transferred data s integrity with checksums. Checksums are very compressed summaries of data, computed in a way that even if just one bit of the data is changed, the checksum will be different.

9 SHA and MD5 checksums The two most common checksum algorithms for ensuring data integrity are MD5 and SHA-1. MD5 is an older checksum algorithm, but one that is still commonly used. Both MD5 and SHA-1 behave similarly, but SHA-1 is newer and generally preferred. The long string of numbers and letters (hexadecimal format, 0-9 and a-f) is the checksum

10 SHA and MD5 checksums Checksums are completely deterministic, meaning that regardless of the time the checksum is calculated or the system used, checksum values will only differ if the input differs (even one bit of data). When downloading many files, it can get rather tedious to check each checksum individually. The program shasum can create and validate against a file containing the checksums of files.

11 SHA and MD5 checksums Then, we can use shasum s check option (-c) to validate that these files match the original versions:

12 SHA and MD5 checksums There are only possible different checksums for SHA-1 checksum. However, is a huge number so the probability of a checksum collision is very small. For the purpose of checking data integrity, the risk of a collision is negligible. Some servers use an antiquated checksum implementation such as sum or chsum. How we use these older commandline checksum programs is similar to using shasum and md5sum.

13 WORKING WITH COMPRESSED DATA

14 Overview Data compression, the process of condensing data so that it takes up less space (on disk drives, in memory, or across network transfer), is an indispensable technology in modern bioinformatics. For example, sequences from a recent Illumina HiSeq run example.fastq: 63,203,414,514 bytes (59 GB) example.fastq.gz: 21,408,674,240 bytes (20 GB) Compression ratio (uncompressed size/compressed size) of this data is 2.95, which translates to a significant space saving of about 66%.

15 Overview Data can remain compressed on the disk throughout processing and analyses. Most well-written bioinformatics tools can work natively with compressed data as input, without requiring us to decompress it to disk first. Using pipes and redirection, we can stream compressed data and write compressed files directly to the disk. Common Unix tools like cat, grep all have variants that work with compressed data. While working with large datasets in bioinformatics can be challenging, using the compression tools in Unix and software libraries make our lives much easier.

16 gzip The two most common compression systems used on Unix are gzip and bzip2. gzip faster than bzip2. bzip2 has a higher compression ratio (the previous fastq file is only about 16 GB when compressed with bzip2) Generally, gzip is used in bioinformatics to compress most sizable files, while bzip2 is more common for long-term data archiving.

17 gzip It can compress results from standard input. This is useful, as we can compress results directly from another bioinformatics program s standard output.

18 gzip It can compress files on disk in place. gzip will compress this file in place, replacing the original uncompressed version with the compressed file (appending the extension.gz to the original filename).

19 gunzip We can decompress files in place with the command gunzip. Note that this replaces tb1.fasta.gz file with the decompressed version.

20 gzip -c Both gzip and gunzip can also output their results to standard out. This can be enabled using the c option:

21 gzip with multiple files

22 gzip with multiple files

23 Working with gzipped files The greatest advantage of gzip (and bzip2) is that many Unix and bioinformatics tools can work directly with compressed files. For example, we can search compressed files using grep s analog for gzipped files, zgrep. Likewise, cat has zcat. If programs cannot handle compressed input, you can use zcat and pipe output directly to the standard input of another program.

24 Working with gzipped files

25 Creating a tar.gz archive

26 Extracting a tar.gz file

27 CASE STUDY: REPRODUCIBLY DOWNLOADING DATA

28 GRCm38 mouse reference genome We usually download genomic resources like sequence and annotation files from remote servers over the Internet, which may change in the future. Furthermore, new versions of sequence and annotation data may be released, so it is imperative that we document everything about how data was acquired for full reproducibility The human, mouse, zebrafish, and chicken genomes releases are coordinated through the Genome Reference Consortium (

29 GRCm38 mouse reference genome The GRC prefix in GRCm38 refers to the Genome Reference Consortium. We can download GRCm38 from Ensembl using wget.

30 Compare checksum values From ftp://ftp.ensembl.org/pub/release-87/fasta/mus_musculus/dna/checksums

31 Extract the FASTA headers

32 Document README Document how and when we downloaded this file in README Copy the SHA-1 checksum values into README

33 UNIX DATA TOOLS

34 Overview Understanding how to use Unix data tools in bioinformatics is not only about learning what each tool does, it is about mastering the practice of connecting tools together creating programs from Unix pipelines. By connecting data tools together with pipes, we can construct programs that parse, manipulate, and summarize data. Unix pipelines can be developed in shell scripts or as one-liners (tiny programs built by connecting Unix tools with pipes directly on the shell).

35 Overview Building more complex programs from small, modular tools capitalizes on the design and philosophy of Unix. The pipeline approach to building programs is a wellestablished tradition in Unix and bioinformatics because it is a fast way to solve problems, incredibly powerful, and adaptable to a variety of problems.

36 When to use the Unix pipeline approach The Unix one-linear approach is not appropriate for all problems. Many bioinformatics tasks are better accomplished through a custom, well-documented script. Knowing when to use a fast and simple engineering solution like a Unix pipeline and when to resort to writing a welldocumented Python and R script takes experience.

37 When to use the Unix pipeline approach Unix pipelines: Fast, low-level data manipulation toolkit to explore data, transform data between formats, and inspect data for potential problems. Useful when we want to get a quick answer and keep moving forward with our project. It is essential that everything that produces a result is documented. Storing pipelines in shell scripts is a good approach. Custom scripts using Python or R: Useful for larger, more complex tasks as these allow for the flexibility in checking input data, structuring programs, use of data structures, code documentation.

38 Inspecting and manipulating text data Many formats in bioinformatics are simple tabular plain-text files delimited by a character. The most common tabular plain-text file format used in bioinformatics is tab-delimited because most Unix tools treat tabs as delimiters by default. Tab-delimited file formats are also simple to parse with scripting language like Python and Perl, and easy to load into R.

39 Tabular plain-text data formats The basic format: Each row (known as a record) is kept on its own line Each column (known as a field) is separated by some delimiter Three formats: Tab-delimited Comma-separated Variable space-delimited

40 Tab-delimited The most commonly used in bioinformatics (e.g. BED, GTF/GFF, SA M, VCF). Columns of a tab-delimited file are separated by a single tab char acter (the escape code: \t). A common convention (not a standard) is to include metadata on the first few lines of a tab-delimited files. These metadata lines be gin with #. Tabs in data are not allowed.

41 Comma-separated values (CSV) CSV is similar to tab-delimited, except the delimiter is a comma character. While not a common occurrence in bioinformatics, it is possible that the data stored in CSV format contain commas. Some variants just do not allow this, while others use quotes around entries that could contain commas.

42 Variable space-delimited In general, tab-delimited formats and CSV are better choices than variable space-delimited formats because it is quite com mon to encounter data containing spaces.

43 How lines are separated In Linux and OS X: use a single linefeed character (the escape code: \n) to separate lines. In Windows: use a DOS-style line separator of a carriage return and a linefeed character (\r\n). To convert DOS to Unix text format, use dos2unix. To convert Unix to DOS text format, use unix2dos.

44 Inspecting data with head and tail Many files in bioinformatics are much too long to inspect with cat. Running cat on a file a million lines long would quickly fill your shell. A better option is to take a look at the top of a file with head.

45 Inspecting data with head and tail

46 Inspecting data with head and tail We can control how many lines we see.

47 Inspecting data with head and tail tail is designed to look at the end of a file. tail works just like head.

48 Inspecting data with head and tail We can also use tail to remove the header of a file. If n is given a number x preceded with a + sign (e.g. +x), tail will start from the x th line.

49 Inspecting data with head and tail head is useful for taking a peek at data resulting from a Unix pipeline. We will use grep s results as the standard input for the next program in our pipeline, but first we want to check grep s standard out to see if everything looks correct. When head exits, your shell catches this and stops the entire pipe. When building complex pipelines that process large amounts of data, this is important.

50 less less is a useful program for a inspecting files and the output of pipes. It is a terminal pager, a program that allows us to view large amounts of text in our terminals at a time. less has more features and is generally preferred than the older terminal pager called more.

51 less Shortcut Space bar b g G j k /<pattern>?<pattern> Action Next page Previous page First line Last line Down one line at a time Up one line at a time Search down for string <pattern> Search up for string <pattern>

When you run the pipe, less will capture the output of the last command

52 less less is useful in debugging our command-line pipelines. Just pipe the output of the command you want to debug to less. When you run the pipe, less will capture the output of the last command and pause so you can inspect it. less is crucial when iteratively building up a pipeline.

53 less A useful behavior of pipes is that the execution of a program with output piped to less will be paused when less has a full screen of data. When you pipe a program s output to less and inspect it, less stops reading input from the pipe. The pipe will block and we can spend as much time as needed to inspect the output.

54 Counting the number of lines By default, wc outputs the number of lines, words, and characters of the supplied file. To return the number of line, we can use option -l

55 Reporting file sizes

56 Counting the number of fields We could manually count the number of columns of the first row with head n 1. But an easier way is to use awk. awk simply prints the number of fields of the first row of the file (due to exit).

57 Counting the number of fields Because the first line of this file is a comment, we need to first chop off the comments. This solution will not work on other files since tail n +6 is tailed to this specific file.

58 Counting the number of fields A less fragile and more robust solution is

59 Working with column data with cut When working with plain-text tabular data, we often need to extract specific columns from the original file. Suppose we want to extract the columns of chromosome, start, and end position from the GTF file. The simplest way to do this is with cut. The f argument is how we specify which columns to keep. cut f 2 input.txt cut f 3-8 input.txt cut f 3,5,8 input.txt cut f 8,3,5 input.txt (no reordering)

Essential Skills for Bioinformatics: Unix/Linux

Essential Skills for Bioinformatics: Unix/Linux WORKING WITH COMPRESSED DATA Overview Data compression, the process of condensing data so that it takes up less space (on disk drives, in memory, or across