Command-Line Data Analysis INX_S17, Day 15,

Size: px

Start display at page:

Download "Command-Line Data Analysis INX_S17, Day 15,"

Madeleine Jefferson
5 years ago
Views:

Command-Line Data Analysis INX_S17, Day 15, 2017-05-12 General tool efficiency, tr, newlines, join, column Learning

1 Command-Line Data Analysis INX_S17, Day 15, General tool efficiency, tr, newlines, join, column Learning Outcome(s): Discuss the theory behind Unix/Linux tool efficiency, e.g., the reasons behind their design from the 1970s and 1980s, and identify these design characteristics in common tools, e.g., cat, grep, awk, sed, and pipes. Matthew Peterson, OSU CGRB, Please do not redistribute outside of OSU; contains copyrighted materials.

2 A note on efficiency The programs we use were written in the 1970s, e.g., sed, awk, sort, etc. Brilliant minds made sure these programs were: efficient, lean, and mean! Why? 1

3 Cray-1 Supercomputer A top of the line 1976 Supercomputer processor ran at 80 Mhz at a cost of $8.86 million. Author: Clemens PFEIFFER 2

4 Cray-2 Supercomputer A top of the line 1985 Supercomputer processor ran at 125 Mhz at a cost of $16 million. 3

Apple Watch An Apple Watch 1 (~$269) running at 520 Mhz is equal in processing capacity to two Cray 2s. = + http://www.dailymail.co.

5 Apple Watch An Apple Watch 1 (~$269) running at 520 Mhz is equal in processing capacity to two Cray 2s. = Author: Justin14 4

6 Pipe example Write Buffer Read cat fruit.txt grep apple Pipe sets up a small buffer in-between programs When the buffer is full it pauses cat When the buffer is empty it pauses grep If the buffer is ½ full both cat and grep can run simultaneously on different CPUs! 5

7 top example If you re running a big pipeline you ll see multiple programs will be running at the sametime, e.g., Program CPU cat 30% grep 20% awk Show zcat example of Illumina FASTQ 6

8 Class activity: Counts of counts!

9 Class activity: The fruit pipeline

10 Well done! Observations Most of the participants needed very little "working memory" Not much storage space was used by the buffers Many participants could be working simultaneously, but Anyone after sort would have had to wait! 9

11 tr Translates sets of characters tr '<set1>' '<set2>' <file> or... tr '<set1>' '<set2> Example: Change al T s to A s and vice-versa echo "ATTTTTA" tr 'TA' 'AT' TAAAAAT 10

12 tr Dealing with newlines \n The newline is represented in Linux by a special character '\n' You cannot see this character other than it wrapping the text to the next, new line. We can use tr to remove all newlines, e.g., cat lines.txt Line 1 Line 2 Line 3 cat lines.txt tr \n # 11

13 tr To reflow a FASTA file FASTA files often have their sequences span lines after each header, e.g., pz_cdnas.fasta Say we wanted to extract all sequences from the pz_cdnas.fasta that had in its header line (>) an nreads greater than 5. There s a general pattern ( trick ) for this using tr, which can be applied in other use cases. 12

14 tr reflow pattern 1) Check the file for an unused character, e.g., or the tab character ('\t'); verify first with grep! 2) Replace all newlines e.g., tr '\n' '@' 3) Replace start of records ('>') with '\n>' sed r 's/>/\n>/g' # 1 seq/line 13

15 tr reflow pattern (continued) 4) Use grep/awk/sed to modify lines of interest sed -r 's/=/ /1' \ awk '{if($3 > 5) {print $0}}' 5) Reformat back to FASTA (Header line + Seq line) tr '@' '\n' # Could also use sed 6) The previous reformatting will have introduced blank lines, which can be removed, e.g., awk '{if(nf > 0) print $0} 14

16 join files on a common column join -1 <key column in file1> -2 <key column in file2> <file1> <file2> Say we have two output files that share a similar key (identifier), e.g., # blastx pz_cdnas.fasta vs. Yeast ORF, outfmt 6 p450s_blastp_yeast_top1.txt # fasta_stats of pz_cdnas.fasta pz_stats.table 15

17 join files on a common column join -1 <key column in file1> -2 <key column in file2> <file1> <file2> Merges two row/column text files based on lines with similar values in a specified key column. By default, join only outputs rows where data is present in both files. Both input files are required to be similarly sorted (either ascending or descending) on the key columns. 16

18 join BLAST and fasta_stats data Say we have two output files that share a similar key (identifier) in this case the sequence identifier # blastx pz_cdnas.fasta vs. Yeast ORF, outfmt 6 p450s_blastp_yeast_top1.txt # fasta_stats of pz_cdnas.fasta pz_stats.table 17

19 join First sort the files Say we have two output files that share a similar key (identifier) in this case the sequence identifier cat pz_blastx_yeast_top1.txt \ > sort -k1,1d > \ > pz_blastx_yeast_top1.sorted.txt cat pz_stats.table \ > sort -k1,1d > \ > pz_stats.sorted.table 18

20 The output contains: join Sorted output All of the columns for the first file, followed by All of the columns of the second file Without the key column Separated by single spaces join \ pz_stats.sorted.table \ pz_blastx_yeast_top1.sorted.txt 19

21 BLAST matches > 50% GC content awk '{if($1 > 0.5) print $0}' With everything separated by spaces it's still hard to read (for humans) 20

22 join Limitation #1 If any entries in the key columns are repeated, the output will contain a row for each matching pair of keys. 21

23 join Limitation #2 Both input files must be sorted. If they are not sorted join will produce a warning. 22

24 join Limitation #3 join does not easily produce a Full outer join By default it produces an Inner join Python and R excel at these use cases 23

25 column Reformatting column -t <file> or... column -t Reformats whitespace separated row/column input so that the output is human readable Done by replacing one or more spaces and tabs by an appropriate number of spaces so that columns are visually aligned. 24

Command-Line Data Analysis INX_S17, Day 10,

Command-Line Data Analysis INX_S17, Day 10, 2017-05-01 Assignment 4 (quiz). sort, head, tail Learning Outcome(s): Use `sort` to build filtering pipelines for bioinformatics data Matthew Peterson, OSU CGRB,