Command-Line Data Analysis INX_S17, Day 10,

Size: px

Start display at page:

Download "Command-Line Data Analysis INX_S17, Day 10,"

Kory Nash
6 years ago
Views:

Command-Line Data Analysis INX_S17, Day 10, 2017-05-01 Assignment 4 (quiz).

1 Command-Line Data Analysis INX_S17, Day 10, Assignment 4 (quiz). sort, head, tail Learning Outcome(s): Use `sort` to build filtering pipelines for bioinformatics data Matthew Peterson, OSU CGRB, matthew@cgrb.oregonstate.edu Please do not redistribute outside of OSU; contains copyrighted materials.

2 Command-line Set 1

3 Bonus! (ok, not really ) 2

4 Review: fasta_stats A Python script from the book Generates statistics about a FASTA file it's supplied./fasta_stats Usage: fasta_dna_stats <fasta_file> This script is for informational purposes only, and requires that the input file be a DNA (As, Ts, Cs, and Gs) FASTA-formatted file../fasta_stats pz_cdnas_sample.fasta # Only 2 sequences in this FASTA file html 3

5 PZ example PZ ACAAA 5 unit:attta 10 pentanucleotide 2: GC content of 37.8% (vs. AT) 3: 486 base pairs (bp) long 4: Most common 5-bp sequence is ACAAA 5: This 5-bp sequence ACAAA appears 5 times 6: Longest perfect repeat sequence is ATTTA 7: and is 10bp long 8: caused by the pentanucleotide ATTTA appearing twice (ATTTAATTTA) 4

6 fasta_stats continued 7 th column contains the length (as an integer) of the longest perfect repeat in characters (bases) 5

7 fasta_stats question Q: What sequence (in pz_cdnas.fasta) contains the longest perfect repeat (column 7)? A: Sort the output by column 7 * PZ TTTCT 5 unit:ctttccg 14 heptanucleotide PZ TTATT 11 unit:tat 15 trinucleotide PZ AGCAA 6 unit:gag 6 trinucleotide PZ4990_P TTATA 5 unit:attata 12 hexanucleotide PZ AAAAA 11 unit:taat 8 tetranucleotide PZ TTTTT 19 unit:taatttt 14 heptanucleotide PZ ATATT 8 unit:gtata 10 pentanucleotide PZ TTTTT 18 unit:attt 8 tetranucleotide PZ TTTAA 6 unit:gct 6 trinucleotide * Filter the (#) header of fasta_stats : grep v # or grep 'unit:' 6

8 sort <file> or sort Lots of options! No options? It sorts all columns. By default sort sorts by dictionary order. Uses whitespace as column separator. 7

9 sort by key (column) sort by columns 2-4 ("conglomerated"), in dictionary order, and in reverse. ast_lines.html 8

10 sort Break ties by Break ties by sorting 1 st column in dictionary order. 9

11 sort Break further ties by Break further ties by sorting by 5 st column in numeric order. numeric is faster than general numeric, which can handle scientific notation (1e-6) 10

12 sort -unique Optional unique flag specifies if there are still ties to only show the first row. 11

13 fasta_stats example sort./fasta_stats pz_cdnas.fasta \ grep v '#' \ sort k7,7nr \ less S Sort on longest perfect repeat in column 7 in reverse, numeric order 12

14 fasta_stats sort output Longest perfect repeat is 94 base pairs Several ties of 18 base pair sequences 13

15 fasta_stats sort unique bp Show one sequence per base pair length: sort k7,7nr u 14

16 fasta_stats sort GC content Sort by GC content sort k7,7nr k2,2g 15

17 Pipelining (chaining) sorts together Sort by highest GC content (column 2) and then Re-sort by unique repeat length (column 7)./fasta_stats pz_cdnas.fasta \ grep v '#' \ sort k7,7nr k2,2gr \ sort k7,7nr u \ less S 16

18 Sort GC content, unique repeat length -s stable guarantees our 2 nd sort will not "undo" the the GC content sorting of our 1 st sort cat x sort k2,2 x sort s -k2,2 x D 2 A 2 D 2 B 2 B 2 B 2 A 2 D 2 A 2 See also: 17

19 Extract first or last lines head n <number> <file> tail n <number> <file> or head n <number> tail n <number> 18

20 First (top) 3 lines of our sort via head./fasta_stats pz_cdnas.fasta \ grep v '#' \ sort k7,7nr k2,2gr \ sort k7,7nr u \ head n3 19

21 Offset in tail Start printing with second row on (skip our top hit)./fasta_stats pz_cdnas.fasta \ grep v '#' \ sort k7,7nr k2,2gr \ sort k7,7nr u \ tail n +2 20

22 tail f Follow Constantly watch output appended to a file When would we use this? e.g., You have a BLAST job (submitted via SGE) that you know is going to take days to complete. You know that it's writing its stdout to a file into the log directory you specified (via SGE_Batch) You can watch the file's stdout "grow" (be appended) in real-time, e.g., tail -f MYJOB.o Ctrl-c to quit 21

23 sort head tail Command / Concept Review 22

Introduction to Unix/Linux INX_S17, Day 8,

Introduction to Unix/Linux INX_S17, Day 8, 2017-04-21 stdin, stdout, stderr, piping, iterative filtering, grep, cat, UUOC Learning Outcome(s): Redirect the standard output to the standard input stream