Week January 27 January. From last week Arrays. Reading for this week Hashes. Files. 24 H: Hour 4 PP Ch 6:29-34, Ch7:51-52

Transcription

1 Week 3 23 January 27 January From last week Arrays 24 H: Hour 4 PP Ch 6:29-34, Ch7:51-52 Reading for this week Hashes 24 H: Hour 7 PP Ch 6:34-37 Files 24 H: Hour 5 PP Ch 19: Biol Practical Biocomputing 1

2 Week 3 Homework 1 # # HW1 # # For a fastq file on standard input # count and report the number of reads # calculate and report the average length of the sequences # # there are four lines for each sequence in a fastq file: # title line # sequence # separator # quality line # # Michael Gribskov 14 january 2016 # # read the file line-by-line count the number of reads and total length $line_num = 0; while ( $line = <> ) { # remove the newline character chomp $line; if ( $line_num % 4 == 1 ) { # the second line is the sequence line #print "$line_num: $line:\n"; $sum_len += length($line); $seq_count++; $line_num++; $ave_len = $sum_len / $seq_count; print "there are $seq_count reads\n"; print "average read length is $ave_len\n"; Biol Practical Biocomputing 2

3 Week 3 Homework 2 Using the same FastQ file as we used in homework 1 tabulate the percentage of A, C, G and T bases versus the position in the sequence. The logical way to use this is to save the count of each base in an array where the array index is the position in the sequence. Following the base output, write a histogram of the base qualities. Base quality is shown in the 4th line of each entry and is encoded as follows Q = chr( 33 + (-10 * log10(error)) ) where chr is a perl function that converts an integer to the letter with the corresponding ascii code, the ord function does the opposite. for the histogram we can simple use the ascii values, i.e. ord($quality_letter) Biol Practical Biocomputing 3

4 Week 3 Homework 2 #Positional base content #A C G T AT GC #quality histogram Biol Practical Biocomputing 4

5 Arrays - Quick review Identified by and $a are completely different (they have different name is only used for multiple values (whole array) $name[$element] is used for single values An ordered set of scalars Initialized using parentheses with the elements separated by commas # Define nucleotide = ( "A", "C", "G", "T" ); foreach $nuc ) { print "$nuc is a valid base\n"; print the first base is $base[0]\n ; Biol Practical Biocomputing 5

6 Hashes Also called associative arrays Hashes are an array indexed by a string rather than a number Hashes are a set of key/value pairs When do you use hashes? When the natural index is a string, such as sequence id base or amino acid Hashes are declared using % and () Hashes elements are accessed using {, e.g. $residue{ala Notice the importance of [ ] array element, vs { hash element, vs ( ) list delimiter Biol Practical Biocomputing 6

7 Hashes Collections of key/value pairs an array whose index is a word # filling a hash, three different ways %student = ( name => Derren, status => Sophomore, gpa => 4.7 ); %student = ( name, Derren, status, Sophomore, gpa, 4.7 ); $student{name = 'Derren'; $student{status = 'Sophomore'; $student{gpa = 4.7; # iterating over hash foreach $attribute ( keys %student ) { print "$attribute: $student{$attribute\n"; Biol Practical Biocomputing 7

8 Hashes Defining hashes Two syntaxes for hashes "," and "=>" %student = ( name => Derren, status => Sophomore, gpa => -7 ); %student = ( name, Derren, status, Sophomore, gpa, -7 ); As with arrays, use % to access the whole hash, $hash_name{key to access the individual elements Under the hood, a hash is just a array in which the elements are paired up as keys and values %residue = ( ala => A, cys cys, C, C asp => D ); ala foreach $k ( %residue ) { A print "$k\n"; asp D Biol Practical Biocomputing 8

9 Hashes Why use hashes? Index is text (string) more natural for named entities than having two parallel arrays, one with the name, one with the information Sparse arrays where many values are absent or = qw/ala cys asp = ( 89.0, 121.0, 133.0, ); %aa_mw = { ala => 89.0, cys => 121.0, asp => 133.0, ; Elements in a hash are arranged for fast lookup, not the order you enter them (unlike an array) Features of hashes There are special functions for ordered access to hashes keys: produces a list of just the keys (heavily used) values: produces a list of just the values (not used much) each: produces a list of name value pairs (hardly ever used) Entire arrays can be copied %working = %original; Hash slices can be used, more complicated than for arrays Biol Practical Biocomputing 9

10 Hashes Iterating over hashes - TMTOWTDI foreach $key ( keys %x ) { Block of code foreach $value ( values %x ) { Block of code while ( ($key, $value) = each %x ) { Block of code %x = ( 1 => "a", 2 => "b" = keys %x; = = values %x ; = ("a","b") Biol Practical Biocomputing 10

11 Hashes and Arrays Minor Arcana When array assignments overflow or underflow, the leftover part is $value # same as $value = $data[3]; = ( 3, 6, 8, 2 ); $value # $value == 4; number of array elements %letter = ( a=>1, b=>2, c=>3 = %letter; foreach $l ) { print "b:$l\n"; letter:c letter:3 letter:a letter:1 letter:b letter:2 Biol Practical Biocomputing 11

12 Input/Output Standard filehandles names are derived from C Must be capitalized, no prefix character (sigil) Standard input: STDIN, default is keyboard if no filehandle is supplied for input, the default is STDIN <> (functionally, the same as <STDIN>) returns the string that is read when the input stream ends, it returns the empty string (false) Standard output: STDOUT, default is terminal display STDOUT is buffered, i.e., saved up until a certain amount is present if no filehandle is supplied, the default output is STDOUT print "test\n" is the same as print STDOUT "test\n"; Standard error: STDERR STDERR is also generally the terminal, but STDERR messages are immediate not buffered As it sounds, STDERR is often used for error messages Also used for printing to terminal when output is going to a file Biol Practical Biocomputing 12

13 Input/Output Reading from (multiple) files If a file name, or list of file names, follows the name of the program, it will be used as standard input rather than the terminal a.txt This is a test file three lines of txt copy_text.pl while ( $line = <> ) { print "$line"; %copytext.pl a.txt %copytext.pl a.txt b.txt c.txt %copytext.pl *.txt (may not work on all systems, or with very long lists) Biol Practical Biocomputing 13

14 Input/Output Reading/writing files using redirection Standard input, STDIN, is the keyboard Change at command line by using input redirect operator (<) Standard output, STDOUT, goes to the terminal Change at command line using the output redirect (>) operator %copytext.pl <a.txt >output.txt %copytext.pl a.txt b.txt c.txt >analysis.dat Biol Practical Biocomputing 14

15 Input/Output Recommended Reading and printing to/from named files Filehandle A symbol that identifies a file Filehandles explicity identify input and output streams, usually files Filehandles have names all in capitals with no symbol preceding Predefined: STDIN, STDOUT, STDERR Usually you will use an indirect filehandle, a scalar variable ($filehandle) Three parameter file open Open files with open( $filehandle, mode, $filename ) You should test whether a filehandle opens correctly open returns true if successful Close with close( $filehandle ); sends all pending output and closes file If you omit close, all filehandles close when the program terminates Biol Practical Biocomputing 15

16 Input/Output Recommended Mode indicates whether you want to read or write to the file read, mode = < write, mode = > append, mode = >> # test for file opening preferred method # die is a system function that terminates the script and prints the # following string (if present) $filename = 'seq.fa'; open ( $in, "<", $filename ) die "$filename cannot be opened\n"; open ( $out, ">", $filename ) or die "$filename cannot be opened\n"; # mode == read # mode == write $line = <$in>; print $out $line; close $in; Biol Practical Biocomputing 16

17 Input/Output old style (deprecated) Open with open( FILEHANDLE, "<$filename" ); Filehandles have names all in capitals with no sigil preceding Predefined: STDIN, STDOUT, STDERR (predefined) Close with close( FILEHANDLE ); sends all pending output and closes file If you omit close, all filehandles close when the program terminates # test for file opening preferred method # die is a system function that terminates the script and prints the # following string (if present) $filename = 'seq.fa'; open ( IN, "< $filename" ) die "$file cannot be opened\n"; open ( OUT, "> $filename" ) or die "$file cannot be opened\n"; # mode == read # mode == write $line = <IN>; print OUT $line; close IN; Biol Practical Biocomputing 17

18 Input/Output Opening files Files must be opened for reading, writing, or appending Files should be closed, but usually it won't matter # open for output open( $outgoing, ">", "filename.txt" ) die "filename.txt cannot be opened\n"; print $outgoing "foo\n"; close( $outgoing ); # open for append open( $add, ">>", "filename.txt" ) die "filename.txt cannot be opened\n"; print $add "foo2\n"; close( $add ); # open for input open( $incoming, "<filename.txt" ) die "filename.txt cannot be opened\n"; while ( $foo = < $incoming > ) { print "=>$foo\n"; close( $incoming ); Biol Practical Biocomputing 18

19 Input/Output Reading from filehandles use the input operator with the filehandle $filehandle Writing to filehandles insert the filehandle after print, print $filehandle "stuff\n" # add line numbers to each line of a file open( $in, "<", "filename.txt" ) die "filename.txt cannot be opened\n"; open( $out, ">", "new.txt" ) die "new.txt cannot be opened\n"; $line_no = 0; while ( $line = <$in> ) { $line_no++; print $out "$line_no $line"; close $in; close $out; Biol Practical Biocomputing 19

20 Input/Output Special functions for files eof: end of file, returns true after the last line is read particularly useful when reading a series of files on STDIN $file_num = 1; $line_num = 0; while ( $line=<> ) { chomp $line; $line_num++; print " ($file_num,$line_num) $line\n"; if ( eof ) { print "end of file\n"; file1.txt $file_num++; file1=a $line_num = 0; file1=b file2.txt file2,first file2,second % eof.pl file1.txt file2.txt (1,1) file1=a (1,2) file1=b end of file (2,1) file2,first (2,2) file2,second end of file Biol Practical Biocomputing 20

21 Input/Output Using eof (end of file) eof is true when you have just read the last line of the file, even if it is not the last line of the input stream in most cases you don't need eof because <FILE> returns false when you reach the end of file when reading multiple files on standard input this is not true, <> is true until the end of the last file #counts the total number of lines in all files $line_count = 0; while ( $line = <> ) { $line_count++; print "$line_count\n"; # counts the number of lines in each file $file_no = 0; $line_count = 0; while ( $line = <> ) { $line_count++; if ( eof ) { $file_no++; print "file number:$file_no lines:$line_count\n"; $line_count = 0; Biol Practical Biocomputing 21

22 Input/Output Special functions for files File test operators (a selection), true means -r file is readable -w file is writable -f file is a plain file (i.e., a text file) -d file is a directory -M age of file (since modification), time in days -A age of file (since accessed ), time in = ( "a.txt", "b.txt", "c.txt" ); foreach $file ) { next unless M $file > 0.5; # files older than 12 hours Biol Practical Biocomputing 22

23 Getting rid of newlines Every line you read is ended by a newline (\n, carriage return) When you split the line, the last item will have the newline Function chomp removes the last character of a string if and only if it is a newline Newline may be different on different hardware while ( $line = <> ) { chomp = split " ", $line; Biol Practical Biocomputing 23

24 Input/Output An entire file can be read into a array at once, this is called slurping Slurping can be done with any filehandle Each line of the file is one element of the array Advantages: fast and simple Disadvantages: entire contents is stored in memory large files may be too = <>; # note the difference from $content = <> foreach $line ) { print "$line"; # same with = <$sequence>; foreach $line ) { print "$line"; Biol Practical Biocomputing 24

25 Input/Output Sorting a file Arthur, Chester A Adams, John Q Buchanan, Jean Williams, Andrew Jackson, Annette $filename = "student.txt"; open( $infile, "<", $filename ) die "Unable to open input file $filename\n"; $filename2 = "student2.txt"; open( $outfile, ">", $filename2 ) die "Unable to open file = <$infile>; # reads all of = ); close $infile; foreach $line ) { print $outfile $line; close $outfile; Biol Practical Biocomputing 25

26 Text Processing Split and Join functions split: break a string into pieces, converts string to list split( pattern, expression ); split "pattern" $string; split( pattern, expression, limit ); split "pattern" $string, 2; the split pattern is removed from string (more about patterns later) join: connect elements of list into a string, converts list to string join( expression, list ); $text = "But soft,\n what light through \t yonder window\n"; print "starting string: $text\n"; # split on white = split " ",$text; $wordcount = 0; foreach $word ) { $wordcount++; print "$wordcount: /$word/\n"; $new_string = join " print "string after joining: $new_string\n"; print "\n$wordcount words found\n\n"; starting string: But soft, what light through yonder window 1: /But/ 2: /soft,/ 3: /what/ 4: /light/ 5: /through/ 6: /yonder/ 7: /window/ string after joining: But soft, what light through yonder window 7 words found Biol Practical Biocomputing 26

27 Text Processing Split can be used to get columns of data ideal for tab-delimited or csv files from ms-excel, GFF files in genomics Arthur Adams Buchanan Williams Jackson while ( $line = <> ) { ( $name, $score1, $score2 ) = split " ", $line; print "score 1: $score1 score 2: $score2 $name\n"; while ( $line = <> ) { ( $name, $score1, $score2 ) = split " ", $line; $test1{ $name = $score1; $test2{ $name = $score2; foreach $name ( sort keys %test1 ) { print "name: $name $test1{$name $test2{$name; Biol Practical Biocomputing 27

28 More splitting use split to break up a sequence into an array of letters $seq = = split "", $seq; # nothing between the double quotes foreach $base ) { use split with a limit to break apart the first line of a FASTA or FASTQ formatted sequence >CPK2 calcium dependent protein kinase 2 2:N:0:GTAGAG AGGCCATGAGGTTCCCCAGAAGGAAAGGTCCGGCCGGACCAGTACTCGCGATGAGGCGGACCGGC ( $name, $doc ) = split " ", $line, 2; # split into two parts at first space Biol Practical Biocomputing 28

29 Unique IDs A very common problem is to identify a set of unique IDs Homework 3 asks you to identify transcripts with the same ID but different isoform numbers The solution relies on the fact that the keys in hashes must be unique, and a special function, defined, that tells whether a scal value has been defined (defined $scalar_value) is true if the variable has been defined using my, or if it has been assigned a value Hash elements are scalar values, they are defined if they have been assigned a value (defined $hash{key ) is true if the hashkey key is defined Biol Practical Biocomputing 29

30 Practical Steps to Writing a Program Figure out what you want to do Write it as a series of comments Take one step at a time, testing while you go Focus on the loops first test to see the loops work Then add the logical tests test to see your logic is correct Only then add the "meat" Biol Practical Biocomputing 30

31 GFF3 Genome annotation file 1. Sequence 2. Source 3. Feature 4. Begin 5. End 6. Score 7. Strand 8. Frame 9. Comment 4. repeat_region ?. description=dust 4. repeat_region ?. description=dust 4 ensembl protein_coding_gene ID=AT4G00060;description=Nucleotidyltransferase family protein;external_name=mee44; 4 ensembl transcript ID=AT4G ;Parent=AT4G00060;biotype=protein_coding;logic_name=tair 4. CDS ID=AT4G ;Parent=AT4G ;rank=1 4. exon ID=AT4G00060-E.1;Parent=AT4G ;constitutive=1;rank=1 4. CDS ID=AT4G ;Parent=AT4G ;rank=2 4. exon ID=AT4G00060-E.2;Parent=AT4G ;constitutive=1;rank=2 4. CDS ID=AT4G ;Parent=AT4G ;rank=3 4. exon ID=AT4G00060-E.3;Parent=AT4G ;constitutive=1;rank= CDS ID=AT4G ;Parent=AT4G ;rank=15 4. exon ID=AT4G00060-E.15;Parent=AT4G ;constitutive=1;ensembl_end_phase=-1;rank=15 4. exon ID=AT4G00060-E.16;Parent=AT4G ;constitutive=1;ensembl_end_phase=-1;ensembl_phase=-1;rank=16 4. exon ID=AT4G00060-E.17;Parent=AT4G ;constitutive=1;ensembl_end_phase=-1;ensembl_phase=-1;rank=17 4. exon ID=AT4G00060-E.18;Parent=AT4G ;constitutive=1;ensembl_end_phase=-1;ensembl_phase=-1;rank=18 4 ensembl protein_coding_gene ID=AT4G00070; description=ring/u-box superfamily protein; 4 ensembl transcript ID=AT4G ;Parent=AT4G exon ID=AT4G00070-E.5;Parent=AT4G ;constitutive=1;ensembl_end_phase=-1;rank=5 4. CDS ID=AT4G ;Parent=AT4G ;rank=5 4. CDS ID=AT4G ;Parent=AT4G ;rank=4 4. exon ID=AT4G00070-E.4;Parent=AT4G ;constitutive=1;rank=4 Biol Practical Biocomputing 31

32 Print out the beginning and ending coordinate of every transcript in the Arabidopsis genome based on the GFF3 file Read one line at a time Use split to get the columns Identify whether the feature is a transcript Split the comment to get the ID Print out the ID, begin, and end information Biol Practical Biocomputing 32