Week 9, Lecture 17. Some Programming Required. Programming Languages. Turing Completeness 10/20/15

Size: px

Start display at page:

Download "Week 9, Lecture 17. Some Programming Required. Programming Languages. Turing Completeness 10/20/15"

Cornelius Richards
5 years ago
Views:

1 Some Programming Required BMMB 852: Applied Bioinforma5cs Week 9, Lecture 17 István Albert Bioinforma5cs Consul5ng Center Penn State, 2015 Exis5ng sohware tools can rarely do all steps Source data in different files may be formaned differently We need to bridge the differences with simple transforma5ons Some programming ability is necessary for every project Two major groups Programming Languages Compiled languages à the output is (usually) a standalone program that can be run Interpreted languages à requires the presence of another sohware that in turn will run the sohware Common ques5on: which language can solve a given problem? Turing Completeness All languages can be used to solve any computable problem. But the solu5ons may end up radically different! Tradeoffs between simplicity, speed of development and the speed of execu5on. 1

they piece together exis5ng func5onality Possibly the first scrip5ng language: awk Appeared in 1977, strong UNIX (1972) roots It is the precursor of languages such as Perl (1987) and Python(1989) Has

2 It is all about the libraries! Modern sohware development is all about reusing exis8ng func5onality Programming languages are turning into so called very high level programming environments Most people rarely code full algorithms they piece together exis5ng func5onality Possibly the first scrip5ng language: awk Appeared in 1977, strong UNIX (1972) roots It is the precursor of languages such as Perl (1987) and Python(1989) Has fallen into disuse for a quite a while, predicted to fail. Large text based datasets have lead to a resurgence of the language. The Art of Unix Programming: TAOUP Seminal open source book by Eric Raymond The structure of an awk program awk pattern { action } Line oriented 1. Tries to match the panern to the line 2. If it is a match execute the ac5on Not quite right: The more things change the more they stay the same. There is now a resurgence of awk because the amount of data made our computers weak again. 3. Automa5cally splits the data by whitespace. 4. No panern means match everything. 2

Special variables Awk automa8cally splits the input by

the en5re line $1 first field $2 second field NF the number

for numerical context > < for comparison % modulo division

3 Special variables Awk automa8cally splits the input by whitespace (spaces and tabs) and assigns names to them: $0 the en5re line $1 first field $2 second field NF the number of fields NR the number of the current line Operators + - * / for numerical context > < for comparison % modulo division (remainder of division) <space> string concatena5on ==,!= equal, not equal ~,!~ match, no match (regular expressions) AWK program in ac5on You may also put the program into a file matching ac5on 3

List the gene names and sizes Full program Write

tabs Many tools will auto-split by whitespace à

source of extremely subtle errors à leads to a

spaces Always specify the character to be split by!

$output field separator alias awk="awk -F '\t' -v$ bashrc file in your root folder so that it is

4 List the gene names and sizes Full program Write one step at a 5me The whitespace curse: spaces and tabs Many tools will auto-split by whitespace à this was thought to be convenient but is also the source of extremely subtle errors à leads to a column shih in a tab file if a field contains spaces Always specify the character to be split by! Customize awk to use tabs as both the input and output field separator alias awk="awk -F '\t' -v OFS='\t'" Tip: you can add this to the.profile or.bashrc file in your root folder so that it is ac5vate all the 5me Note: file names that start with a dot. are only listed if you do a ls -a This refers to programming languages as well! Do not use the split() methods with their default behavior (Python, Perl etc) unless you perfectly understand what they do) 4

Special panerns BEGIN à before the stream starts END à aher the stream ends Advanced awk usually not needed condi5onals : if

probably bener to learn Python But you can do a lot with just basic awk!

Large number of resources, the quirky name makes it very searchable à How to do X with awk?

5 Special panerns BEGIN à before the stream starts END à aher the stream ends Advanced awk usually not needed condi5onals : if loops: for, while break, con8nue associa8ve data structures (hash, dic8onary) You can do all that though at that point it is probably bener to learn Python But you can do a lot with just basic awk! Awk s power comes from its simplicity more complex programming is done with specialized programming languages A few awk resources Large number of resources, the quirky name makes it very searchable à How to do X with awk? How to use awk see course webpage for links Bioinforma5cs Survival Kit Everyday bioinforma5cs func5ons Bioawk (awk with bioinforma5cs super powers) Seqtk (sequence toolkit) Tabix (indexed, searchable tab files) Tabtk (tab tookit) 5

genomic indices that have a coverage that is over a half of the largest coverage observed in your data. What percent of your genome is that?

6 Bioawk func5ons Bioawk examples When used with a format it will populate internal variables such as: $name, $seq, $start, $end etc. It also works with both gzipped or regular text files: it s magic no less Homework 17 Using the alignments used in lecture 16 write awk scripts that perform the following: Print out those genomic indices that have a coverage that is over a half of the largest coverage observed in your data. What percent of your genome is that? Print the alignments for which the DNA fragment length is over half of the maximum observed DNA fragment length. (SAM column 9) Hints: The simple solu5on is to break each problem into two steps as, find the value, filter with the value) Only show the first few lines (don t submit all the data) 6

Practical Bioinformatics for Life Scientists. Week 4, Lecture 8. István Albert Bioinformatics Consulting Center Penn State

Practical Bioinformatics for Life Scientists Week 4, Lecture 8 István Albert Bioinformatics Consulting Center Penn State Reminder Before any serious work re-check the documentation for small but essential