More text file manipulation: sorting, cutting, pasting, joining, subsetting,

Transcription

1 More text file manipulation: sorting, cutting, pasting, joining, subsetting, Laboratory of Genomics & Bioinformatics in Parasitology Department of Parasitology, ICB, USP

2 Inverse cat Last week we learned about viewing text with the cat (for concatenate) command cat sends the contents of one or more text fles (or STDIN) to STDOUT But what if you want to get the contents from each fle starting from the end instead of the beginning? Then, you invert cat and use the tac command cat file Program version: Date: Thu May 4 Seed:./18s Seed type: DNA Database: db/dma97 Database type: fastq Duplicate headers name: yes Expansion direction: both Number of threads: 4 Assembler used: abyss-pe tac file Assembler used: abyss-pe Number of threads: 4 Expansion direction: both Duplicate headers name: yes Database type: fastq Database: db/dma97 Seed type: DNA Seed:./18s Date: Thu May 4 Program version:

3 Getting columns Tabular fles frequently contain much more data than we are interested in It is thus sometimes useful to be able to get at just one or a few columns of the data The cut command allows us to do exactly that cut uses the TAB character as the default column delimiter Of course, there is an option to change that Another option tells the program which column(s) to retrieve

4 Getting columns Let s try! In the remote server ( ), run: cut -f 1 /data/column_example Notice that only the frst column (tab-delimited!) of the fle was sent to STDOUT Now try: cut -f 1 /data/column_example2 cut -f 1 -d, /data/column_example2 In the frst instance, nothing got cut, since there was no line containing a tab character (felds are separated by commas, in that fle) In the second instance, we change the feld delimiter to comma (,)

5 Quiz time! Go to the Moodle site and choose Quiz 24 (beware time limits!)

6 Sorting data One of the most basic, and useful, things a computer is used for is sorting data many complex data analysis algorithms depend on sorted data The sort command is the command-line tool that we can use for that sort sorts the contents of STDIN or one or more fles and sends the resulting data to STDOUT Notice that it said one or more fles! We can merge the contents of diferent fles while sorting By default, sorting is performed on the content of the whole line But there are many options that allow us to change sorting behavior Besides sorting, sort can also be use to just check whether some data is sorted

7 Sorting data There are diferent kinds of sorting available, e.g., numerical, alphabetical By default, the sort command does case sensitive lexical sorting Search felds are separated by white-space (spaces or TAB), but that can be changed using an option Let s try! In the remote server, as usual First, let s look at the contents of fle /data/column_example as they are Then, sort the fle s contents: sort /data/column_example That is the simplest sort command: uses the whole line, default delimiter, default search type

8 Sorting data Before avocado 12 lime 4 apple 5 banana 3 orange 6 date 20 After apple 5 avocado 12 banana 3 date 20 lime 4 orange 6 But what if we want the results to be in descending order? Or ordered using diferent felds (columns), instead of the whole line? Diferent feld delimiter than white-space? Or using numerical comparison?

9 There are diferent kinds of sorting available, e.g., numerical, alphabetical Let s try sorting numerically now (using the second column): sort -k 2 -n /data/column_example Option -k determines the column(s) to use in sorting As used above, we are telling the program to use column 2 and whatever comes afterwards; but there are many other ways: -k 2,2 : use just column 2 Sorting data -k 3,7 : use columns 3, 4, 5, 6, and 7 -k 4n,5d : sort by columns 4 (numerically) and 5 (dictionary order) -k 1,1n -k 2,2M : sort frst by column 1 (numerically), then break any ties by sorting column 2 (by month name, like JAN, FEB etc.)

10 Quiz time! Go to the Moodle site and choose Quiz 25 (beware time limits!)

11 Now you do it! Go to the Moodle site, Practical Exercise 23 Follow the instructions to answer the questions in the exercise (and beware any time limits!) Remember: in the PE, you should do things in practice before answering the question!

12 Unsorting data We have learned how to sort data to get it in order After that is done, there is no way to undo it, of course; the original order is lost (unless you kept the original fle, obviously) But we can do something else to get data out of sorted order: shufe That is done by the shuf command: shuf /data/shuf_example shuf works generally as a randomness generator For example: shuf -i 0-9 : print digits 0 to 9, one per line, randomly shuf -i 0-9 -r -n 50 : print 50 digits (0 to 9) randomly shuf -e heads tails -r -n 50 : simulate 50 coin tosses

13 Deleting parts of the data The Unix program colrm allows us to delete columns from the data (from STDIN) Here, a column is defned as a single character For example, the line: acbce example column 1 column 2 column 19 column 3 column 6

14 Deleting parts of the data The exception here is tab: each one counts for eight columns colrm gets two numbers to specify the columns The frst number is the frst column to remove; the second, the last column to remove If only one number is given, remove everything from there to the end of the line For example: colrm 8 <<< "acbce example" Will result in: acbce 1

15 Deleting parts of the data Another example: colrm 3 10 <<< "acbce example" Will result in: Finally: ac5 example colrm 3 30 <<< "acbce example" Will result in: ac That is: if the line is shorter than the second number, just delete to the end

16 Putting things together We have learned how get columns out of a fle with cut and colrm Some other commands, such as paste, join diferent fles into one Typically, each one of the input fles to paste will be present as a column in the new fle, creating a table For example: paste /data/p1 /data/p2 /data/p3 If you look at the original contents of those two fles, you will see that everything that was in the frst fle was put in column 1 of the output, while everything that was in the second one was put in column 2 One interesting option to paste is -s (for serial), which basically transposes the data. Try it!

17 Putting things together Another command to join fles is join join works as a very primitive database function It joins lines from two fles based on one common column containing an identifer string The fles must be sorted based on that common column! For example: join -j 1 /data/j4 /data/j5 If you look at the original contents of those two fles, you will see that column 1 was the join feld : whenever that was the same in both fles, the lines got joined in the output When there is no match, by default the line is not included in the output

18 Quiz time! Go to the Moodle site and choose Quiz 26 (beware time limits!)

19 Now you do it! Go to the Moodle site, Practical Exercise 24 Follow the instructions to answer the questions in the exercise (and beware any time limits!) Remember: in the PE, you should do things in practice before answering the question!

20 Translating characters Another way to modify a text stream is by modifying some characters from a list The tr (translate) program translates (i.e., substitutes), squeezes (i.e., eliminates repetitions), and deletes characters The default for tr is to translate: characters from set 1 will be replaced with characters from set 2 whenever found For example: tr a-z A-Z < /data/file_y If you look at the original contents of that fle, you will see that all lower-case letters have been changed to upper-case

21 Translating characters That could be written as: tr '[:lower:]' '[:upper:]' The [: :] part is what is called a character class There are several such classes recognized by the shell, and they are shortcut ways of referring to a set of characters by a name The character classes listed in the man and info pages for tr are (check it out for explanations on each): alnum blank digit lower punct upper alpha cntrl graph print space xdigit

22 Translating characters Some of the character classes, such as cntrl can be very useful to remove certain non-printing (invisible) characters from data, which often lead to errors in data processing and are hard to manually spot and edit out A useful option is -c (or -C or --complement), which will get the complement of the characters listed, i.e., whatever is not in the list For example: tr -d -c '[:alnum:]' < /data/p3 That command will delete everything in fle /data/p3 that is not a letter or number

23 Splitting data Sometimes a fle is too large, and you want to split it into smaller fles Maybe you want to send diferent parts to diferent people, or you only need a sample etc. The split command allows us to break fles into smaller subsets By default, it separates one thousand lines at a time into a new fle, then starts another fle with up to one thousand lines and so on But it is possible to ask for other criteria, such as number of total fnal fles you want or maximum amount of bytes per fle, for example For example: split /data/file_x This is the simplest case, and will split the fle into smaller fles, each containing 1000 lines from the original, until the end. Try it!

24 Splitting data By default, split creates new fles called xaa, xab, xac etc. Those names are made up of two parts: the prefx (x) and a sufx (aa, ab, ac etc.) Both the prefx and the length of the sufx (in number of characters) can be changed For example: split -n 10 file_1 P Now, the prefx for the fles created will be P instead of x Options control the length of the sufx and what (if any) additional sufx you want to use

25 Now you do it! Go to the Moodle site, Practical Exercise 25 Follow the instructions to answer the questions in the exercise (and beware any time limits!) Remember: in the PE, you should do things in practice before answering the question!

26 Recap The number of Unix text tools keeps on growing tac is an easy way to invert the lines of a fle One of the most basic computer tasks, being at the basis of many advanced data analysis algorithms, is sorting; sort is a very fexible program to do that shuf provides a randomness generator, which can be very useful Large data fles are often tabular in nature, and Unix has a few programs designed to deal with that kind of fle cut allows us to retrieve one or more columns from a fle

27 Recap paste and join, on the other hand, enable the building of larger, multi-column fles out of smaller fles containing the individual columns tr (translate) is a quick way of replacing or deleting certain characters colrm provides a way to delete certain columns (for example, from characters 5 to 9) from each line of a fle split lets us break a large fle into smaller ones, based on the number of lines or bytes that we would like to have in each resulting fle