More regular expressions, synchronizing data, comparing files

Transcription

1 More regular expressions, synchronizing data, comparing files Laboratory of Genomics & Bioinformatics in Parasitology Department of Parasitology, ICB, USP

2 Regular expressions POSIX regular expressions are divided in two sets: basic and extended The frst thing about regexes: anything is a regex; in the case of regular strings, like banana, a regex composed exclusively of literal characters Metacharacters, on the other hand, have special meanings and are responsible for the fexibility and power of regexes The metacharacters in basic regular expressions are: ^. [ ] $ *

3 Regular expressions As can be seen, some of these metacharacters have special meaning to the shell as well, so it is essential to enclose a regex in quotes to avoid trouble The meanings of the metacharacters for basic POSIX regexes are:. : any one character (including the period itself) ^ and $ : anchors to the beginning or end of the line, respectively [ and ] : enclose a character class, as we ve seen before (using the dash gives us a range, e.g., [a-z]); within these, the ^ means negation * : zero or more occurrences of something specifc As we saw earlier, the meaning of a metacharacter can change depending on where it occurs!

4 Regular expressions The caret (^) only has special meaning when it is either: The first character in the regex (in which case it is an anchor, it represents the beginning of the line) Or the first character aftter the [ that starts a character class (in which case it means negation: match characters that are not in the class) In any other place, a caret is literally just a caret The same happens with the other anchor: the $ sign In this case, the sign has just one special meaning: it represents the end of the line, but only if it is the last character in the regex Anywhere else in the pattern, the dollar sign just matches a dollar sign

5 Examples '[i^g]' : will match lines that contain i and/or ^ and/or g '[^ig]' : will match lines that contain characters other than i and g (even if those line also contain i and/or g) '^[ig]' : will match lines that start in i or g '[ig$]' : will match lines that contain i and/or $ and/or g '[ig]$' : will match lines that end in i or g Except for - and an initial ^, all other metacharacters lose their special meanings when between [ and ] Between characters in a class, the - sign behaves the same way as in shell expansions: it means a range oft characters, e.g., [a-z]

6 Quiz time! Go to the Moodle site and choose Quiz 30 (beware time limits!)

7 We have seen the Basic Regular Expressions (BRE) in the POSIX standard But there are also Extended Regular Expressions (ERE) available in POSIX Regular expressions The ERE include everything that is in the BRE and adds a few more metacharacters Again, the metacharacters in BRE are: ^. [ ] $ * The ERE include, in addition: ( ) { }? +

8 Regular expressions Now we have the full set: ^. [ ] $ * ( ) { }? + Again, some of the metacharacters have special meanings to the shell, so always enclose the regex in quotes The ERE metacharaters have the following meanings: : alternation, or; A B C means match A or B or C ( ) : group some parts of the regex, and save the match? : match an element zero or one time + : match an element one or more times { } : match an element a specific number (or a range) of times

9 Important! What if one wants to search the text for characters that are usually metacharacters? We have to escape them, as mentioned last week Escaping strips the metacharacter of its special powers Escaping is done by putting a back-slash (\) before the character: '\(..(aa bb cc)\)' : will match a ( followed by two characters and either aa or bb or cc and a ) Putting the back-slash before a character that is not a metacharacter, on the other hand, can change its meaning, for some tools! For example: \n : line break \1 : back reference

10 Regular expressions The ( ) can group parts of the regex, like in arithmetic operations: ^aa bb cc is diferent from ^(aa bb cc) The frst: match lines that (start with aa) or (contain bb) or (contain cc) The second: match lines that (start with aa) or (start with bb) or (start with cc) The matches to the subexpressions created when the ( ) is used will be saved and can be accessed in back refterences called \1, \2, \3 etc. (for the frst, second, third etc. subexpressions) Example: '([i^g])..(aa bb cc)' : will save any matches to '[i^g]' in \1 and any matches to 'aa bb cc' in \2 Back references can be useful in substitutions, for example, by sed

11 A sed example Let s say you have fle /data/distros, which has dates in the third column (in the remote server, ) However, the dates are in a format you do not like; for example: Fedora 10 11/25/2008 You would prefer to have those dates written in the YYYY-MM-DD format sed can, with one (ugly) command, do that editing all at once: sed -r 's/([0-9]{2})\/([0-9]{2})\/([0-9]{4})/\3-\1-\2/' /data/distros See that we have used an option (-r) to tell the program to use extended regular expressions We also had to escape the / character, writing it as \/ instead sed can use other characters to separate the regex and replacement: sed -r 's#([0-9]{2})/([0-9]{2})/([0-9]{4})#\3-\1-\2#' /data/distros

12 A sed example sed -r 's#([0-9]{2})/([0-9]{2})/([0-9]{4})#\3-\1-\2#' /data/distros Let s dissect that example, looking just at the regex: 's#([0-9]{2})/([0-9]{2})/([0-9]{4})#\3-\1-\2#' substitution command, using # as separator ([0-9]{2}) ([0-9]{2}) ([0-9]{4}) \3-\1-\2 match 2 digits, match 2 digits, match 4 digits, replacement, put in \1 put in \2 put in \3 with - between each match

13 A sed example All of the following work just as well: 's#([0-9]{2})/([0-9]{2})/([0-9]{4})#\3-\1-\2#' 's*([0-9]{2})/([0-9]{2})/([0-9]{4})*\3-\1-\2*' 's%([0-9]{2})/([0-9]{2})/([0-9]{4})%\3-\1-\2%' 's ([0-9]{2})/([0-9]{2})/([0-9]{4}) \3-\1-\2 ' 's.([0-9]{2})/([0-9]{2})/([0-9]{4}).\3-\1-\2.' etc. etc. etc. But: 's/([0-9]{2})\/([0-9]{2})\/([0-9]{4})/\3-\1-\2/'

14 Regular expressions Some programs support just basic regular expressions, but other support those plus the extended ones Sometimes, it is necessary to tell the program which kind of regex you want it to use For grep, one can explicitly ftorce the use of BRE with the -G option, or of ERE with the -E option; alternatively, you can use the egrep command, and always use ERE sed uses BRE by default, but the -r option turns on ERE support

15 Regular expressions As seen earlier, {2} specifes that a match requires two occurrences of the character (or group of characters, if parentheses are used) But the { } metacharacters can also specifes ranges or a minimum or maximum number of occurrences Some examples of the quantifcation formats are: 'a{4,8}' : match if there are 4 to 8 letters a in sequence 'a{,4}' : match if there are up to 4 letters a in sequence 'a{4,}' : match if there are at least 4 letters a in sequence 'a{4}' : match if there are 4 letters a in sequence Notice that, if you use just the pattern 'a{,4}' above, for example, string aaaaaaa will match! That is because we have said nothing about what comes aftter the up to 4 letters a that we required

16 Quiz time! Go to the Moodle site and choose Quiz 31 (beware time limits!)

17 Now you do it! Go to the Moodle site, Practical Exercise 29 Follow the instructions to answer the questions in the exercise (and beware any time limits!) Remember: in the PE, you should do things in practice before answering the question!

18 Syncing data We have learned how to transfter data to and ftrom remote computers, specially using scp While a great tool, scp always copies everything you tell it to Wait Isn't that what we want anyway!? Actually, there is a better way: transferring only what has been updated after the previous transfer The rsync program is a great Unix tool for mirroring and backing up directories rsync can sync local and remote files, and its commands take the form: rsync options source destination The source and destination fles can be either local-local, local-remote, or remote-local (but not remote-remote)

19 Syncing data rsync frequently uses a few options : -a : archive mode, short-hand for several options most importantly, recursive copy and keep permissions (see man page) -z : compress data on-the-fy for transfer; more use for transfers over the network -v : verbose messages, gives details of the sync and a summary when the operation has fnished As usual, the options can be combined, for example: rsync -azv local_dir computer_name:remote_dir It is customary to create cron jobs to periodically and automatically run rsync commands that synchronize data between machines or between a machine and external storage, for example

20 Syncing data Try it on the remote server ( ) Let's sync a local directory into your home directory: time rsync -av /data/jc JC The time command is not needed! It is here just to measure how long it took to run the command that comes after it There was no JC directory in your area; rsync automatically creates a destination directory if it does not exist Since this is a transfer within the same computer, the -z option is not necessary; but does it hurt to use it? Try it! Now that you know the time that it took to sync the data without compression, either delete the JC directory created in your area or use a diferent destination (e.g., JC2), and run rsync again, this time with -z

21 Syncing data If you did everything correctly, you must have noticed that using compression took about 10 times more time than not Compressing the data takes time, so it is only worth doing if you are transferring data to a remote computer, using a slow network connection What happens if you now try to sync the data to a directory where you have already saved it previously? Do it! In our example: rsync -av /data/jc JC As you can see, nothing was copied this time; rsync looks for updated fles (looking at modifcation time stamps) and only copies those

22 About time Now, let s look at a few (other) CLI commands that deal with time There is one command that simply waits for a certain amount of time: sleep sleep has no options (besides --help and --version, a.k.a the common options of the core utilities package) ; it only takes the time to wait: sleep 10 Will wait 10 seconds before returning the prompt sleep 5m ; echo 'BOOM!' Will wait 5 minutes before running echo The deftault unit is seconds (s), but one can also use minutes (m), hours (h) or days (d)

23 Quick parenthesis... ( ) To emphasize what we saw in the previous slide: you can give the system more than one command at once Separate the two or more diferent commands with a semicolon, i.e., ; As seen ealier: sleep 5m ; echo 'BOOM!' Will tell the system to run the two commands, one after the other, independently There is no redirection or any other infuence of the command preceding the semicolon on the one after it

24 About time We have seen the time command, which tells us how long it took for a command to run There is another command in Unix that tells the system to stop executing a program if it has not fnished running after a certain amount of time: timeout Like time, timeout is placed right before the command it will afect The main option is the amount of time (by default, in seconds) timeout 2d command -x filex Will run the hypothetical program command (with option -x, on fle filex) for no more than two days If the program fnishes beftore that deadline, timeout does not do anything; otherwise, it kills the program

25 About time The date command allows us to perform some operations on dates Without any options, date prints the current date and time in the system With the +%s option (and nothing else), date prints the number of seconds since :00:00 UTC (the Unix epoch) The -d option specifes which date to use (default is today); for example: Will print: date -d ' ' Mon Jan 1 00:00:00 BRST 2018

26 About time date also understands human language ; for example: date -d '180 days' date -d '37000 minutes ago' date -d '-10 days 5 hours' +%s option is a ftormat option, and there are many such formats available for date (see the man page) For example: date -d 'Jun 14' +%j Will print 165: June 14 th is day number 165 in the year

27 About time Another useful tool for dealing with dates is cal or ncal It can print calendars and, in the case of ncal, the date of Easter ncal -e 2017 cal 10 cal cal -3 These commands print, respectively: the date of Easter Sunday in 2017; the calendar for year 10 (yes, 2008 years ago); the calendar for October 2018; and the calendar for this month, the previous and the next It is also possible to display the old Julian calendar

28 About time Yet another tool for dealing with time is tzselect (time zone selection) This tool allows one to fnd the ofcial name (in English) of any time zone, as well as see the current time there tzselect does not take any options, but is rather an interactive program Try it on the remote server: tzselect...and follow the instructions on the screen As can be seen, the program also accepts geographical coordinates (latitude and longitude)

29 Now you do it! Go to the Moodle site, Practical Exercise 30 Follow the instructions to answer the questions in the exercise (and beware any time limits!) Remember: in the PE, you should do things in practice before answering the question!

30 Comparing file content It is often useful to see what the diferences (or commonalities) between two text fle are The comm program accepts two sorted fles and prints out three columns: Lines unique to the frst fle Lines unique to the second fle Lines common to both fles comm can be specially useful when one has two long lists of items to compare Example: comm -3 file1 file2

31 Comparing file content comm -3 file1 file2 This will output only the lines that are unique to file1 and to file2, but not those that are shared by both fles Try it! In the remote server, run: comm -3 /data/file_coma /data/file_comb This will print in one column all lines that are unique to file_coma and, in the other, all that are unique to file_comb 1 2 w 3 h s

32 Comparing file content While comm is quite useful, it is not very powerful The diff program is used a lot, specially by programmers (e.g., comparing diferent source code versions) and system administrators (e.g., comparing confguration fles), to compare fles Diferently from comm, for diff fles do not need to be sorted actually, they should not be sorted just in order to run diff! diff is used to create the so-called dif file, which allows us to use another tool (patch) to transftorm one file into the other There are many options to control what should be taken into account when comparing fles: how to treat blank space, empty lines, case sensitivity etc. diff can compare ftull directories (and their subdirectories, recursively, if told to), fle by fle

33 Comparing file content The main advantages of a dif (or patch) fle are that: It is very small compared to the full size of the original fles It concisely shows the changes between the fles, allowing one to quickly evaluate the diferences diff can list the diferences between two fles in diferent ways: Deftault ftormat, the shortest (but hardest to read) Context ftormat, shows lines around the changes, for context Unified ftormat, eliminates redundancies of the context format Let s try! In the remote server, run: diff /data/darwin_1 /data/darwin_2

34 Quiz time! Go to the Moodle site and choose Quiz 32 (beware time limits!)

35 Now you do it! Go to the Moodle site, Practical Exercise 31 Follow the instructions to answer the questions in the exercise (and beware any time limits!) Remember: in the PE, you should do things in practice before answering the question!

36 Recap Extended regular expressions (ERE) include a few more capabilities to the basic set: ( ) { } +? egrep or grep -E and sed -r are able to use ERE in searching and editing of text fles rsync is a great tool for incremental transfter of data (i.e., only transfer what has changed), and is thus used a lot for backups Transfers within the same machine, i.e., disks connected to the same computer, should not use compression (-z): it is much slower in this case! Linux has several tools to deal with time time measures how long it took for a command to run timeout stops a program if it is not fnished before a certain amount of time

37 Recap tzselect gives information about time zones and the current time anywhere in the world cal and ncal show calendars and the day of Easter (ncal only) date does several operations and calculations on dates and times sleep simply sleeps for a given period of time, working as a timer Comparing the contents of text fles is an important task comm compares two previously sorted fles and lists which lines are unique or shared diff is the main text fle comparison tool in the Unix world, and has three output styles: deftault, context, and unified It is used a lot to distribute program source code updates, which can be applied with the patch program