Linux Bash Shell Scripting

Transcription

1 University of Chicago Initiative in Biomedical Informatics Computation Institute Linux Bash Shell Scripting Present by: Mohammad Reza Gerami

2 Day 2 Outline Support Review of Day 1 exercise Unix Shell programming: Shell scripts Variables Exercise #1 10 minute break Expression statements Control statements Input parameters (read and variables) Exercise #2 2

3 Where to get the slides 3

4 University of Chicago Initiative in Biomedical Informatics Computation Institute Support

5 Major Offerings Main Cluster (ibicluster) Large Memory Computation Linux (brdfbigl) Large Memory Computation Windows(brdfbigw) Ancillary Clusters (brdfbmem,ibidaily, etc) Database Offerings (mysql, postgres, mssql) Project Shares 5

6 CI-iBi Support ibi Website: BRDF Website: Support Address By Support Phone Number (773)

7 Day 2 Outline Support Review of Day 1 exercise Unix Shell programming: Shell scripts Variables Exercise #1 10 minute break Expression statements Control statements Input parameters (read and variables) Exercise #2 7

8 University of Chicago Initiative in Biomedical Informatics Computation Institute Connect to Linux via Shell $ ssh username@brdfgate.uchicago.edu $ ssh brdfbigl

9 Day 1 Exercise Overview Question: what organisms and functions do the given sequences belong to? Commands used: wget get the protein sequence data from the internet tar to uncompress the data acquired mkdir & cd to organize the data cat to concatenate similar files into one file edit a file used nano to edit the file containing multiple sequences with the same id blastall to perform sequence alignment and comparison to a set of sequences from a database grep to get the top hit for each query sequence cut to get the id of the top hit fastacmd to get the fasta sequence given the id we are interested in 9

10 fasta file >gi gb AAD cytochrome b [Elephas maximus maximus] LCLYTHIGRNIYYGSYLYSETWNTGIMLLLITMATAFMGYVLPWGQMSFWGATVITNLFSAIPYIGTNLV EWIWGGFSVDKATLNRFFAFHFILPFTMVALAGVHLTFLHETGSNNPLGLTSDSDKIPFHPYYTIKDFLG LLILILLLLLLALLSPDMLGDPDNHMPADPLNTPLHIKPEWYFLFAYAILRSVPNKLGGVLALFLSIVIL GLMPFLHTSKHRSMMLRPLSQALFWTLTMDLLTLTWIGSQPVEYPYTIIGQMASILYFSIILAFLPIAGX IENY >sequence_1 MDSKGSSQKGSRLLLLLVVSNLLLCQGVVSTPVCPNGPGNCQVSLRDLFDRAVMVSHYIHDLSS EMFNEFDKRYAQGKGFITMALNSCHTSSLPTPEDKEQAQQTHHEVLMSLILGLLRSWNDPLYHL VTEVRGMKGAPDAILSRAIEIEEENKRLLEGMEMIFGQVIPGAKETEPYPVWSGLPSLQTKDED ARYSAFYNLLHCLRRDSSKIDTYLKLLNCRIIYNNNC Sequences are separated by the ">" character First line sequence ID (unique) description (optional) Next lines Amino acid or nucelotide sequence composition 10

11 BLAST output QUERY_ID HIT_ID IDENTITY LENGTH MIS GAPS Q_ST Q_END H_ST H_END E-VAL P-SCO sequence_1 gi sequence_1 gi sequence_1 gi sequence_1 gi sequence_1 gi sequence_1 gi sequence_1 gi sequence_1 gi sequence_1 gi sequence_2 gi sequence_2 gi QUERY_ID ID for your query sequence HIT_ID ID for the alignment hit IDENTITY percent of amino acids in alignment that exactly match LENGTH length of alignment MIS number of mismatches in the alignment GAPS number of gaps in the alignment Q_ST query start Q_END query end H_ST hit start H_END hit end E-VAL e-value (the lower the better) P-SCO p-score (the higher the better) 11

12 Caveats There are no "one size fits all" solutions Always test scripts Make sure you completely understand new commands and techniques You will not be able to learn everything about bash scripting today Try examples to get familiar Use the web to find help on how to do things All the examples we go through are available in: /biodba/workshopdir/april_2010/day_2/exercises 12

13 The shell of Linux Linux has a variety of different shells: Bourne Again shell (bash) Why write shell scripts? To avoid repetition: If you do a sequence of steps with standard Unix commands over and over, why not do it all with just one command? Or in other words, store all these commands in a file and execute them one by one To automate difficult tasks: Many commands have subtle and difficult options that you don't want to figure out or remember every time 13

14 Simple Clean-up Example Assume that I need to clean up a directory every time before executing BLAST program $ rm rf *.stderr $ rm rf *.stdout $ rm rf *.tmp 14

15 Shell scripts List of command, executed in order The shebang line: #! tells the CPU what shell to use to execute script The shell name is the shell that will execute this script. E.g: #!/bin/bash (which we will use) which command $ which bash /bin/bash If no shell is specified in the script file, the default is chosen to be the executing shell 15

16 The first bash script Write programs using your favorite text editor vi, emacs, nano, pico, etc. So fire up a text editor; for example: $ mkdir ~/scripts $ cd scripts $ nano hello.sh Type the following inside it: #!/bin/bash # This is a commented line, will not be executed # This is my first script "Hello World" echo "Hello World" The first line tells Linux to use the bash interpreter to run this script Make the script executable: $chmod u+x hello.sh $ls l -rwxr--r-- hello.sh 16

17 To execute the program: $ hello.sh The first bash program -bash: hello.sh: command not found $PATH environment variable holds the location where all commands are stored $ echo $PATH /usr/bin:/bin:/usr/sbin The home directory (where the command hello.sh is located) is not in the variable PATH We must specify the path of hello.sh $ /cchome/arodrigu1/scripts/hello.sh 17 $./hello.sh Hello World

18 Back to the clean-up example We can put all those commands into a shell script, called mycleandir.sh $ nano mycleandir.sh #!/bin/bash rm rf *.stderr rm rf *.stdout rm rf *.tmp echo "Deleted files with suffix blastout, stderr, stdout, tmp" Need to make it executable $ chmod u+x mycleandir.sh Run your script and it will be processed in order $./mycleandir.sh 18

19 Variables There are two types of variables: Environmental variables Local variables 19

20 Environmental Variables Environmental variables hold special values. Environmental variables are set by the system on initial login /etc/profile, /etc/profile.d/ and ~/.bash_profile or ~/.profile. If you want to know what the variable holds call it with a "$" sign: $ echo SHELL SHELL $ echo $SHELL /bin/bash $ echo $HOME /cchome/arodrigu1 $ echo $PATH /usr/x11r6/bin:/usr/local/bin:/bin:/usr/bin env command 20

21 Special Environmental Variables: PATH PATH: The search path for commands Usually, we type in the commands in the following way: $./hello.sh By setting PATH=$PATH:~/scripts our working directory is included in the search path for commands, and we simply use the export command: 21 $ export PATH=$PATH:~/scripts $ hello.sh

22 More Environmental Variables LOGNAME: contains the user name HOSTNAME: contains the computer name. MACHTYPE: system hardware BLASTDB: directory path where the blast database are stored PWD: current path you are at 22

23 Local Variables We can use variables as in any programming languages Stored as strings Declaring a variable: $ STR='Hello World!' $ echo $STR Hello World! Assign a value to a variable Call the variable by putting the '$' at the beginning 23

24 Double Quotes When assigning character data containing spaces or special characters, the data must be enclosed in either single or double quotes. Using double quotes (partial quoting) to show a string of characters will allow any variables in the quotes to be resolved $ var="test string" $ newvar="value of var is $var" $ echo $newvar Value of var is test string 24

25 Single Quotes Using single quotes (full quoting) to show a string of characters will not allow variable resolution $ newvar='value of var is $var' $ echo $newvar Value of var is $var 25

26 Command Substitution The backquote "`" is different from the single quote " ". It is used for command substitution: `command` You can assign the output of a command to a variable $ ls hello.sh mycleandir.sh $ LIST=`ls` $ echo $LIST hello.sh mycleandir.sh 26

27 Exercise #1 Create a bash shell script that when executed will print: $ hello_user.sh Hello arodrigu1! Your home is: /cchome/arodrigu1 Today's date is: Fri Mar 19 18:19:42 CDT 2010 Hint: You can get the date by using the date command 27

28 Exercise #1: Solution $ nano hello_user.sh #!/bin/bash echo "Hello $LOGNAME!" echo "Your home is: $HOME" mydate=`date` echo "Today's date is: $mydate" $ chmod u+x hello_user.sh $ export PATH=$PATH:~/scripts $ hello_user.sh 28

29 29 Arithmetic Evaluation expr expression expression operators: arithmetic: +, -, *, /, % comparison: <, <=, ==,!=, >=, > boolean/logical: &, parentheses: (, ) Calculates the value of an expression $ result=`expr 1 + 2` $ echo $result 3 Why do we need the expr command? $ result=1+2 $ echo $result 1+2

30 Control statements Control statements control the flow of execution in a programming language Most common types of control statements: conditionals: if/then/else, case,... loop statements: while, for, until, do,... branch statements: subroutine calls 30

31 Conditional Statements Conditionals lets us decide whether to perform an action or not, this decision is taken by evaluating an expression if [ expression ]; then statements elif [ expression ]; then statements else statements fi ## must have space between brackets ## brackets test an expression the elif (else if) and else sections are optional 31

32 Example grep returns 0 if it finds something returns non-zero otherwise Lets write a script that determines whether unix exists in the file "myfile" 32 $ cp /biodba/workshopdir/april_2010/day2/exercises/*. $ cat if1.sh if grep "UNIX" myfile >/dev/null then echo "It's there" fi $./if1.sh It's there redirect to /dev/null so that "intermediate" results do not get printed This file is available for everyone

33 Using else with if Example: $ cat if2.sh #!/bin/bash if grep "UNIX" myfile2 >/dev/null then echo "UNIX occurs in myfile2" else echo "No!" echo "UNIX does not occur in myfile2" fi $./if2.sh No! UNIX does not occur in myfile2 33

34 Using elif with if $ cat if3.sh #!/bin/bash if grep "UNIX" myfile3 >/dev/null then echo "UNIX occurs in myfile3" elif grep "DOS" myfile3 > /dev/null then echo "DOS appears in myfile3 not UNIX" else echo "nobody is here in myfile3" fi $./if3.sh DOS appears in myfile3 not UNIX 34

35 Expressions Expressions can be: String comparison Numeric comparison File operators Logical operators 35

36 Expressions: String Comparisons String Comparisons: = compare if two strings are equal!= compare if two strings are not equal -n evaluate if string length is greater than zero -z evaluate if string length is equal to zero Examples: [ s1 = s2 ] (true if s1 same as s2, else false) [ s1!= s2 ] (true if s1 not same as s2, else false) [ s1 ] (true if s1 is not empty, else false) [ -n s1 ] (true if s1 has a length greater then 0, else false) [ -z s2 ] (true if s2 has a length of 0, otherwise false) 36

37 Expressions: String Comparisons Compare the user's name given with the environment variable $USER $ cp /biodba/workshopdir/april_2010/day_2/if4.sh./if4.sh $ cat if4.sh #!/bin/bash echo -n "Enter your login name: " # ask user input read name # store input in var if [ "$name" = "$USER" ]; then echo "Hello, $name. How are you today?" else echo "You are not $USER, so who are you?" fi $./if4.sh Enter your login name: arodri You are not arodrigu1, so who are you? 37

38 Expressions: Number Comparisons Number Comparisons: -eq compare if two numbers are equal -ge compare if one number is greater than or equal to a number -le compare if one number is less than or equal to a number -ne compare if two numbers are not equal -gt compare if one number is greater than another number -lt compare if one number is less than another number Examples: [ n1 -eq n2 ] (true if n1 same as n2, else false) [ n1 -ge n2 ] (true if n1greater then or equal to n2, else false) [ n1 -le n2 ] (true if n1 less then or equal to n2, else false) [ n1 -ne n2 ] (true if n1 is not same as n2, else false) [ n1 -gt n2 ] (true if n1 greater then n2, else false) [ n1 -lt n2 ] (true if n1 less then n2, else false) 38

39 Expressions: Number Comparisons Perform a mathematical operation if the number is between a range, otherwise let the user know the number entered is incorrect $ cat if5.sh #!/bin/bash echo -n "Enter a number 1 < x < 10: " # ask user input read num # store input in var if [ "$num" -lt 10 ]; then if [ "$num" -gt 1 ]; then echo "$num*$num=$(($num*$num))" else echo "Wrong insertion!" fi else echo "Wrong insertion!" fi $./if5.sh Enter a number 1 < x < 10: 5 5*5=25 39

40 Expressions: Logical operators Logical operators: && logically AND two logical expressions logically OR two logical expressions Example: the numerical example (if5.sh) can be made into one if statement by using logical operators $ cp /biodba/workshopdir/april_2010/day_2/if7.sh. $ cat if7.sh #!/bin/bash echo -n "Enter a number 1 < x < 10: " read number if [ "$number" -gt 1 ] && [ "$number" -lt 10 ]; then echo "$num*$num=$(($num*$num))" else echo "Wrong insertion!" fi 40 $./if7.sh Enter a number 1 < x < 10: 5 5*5=$25

41 Expressions: File Operators Files operators: -d check if path given is a directory -f check if path given is a file -s check if path given is a symbolic link -e check if file name exists -s check if a file has a length greater than 0 -r check if read permission is set for file or directory -w check if write permission is set for a file or directory -x check if execute permission is set for a file or directory Examples: [ -d fname ] (true if fname is a directory, otherwise false) [ -f fname ] (true if fname is a file, otherwise false) [ -e fname ] (true if fname exists, otherwise false) [ -s fname ] (true if fname length is greater then 0, else false) [ -r fname ] (true if fname has the read permission, else false) [ -w fname ] (true if fname has the write permission, else false) [ -x fname ] (true if fname has the execute permission, else false) 41

42 Expressions: File Operators Check if a certain file exists $ cat if6.sh #!/bin/bash if [ -f /etc/fstab ]; then cp /etc/fstab. echo "Done." else echo "This file does not exist." fi exit 1 $./if6.sh Done. 42

43 for loops for loops allow the repetition of a command for a specific set of values. Syntax: for var in value1 value2... do command_set done command_set is executed with each value of var (value1, value2,...) in sequence 43

44 for loop example 44 Lets calculate the smallest number among a set $ cat for3.sh #!/bin/bash smallest=10000 for i in do echo "evaluation $i" if [ $i -lt $smallest ] then smallest=$i fi done echo $smallest $./for3.sh 3

45 for loop example Lets count the number of files that are executable in current directory $ cat for2.sh #!/bin/bash # initialize variable count=0 for filename in `ls`; do if [ x $filename ] then count=`expr $count + 1` fi done echo "Total of $count files executable" $./for2.sh Total of 5 files executable NOTE: expr $count + 1 adds one to variable 45

46 The while loop While loops repeat statements as long as the next Unix command is successful Syntax: while [ expression ] do command_set done 46

47 while loop example Lets do a summation of every number from 1 to 100 $cat while1.sh #! /bin/bash i=1 # declare var sum=0 # declare var while [ $i -le 100 ] do sum=`expr $sum + $i` i=`expr $i + 1` done echo The sum is $sum. $./while1.sh The sum is

48 while loop example Lets calculate the number of fasta files in a sequence file #!/bin/bash # while2.sh counts the number of fasta sequences found in a fasta file count=0 # initialize a variable that holds # the count of sequences # in each iteration we are going to read the next line in flie while read line do if echo $line grep ">" > /dev/null # determine if the current line # is a sequence header ">" then count=`expr $count + 1` # perform an addition to the count fi done < "sequences.fasta" # define the filename you wish to use # tell me the total count of fasta sequences echo "Your fasta file contains $count sequences" 48

49 Shell Parameters Positional parameters are assigned from the shell's argument when it is invoked ( i.e. ls -l dir1 dir2) Positional parameter "N" may be referenced as "${N}", or as "$N" when "N" consists of a single digit. Special parameters $# is the number of parameters passed $0 returns the name of the shell script running as well as its location in the filesystem $* gives a single word containing all the parameters passed to the script $@ gives an array of words containing all the parameters passed to the script $ cat sparameters.sh #!/bin/bash echo "$#; $0; $1; $2; $*; $@" $ sparameters.sh alba chiara 2;./sparameters.sh; alba; chiara; alba chiara; alba chiara 49

50 Using parameters in while loop example Lets calculate the number of fasta files in a sequence file #!/bin/bash # while3.sh counts the number of fasta sequences found in a fasta file filename=$1 count= # assign the input parameter filename to a variable # initialize a variable that holds the count of sequences # in each iteration we are going to read the next line in flie while read line do if echo $line grep ">" > /dev/null # determine if the current line # is a sequence header ">" then count=`expr $count + 1` # perform an addition to the count fi done < $filename # define the filename you wish to use # tell me the total count of fasta sequences echo "Your fasta file contains $count sequences" 50

51 University of Chicago Initiative in Biomedical Informatics Computation Institute Exercise 2

52 Overview of Exercise Problem: Given a set protein sequences, find out what organism each sequence is from and what each sequence does? Tarball file location: Tarball contains sequence files with multiple sequence with the same ids 52

53 Exercise #2 Create a bash shell script for yesterday's exercise and try to use as many control statements: Get the tar files via wget ( untar the file (fasta_files.tar.gz) concatenate the files (sequences may have the same id) modify the sequence ids use exercise2-skeleton.sh as a startup, variables have been setup Home exercise submit file to blast (blastall command) grep the top hit for each sequence get the hit id use the fastacmd command to get the fasta sequence Solutions located at: /biodba/workshopdir/april_2010/day_2/exercises/exercise2-solution.sh 53

54 fasta file >gi gb AAD cytochrome b [Elephas maximus maximus] LCLYTHIGRNIYYGSYLYSETWNTGIMLLLITMATAFMGYVLPWGQMSFWGATVITNLFSAIPYIGTNLV EWIWGGFSVDKATLNRFFAFHFILPFTMVALAGVHLTFLHETGSNNPLGLTSDSDKIPFHPYYTIKDFLG LLILILLLLLLALLSPDMLGDPDNHMPADPLNTPLHIKPEWYFLFAYAILRSVPNKLGGVLALFLSIVIL GLMPFLHTSKHRSMMLRPLSQALFWTLTMDLLTLTWIGSQPVEYPYTIIGQMASILYFSIILAFLPIAGX IENY >sequence_1 MDSKGSSQKGSRLLLLLVVSNLLLCQGVVSTPVCPNGPGNCQVSLRDLFDRAVMVSHYIHDLSS EMFNEFDKRYAQGKGFITMALNSCHTSSLPTPEDKEQAQQTHHEVLMSLILGLLRSWNDPLYHL VTEVRGMKGAPDAILSRAIEIEEENKRLLEGMEMIFGQVIPGAKETEPYPVWSGLPSLQTKDED ARYSAFYNLLHCLRRDSSKIDTYLKLLNCRIIYNNNC Sequences are separated by the ">" character First line sequence ID (unique) description (optional) Next lines Amino acid or nucelotide sequence composition 54

55 BLAST output QUERY_ID HIT_ID IDENTITY LENGTH MIS GAPS Q_ST Q_END H_ST H_END E-VAL P-SCO sequence_1 gi sequence_1 gi sequence_1 gi sequence_1 gi sequence_1 gi sequence_1 gi sequence_1 gi sequence_1 gi sequence_1 gi sequence_2 gi sequence_2 gi QUERY_ID ID for your query sequence HIT_ID ID for the alignment hit IDENTITY percent of amino acids in alignment that exactly match LENGTH length of alignment MIS number of mismatches in the alignment GAPS number of gaps in the alignment Q_ST query start Q_END query end H_ST hit start H_END hit end E-VAL e-value (the lower the better) P-SCO p-score (the higher the better) 55

56 Take-home Message What we have covered today covers only a little of what is needed You can use what you used today to expand on your knowledge Use different websites to get help Go through the examples on your own time We can use bash shell scripting to submit jobs to the cluster environment 56

57 University of Chicago Initiative in Biomedical Informatics Computation Institute Thank you! Please evaluate our performance for this workshop: Evaluation [ ] Surveys created with REDCap. Survey Hosted at ibi