Practical 02 Bash & shell scripting 1
imac lab login: maclab password: 10khem 1.use the Finder to visually browse the file system (single click opens) 2.find the /Applications folder 3.open the Utilities folder 4.open the Terminal application 2
Notes on homework assignments add name to text file at top make sure you don t get weird HTML characters eg & should really be & >> should be >> email subject: include PHY598: assignment 02 reverse BAG BAG="towel teacup guide babelfish" BAG2="" for item in $BAG do BAG2="${item} ${BAG2}" done echo ${BAG2} babelfish guide teacup towel 3
Notes as PDF Blackboard http://becksteinlab.physics.asu.edu/pages/courses/ 2013/SimBioNano/02/... almost complete. 4
Input/Ouput redirection >, >>, < cat < FILE cat FILE1 FILE1 > FILE3 cat FILE1 FILE2 >> FILE3 What happens with cd p01 cat 1AKE.pdb 1AKE.pdb > tmp/new.pdb cat 1AKE.pdb 1AKE.pdb >> tmp/new.pdb cat 1AKE.pdb 1AKE.pdb > tmp/new.pdb Use ls -l... 5
Counting file contents: wc wc [options] FILE wc < FILE wc -l FILE $ wc 1AKE.pdb 4424 50152 358344 1AKE.pdb lines words characters Count the lines in 1AKE.pdb and tmp/new.pdb $ wc -l 1AKE.pdb 4424 1AKE.pdb $ wc -l tmp/new.pdb 8848 tmp/new.pdb 6
Downloading with curl Instead of using a web browser, use curl -O URL curl -O http://becksteinlab.physics.asu.edu/ pages/courses/2013/simbionano/01/1ake.pdb cat URL Note: type the URL on a single line! curl URL > name curl URL wc -l creates a pipeline between commands; see later slides. 7
Variables BASEURL="http://becksteinlab.physics.asu.edu/ pages/courses/2013/simbionano/" PRACTICAL=01 FILE=1AKE.pdb echo "Practical ${PRACTICAL}: " echo "downloading file ${FILE}" URL="${BASEURL}/${PRACTICAL}/${FILE}" curl -O "${URL}" 8
Editing files vi editor ( vim ) vi hello.sh 9
A simple shell script #!/bin/bash # usage: hello.sh NAME # Greets NAME. NAME="$1" echo "Hello ${NAME}!" Run the script: bash./hello.sh your_name 10
Executable scripts A script can only be directly executed if the executable flag is set: chmod a+x./hello.sh Check with ls -la./hello.sh: -rwxr-xr-x 1 oliver staff 91 Jan 14 14:15 hello.sh Run the script:./hello.sh your_name 11
Argument processing white space splits arguments:./hello.sh your name./hello.sh your name individual positional arguments can be accessed as $1, $2, $3,..., ${10},... all arguments: $* optional arguments (-h, -f FILE,...): use the getopts (shell builtin) getopt (GNU, not always available) functions (advanced) 12
Write your own course downloader cdl Should work like this: cdl.sh 01 vim_basics.txt downloads the file http://becksteinlab.physics.asu.edu/ pages/courses/2013/simbionano/01/vim_basics.txt Start your script with a standard header #!/bin/bash # usage: cdl.sh PRACTICAL FILENAME # downloads FILENAME for PRACTICAL # example: # cdl.sh 01 vim_basics.txt 13
Course DownLoad script cdl $ vi cdl.sh #!/bin/bash # usage: cdl.sh PRACTICAL FILENAME # downloads FILENAME for PRACTICAL BASEURL="http://becksteinlab.physics.asu.edu/ pages/courses/2013/simbionano" PRACTICAL="$1" FILE="$2" echo "Practical ${PRACTICAL}: " echo "downloading file ${FILE}" URL="${BASEURL}/${PRACTICAL}/${FILE}" curl -O "${URL}" Note: The URL should be on a single line! 14
Download script make executable chmod a+x cdl store in your ~/bin directory (and name it just cdl so that you don t have to type.sh all the time... in Unix lazy is good :-) ) mkdir ~/bin cp cdl.sh ~/bin/cdl try it ~/bin/cdl 01 1AKE.pdb ~/bin/cdl 01 vimqrc.pdf 15
Environment variables The shell sets a number of variables with special meaning. See all of them with env less The PATH contains a list of directories (separated by : ) where the shell looks for commands ( executables ): echo $PATH.:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/ local/bin:/usr/x11/bin 16
Environment variables Add your own $HOME/bin directory to the PATH PATH=${PATH}:$HOME/bin export PATH Make this change permanent by adding the line export PATH=${PATH}:$HOME/bin to the end of the shell startup file ~/.bashrc try it: open a new terminal which cdl cdl 01 1AKE.pdb cdl 02b extra.txt instructions for writing a more challenging script 17
Pipes: Unix commands can be joined: standard output standard input... command command... cat FILE wc -l cat 1AKE.pdb tmp/new.pdb wc -l 18
Filter pipe lines The following commands are very useful to know for processing data in pipelines: grep or egrep (egrep is preferred) sort uniq cut sed awk The next few slides only give you a rough idea what these commands are good for. Read the man pages! (e.g. man sort) sed and awk are pretty complicated but basic use is fairly straightforward 19
Filters: grep and egrep egrep == grep -E prints lines matching a pattern ( regular expression ) cat FILE egrep PATTERN cat 1AKE.pdb egrep '^ATOM' egrep '^ATOM.* A *[0-9]+' 1ake.pdb \ > 1ake_A.pdb find all lines starting with ATOM extract chain A from this PDB file (Note: can fail if the chain identifier is not surrounded by white space... do not rely on it.) 20
Filters: grep and egrep Count the atoms in 1AKE cat 1AKE.pdb egrep '^ATOM' wc -l Count the residues in 1AKE (each residue contains a single CA atom) cat 1AKE.pdb egrep '^ATOM.*CA' wc -l 21
Filters: sort and uniq alphabetical sort cat FILE sort numerical sort cat FILE sort -n select fields for sorting (white space separated) cat FILE sort -n -k2,2 only keep unique entries cat FILE sort uniq count unique entries ( histogram ) cat FILE sort uniq -c 22
Filters: cut Cut fields or columns from a FILE or input stream cat FILE cut OPTIONS cut OPTIONS FILE data from columns N-M and X-Y cut -c N-M,X-Y FILE Example: get all residue names from a PDB file: cat 1AKE.pdb egrep ^ATOM \ cut -c 18-20 data from fields N-M separated by white space cut -f N,M -d ' 23
sed and awk sed is a stream editor (similar to some of the commands used in vi) common use is search and replace: cat FILE sed 's/search/replace/g' awk applies a small program to each input line (it is its own programming language) common use: working on separated column data: cat FILE \ awk '/REGEX/ {awk-command; \ awk-command;...}' cat /etc/passwd awk -F ':' '$1 ~ /^[a-z].*/ {print $1}' Finds all usernames that start with a letter. 24
Bash functions small piece of code that can be used repeatedly function NAME () { } local a= $1 b= $2...... return VAL example: function revjoin () { # usage: revjoin a b --> ba local a= $1 b= $2... echo ${b}${a} } revjoin hot dog 25
see Introduction to Unix More... http://becksteinlab.physics.asu.edu/pages/unix/introunix/ p01_unix.html or as PDF file cdl IntroductiontoUNIX.pdf assignment (will be posted on Blackboard) 26