Practical 02. Bash & shell scripting

Size: px

Start display at page:

Download "Practical 02. Bash & shell scripting"

Robert Maxwell
5 years ago
Views:

1 Practical 02 Bash & shell scripting 1

2 imac lab login: maclab password: 10khem 1.use the Finder to visually browse the file system (single click opens) 2.find the /Applications folder 3.open the Utilities folder 4.open the Terminal application 2

3 Notes on homework assignments add name to text file at top make sure you don t get weird HTML characters eg & should really be & >> should be >> subject: include PHY598: assignment 02 reverse BAG BAG="towel teacup guide babelfish" BAG2="" for item in $BAG do BAG2="${item} ${BAG2}" done echo ${BAG2} babelfish guide teacup towel 3

4 Notes as PDF Blackboard /SimBioNano/02/... almost complete. 4

5 Input/Ouput redirection >, >>, < cat < FILE cat FILE1 FILE1 > FILE3 cat FILE1 FILE2 >> FILE3 What happens with cd p01 cat 1AKE.pdb 1AKE.pdb > tmp/new.pdb cat 1AKE.pdb 1AKE.pdb >> tmp/new.pdb cat 1AKE.pdb 1AKE.pdb > tmp/new.pdb Use ls -l... 5

6 Counting file contents: wc wc [options] FILE wc < FILE wc -l FILE $ wc 1AKE.pdb AKE.pdb lines words characters Count the lines in 1AKE.pdb and tmp/new.pdb $ wc -l 1AKE.pdb AKE.pdb $ wc -l tmp/new.pdb 8848 tmp/new.pdb 6

7 Downloading with curl Instead of using a web browser, use curl -O URL curl -O pages/courses/2013/simbionano/01/1ake.pdb cat URL Note: type the URL on a single line! curl URL > name curl URL wc -l creates a pipeline between commands; see later slides. 7

8 Variables BASEURL=" pages/courses/2013/simbionano/" PRACTICAL=01 FILE=1AKE.pdb echo "Practical ${PRACTICAL}: " echo "downloading file ${FILE}" URL="${BASEURL}/${PRACTICAL}/${FILE}" curl -O "${URL}" 8

9 Editing files vi editor ( vim ) vi hello.sh 9

10 A simple shell script #!/bin/bash # usage: hello.sh NAME # Greets NAME. NAME="$1" echo "Hello ${NAME}!" Run the script: bash./hello.sh your_name 10

11 Executable scripts A script can only be directly executed if the executable flag is set: chmod a+x./hello.sh Check with ls -la./hello.sh: -rwxr-xr-x 1 oliver staff 91 Jan 14 14:15 hello.sh Run the script:./hello.sh your_name 11

12 Argument processing white space splits arguments:./hello.sh your name./hello.sh your name individual positional arguments can be accessed as $1, $2, $3,..., ${10},... all arguments: $* optional arguments (-h, -f FILE,...): use the getopts (shell builtin) getopt (GNU, not always available) functions (advanced) 12

Write your own course downloader cdl Should work like this: cdl.sh 01 vim_basics.txt downloads the file http://becksteinlab.physics.asu.

13 Write your own course downloader cdl Should work like this: cdl.sh 01 vim_basics.txt downloads the file pages/courses/2013/simbionano/01/vim_basics.txt Start your script with a standard header #!/bin/bash # usage: cdl.sh PRACTICAL FILENAME # downloads FILENAME for PRACTICAL # example: # cdl.sh 01 vim_basics.txt 13

asu.edu/ pages/courses/2013/simbionano" PRACTICAL="$1" FILE="$2" echo "Practical ${PRACTICAL}: "

14 Course DownLoad script cdl $ vi cdl.sh #!/bin/bash # usage: cdl.sh PRACTICAL FILENAME # downloads FILENAME for PRACTICAL BASEURL=" pages/courses/2013/simbionano" PRACTICAL="$1" FILE="$2" echo "Practical ${PRACTICAL}: " echo "downloading file ${FILE}" URL="${BASEURL}/${PRACTICAL}/${FILE}" curl -O "${URL}" Note: The URL should be on a single line! 14

15 Download script make executable chmod a+x cdl store in your ~/bin directory (and name it just cdl so that you don t have to type.sh all the time... in Unix lazy is good :-) ) mkdir ~/bin cp cdl.sh ~/bin/cdl try it ~/bin/cdl 01 1AKE.pdb ~/bin/cdl 01 vimqrc.pdf 15

16 Environment variables The shell sets a number of variables with special meaning. See all of them with env less The PATH contains a list of directories (separated by : ) where the shell looks for commands ( executables ): echo $PATH.:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/ local/bin:/usr/x11/bin 16

17 Environment variables Add your own $HOME/bin directory to the PATH PATH=${PATH}:$HOME/bin export PATH Make this change permanent by adding the line export PATH=${PATH}:$HOME/bin to the end of the shell startup file ~/.bashrc try it: open a new terminal which cdl cdl 01 1AKE.pdb cdl 02b extra.txt instructions for writing a more challenging script 17

18 Pipes: Unix commands can be joined: standard output standard input... command command... cat FILE wc -l cat 1AKE.pdb tmp/new.pdb wc -l 18

19 Filter pipe lines The following commands are very useful to know for processing data in pipelines: grep or egrep (egrep is preferred) sort uniq cut sed awk The next few slides only give you a rough idea what these commands are good for. Read the man pages! (e.g. man sort) sed and awk are pretty complicated but basic use is fairly straightforward 19

20 Filters: grep and egrep egrep == grep -E prints lines matching a pattern ( regular expression ) cat FILE egrep PATTERN cat 1AKE.pdb egrep '^ATOM' egrep '^ATOM.* A *[0-9]+' 1ake.pdb \ > 1ake_A.pdb find all lines starting with ATOM extract chain A from this PDB file (Note: can fail if the chain identifier is not surrounded by white space... do not rely on it.) 20

21 Filters: grep and egrep Count the atoms in 1AKE cat 1AKE.pdb egrep '^ATOM' wc -l Count the residues in 1AKE (each residue contains a single CA atom) cat 1AKE.pdb egrep '^ATOM.*CA' wc -l 21

22 Filters: sort and uniq alphabetical sort cat FILE sort numerical sort cat FILE sort -n select fields for sorting (white space separated) cat FILE sort -n -k2,2 only keep unique entries cat FILE sort uniq count unique entries ( histogram ) cat FILE sort uniq -c 22

23 Filters: cut Cut fields or columns from a FILE or input stream cat FILE cut OPTIONS cut OPTIONS FILE data from columns N-M and X-Y cut -c N-M,X-Y FILE Example: get all residue names from a PDB file: cat 1AKE.pdb egrep ^ATOM \ cut -c data from fields N-M separated by white space cut -f N,M -d ' 23

24 sed and awk sed is a stream editor (similar to some of the commands used in vi) common use is search and replace: cat FILE sed 's/search/replace/g' awk applies a small program to each input line (it is its own programming language) common use: working on separated column data: cat FILE \ awk '/REGEX/ {awk-command; \ awk-command;...}' cat /etc/passwd awk -F ':' '$1 ~ /^[a-z].*/ {print $1}' Finds all usernames that start with a letter. 24

25 Bash functions small piece of code that can be used repeatedly function NAME () { } local a= $1 b= $ return VAL example: function revjoin () { # usage: revjoin a b --> ba local a= $1 b= $2... echo ${b}${a} } revjoin hot dog 25

26 see Introduction to Unix More... p01_unix.html or as PDF file cdl IntroductiontoUNIX.pdf assignment (will be posted on Blackboard) 26

Table of contents. Our goal. Notes. Notes. Notes. Summer June 29, Our goal is to see how we can use Unix as a tool for developing programs

Table of contents. Our goal. Notes. Notes. Notes. Summer June 29, Our goal is to see how we can use Unix as a tool for developing programs Summer 2010 Department of Computer Science and Engineering York University Toronto June 29, 2010 1 / 36 Table of contents 1 2 3 4 2 / 36 Our goal Our goal is to see how we can use Unix as a tool for developing