COMP2100/2500 Lecture 17: Shell Programming II

Transcription

1 [ANU] [DCS] [COMP2100/2500] [Description] [Schedule] [Lectures] [Labs] [Homework] [Assignments] [COMP2500] [Assessment] [PSP] [Java] [Reading] [Help] COMP2100/2500 Lecture 17: Shell Programming II Summary I work through a more substantial shell scripting example. Aims To demonstrate some of the processes behind developing a shell script. To introduce more features of the Bash language that weren't covered in the previous lecture. 1. The Problem Since student accounts have a strict quota and students are often running into trouble with the quota, it would be nice to be able to produce a list of the ten largest les in your account. These would be good places to start when deleting les to get back under quota. So our task is: Write a bash script which produces a list of the ten largest les in the user's area, together with their sizes. That's the original statement of the problem, and it's not particularly 1 of 15 01/05/15 09:57

2 precise. It will get modied as we go. This is one of the things that happens in any software project: once you understand the problem a little, the requirements must be changed, and they keep on changing throughout the project. [Richard's comment: Work really hard at getting the requirements right before you start designing and coding. For such a simple problem it is all too easy to start coding right away; jumping in early frequently leads to disaster.] 2. A Rough Strategy Here's a plan for how to do it. I'm remembering that in the previous lecture we had a way to list (and count) all les and directories, so I plan to start with that. 1. List all les and directories. 2. Weed out the directories to get a list of all les. 3. Add the size to each item in the list. 4. Sort this list into descending order of size. 5. Print the rst ten. Step 1 we know how to do already. We can try it at the command line before we even start writing a script. (But note the caveat in the previous lecture about the output including blank lines.) bash$ ls -R.: CVS bigles1 bigles2 bigles3 bigles4 bigles5 lec03.html CVS: Entries Repository Root bash$ ls -R grep -v ':$' CVS bigles1 bigles2 bigles3 bigles4 bigles5 lec03.html Entries Repository 2 of 15 01/05/15 09:57

3 Root I just ran that in the directory where I'm writing this, so you can see the les I'm working on. The les bigles1... bigles5 are the ve stages in the development of the script for this lecture. The CVS directory is where the CVS version control system keeps stuff it needs. We'll be covering version control - using a simpler system called RCS - later this semester. This is working, but it has a few problems. One is that it still lists the directories as well as the ordinary les. That's not too much of a problem, because there is a way to tell the difference. We'll be see that later. The bigger problem is that for les in the CVS subdirectory it's just listing their names, but not the path to them. And for this script to be useful, the user needs to know where the big les are, not just what their names are. So we need a way to display the full paths to the les we nd. I looked through the manual for the ls command, (by typing man ls but didn't nd anything useful. I could imagine having the script sort through and instead of throwing away all the lines ending in a colon, keeping them and sticking that directory path onto the start of each le name. But it sounds complicated, and there's a better way. 3. The du command There's a very useful command du which stands for disk usage. Without any options it just lists the size (in something called blocks, which are a different size on different computers) of a directory and all its subdirectories: bash$ du 4./CVS 34. Not particularly useful, but looking through the manual page (by typing man du) I found that there's an option -a to have it list sizes of all les as well as directories. Just what we want, and the important thing is that it lists each le with its full path (either from the current directory, or from the directory given on the command line). There's also a -k option which makes sure that the sizes are in blocks of 1024 bytes - K's. bash$ du -ka 1./CVS/Root 3 of 15 01/05/15 09:57

4 1./CVS/Repository 1./CVS/Entries 4./CVS 1./bigles1 22./lec03.html 1./bigles2 1./bigles3 1./bigles4 2./bigles5 33. bash$ du -ka /home/barnes/2001/comp2100/lectures/lec03/ 1 /home/barnes/2001/comp2100/lectures/lec03/cvs/root 1 /home/barnes/2001/comp2100/lectures/lec03/cvs/repository 1 /home/barnes/2001/comp2100/lectures/lec03/cvs/entries 4 /home/barnes/2001/comp2100/lectures/lec03/cvs 1 /home/barnes/2001/comp2100/lectures/lec03/bigles1 22 /home/barnes/2001/comp2100/lectures/lec03/lec03.html 1 /home/barnes/2001/comp2100/lectures/lec03/bigles2 1 /home/barnes/2001/comp2100/lectures/lec03/bigles3 1 /home/barnes/2001/comp2100/lectures/lec03/bigles4 2 /home/barnes/2001/comp2100/lectures/lec03/bigles5 33 /home/barnes/2001/comp2100/lectures/lec03 This is the sort of list we can work with. 4. Removing the sizes In order to weed out the directories from this list, we actually need to throw away those sizes. We'll get them back later when we only have ordinary les in our list. We can get rid of the sizes using the cut command from last lecture. A quick check on the manual page gives information on the command-line options: -f, --elds eld-list Print only the elds listed in eld-list. Fields are separated by a TAB by default. -d, --delimiter delim For -f, elds are separated by the rst character in delim instead of by TAB. (When you type man cut, the information is presented to you by the paging program less. This is useful for reading through large les: you can invoke it by typing less le. When you're inside less, you can use the space bar to move down a page, the Enter key to move down a line, the `u' key to move up a page, the `g' to go to the top of the le, `G' to go to the end. You can also search for a word by typing `/' followed by the word. This will highlight all occurrences, and you can move from one to the next by typing `n'. You get out by typing 4 of 15 01/05/15 09:57

5 `q'.) OK, so it looks like the numbers are separated from the le and directory names by TABs, so we can try bash$ du -ka cut -f2./cvs/root./cvs/repository./cvs/entries./cvs./bigles1./lec03.html./bigles2./bigles3./bigles4./bigles5. This is what we need for the next step. 5. Weeding out the directories Some of the lines we have there are directories rather than les, so we want to get rid of them and leave a list of just the les. At this point I think it's time to stop playing around interactively and start writing my script. The rst version, with nothing new in it, is: #!/bin/bash # Get a list of all les (and directories etc) les=$(du -ka cut -f2) # Print the list echo $les When I run this (after remembering to make it executable), I get: bash$ bigles1./cvs/root./cvs/repository./cvs/entries./cvs./bigles1./le c03.html./bigles2./bigles3./bigles4./bigles5. As you can see, the result looks a little different, but that's OK. It's a list of all the les and directories, ready for us to do some more work. Now we want to go through that list, one at a time, and only keep the real les. How to do this? Back to the manual pages. This time look at the manual for bash itself. It turns out that there's a very useful builtin 5 of 15 01/05/15 09:57

6 command called test, which evaluates an expression and returns a true or false answer (as zero or not-zero) as its return code. It can do comparisons similar to boolean expressions in Eiffel. It can also check whether a le exists, whether it is a directory, whether it is an ordinary le, and so on. To make it look even more like ordinary boolean expressions, it has two alternative forms: test expr and [ expr ]. (The spaces around the expression are important in the second form.) The command [ -d le ] returns 0 (true) if le is a directory, non-zero otherwise. Similarly [ -f le ] tells whether le is a normal le or not, and [ -e le ] simply tells whether le exists. We can use a for loop to examine the les on our list one by one, and use this test to select only the real les. Here's the second version of the script. #!/bin/bash # Get a list of all les (and directories etc) les=$(du -ka cut -f2) # echo $les # Write list of all regular les with sizes to a temporary le tmp=".tmp_bigles" for f in $les do if [ -f $f ] du -ka $f >> $tmp done # Print the list cat $tmp This excludes not just directories, but any other weird things that might be there. (Is this the right thing to do? Good question. That's where our requirements aren't really precise enough.) Running the new version of the script gives us: bash$ bigles2 1./CVS/Root 1./CVS/Repository 1./CVS/Entries 1./bigles1 23./lec03.html 6 of 15 01/05/15 09:57

7 1./bigles2 1./bigles3 1./bigles4 2./bigles5 A couple of points. The usual output redirection is with >. Using >> appends output to the end of the le named, rather than overwriting it. I got the sizes back by just calling du again. Now I've made a mistake here, which you'll see when I run the script again: bash$ bigles2 1./CVS/Root 1./CVS/Repository 1./CVS/Entries 1./bigles1 23./lec03.html 1./bigles2 1./bigles3 1./bigles4 2./bigles5 1./CVS/Root 1./CVS/Repository 1./CVS/Entries 1./bigles1 1./.tmp_bigles 24./lec03.html 1./bigles2 1./bigles3 1./bigles4 2./bigles5 Since my script just appends to.bigles, that le is just going to grow and grow. I need to delete it when I'm nished. And in order to be sure that I don't clobber something important, I'd better also check that it doesn't already exist before I do anything to it. So here's version 3. #!/bin/bash # Get a list of all les (and directories etc) les=$(du -ka cut -f2) # echo $les # Make sure we're not about to clobber someone's data tmp=".tmp_bigles" if [ -e $tmp ] 7 of 15 01/05/15 09:57

8 echo "$tmp already exists - exiting." exit 1 # Write list of all regular les with sizes to $tmp for f in $les do if [ -f $f ] du -ka $f >> $tmp done # Print the list cat $tmp # Remove the temporary le rm -f $tmp The [ -e $tmp ] test checks whether there is already a le with that name. If there is, we'll just stop, rather than messing up any data that might be there. (Chances are that any le with a name starting with ``tmp'' probably isn't important, but you never know. Better to leave it up to the user to decide what to do.) The command exit 1 terminates execution of our script with return code 1, which says that the script was unsuccessful. This is important: if anyone ever calls this script from another script, they need to be able to check the return value to tell whether it succeeded or failed. The other thing I added is the last line. As you know, the rm command removes les. The option -f stands for ``force''. In other words, do it anyway, even if the user has their account set up so that rm usually asks for conrmation from the user. (You can do that by editing your.cshrc and inserting the line alias rm 'rm -i'.) Let's try it. bash$ bigles3.tmp_bigles already exists - exiting. bash$ rm.tmp_bigles bash$ bigles3 1./CVS/Root 1./CVS/Repository 1./CVS/Entries 1./bigles1 24./lec03.html 1./bigles2 8 of 15 01/05/15 09:57

9 1./bigles3 1./bigles4 2./bigles5 6. Sorting the list Our next task is to take that list of le sizes and names, sort it into descending order (of size) and print the rst ten. This is much easier than you might think. We don't have to go back to our rst year notes and try to code the quick-sort algorithm in bash. The philosophy of shell scripting is to always look for a quick-and-dirty way to do things, and the quickest and dirtiest is to re-use code someone else has already written. The Unix system has hundreds, perhaps thousands of well-written utility programs for all sorts of applications. For most tasks you might want to do, you're probably not the rst person to want to do them. In our case, we want to sort some data - and we're certainly not the rst people to want to do that. The sort program exists just for this purpose, and has been tried and tested for many years. No matter how hard you try, you would be very unlikely to write a better generalpurpose sorting program. [Richard's note: `tried and tested' doesn't mean all that much. Better to have a proof that the algorithms used are correct.] Again, we check the manual page for the command options: -n Restrict the sort key to an initial numeric string, consisting of optional blank characters, optional minus sign, and zero or more digits with an optional radix character and thousands separators (as dened in the current locale), which will be sorted by arithmetic value. An empty digit string is treated as zero. Leading zeros and signs on zeros do not affect ordering. -r Reverse the sense of comparisons. So it looks as if we want sort -rn $tmp. This will read the lines from the le $tmp, sort them into reverse numeric order (i.e. descending order of size) and write the results to standard output. All that remains for us to do is to print the rst ten lines. We can do that with a pipe to the head command. So the nished script is: #!/bin/bash # Get a list of all les (and directories etc) 9 of 15 01/05/15 09:57

10 les=$(du -ka cut -f2) # Make sure we're not about to clobber someone's data tmp=".tmp_bigles" if [ -e $tmp ] echo "$tmp already exists - exiting." exit 1 # Write list of all regular les with sizes to $tmp for f in $les do if [ -f $f ] du -ka $f >> $tmp done # Sort the list and print the top ten sort -rn $tmp head -10 # Remove the temporary le rm -f $tmp Here's the result of running it. bash$ bigles4 24./lec03.html 2./bigles5 1./bigles4 1./bigles3 1./bigles2 1./bigles1 1./CVS/Root 1./CVS/Repository 1./CVS/Entries bash$ cd.. bash$ lec03/bigles4 29./lec02/lec02.html 24./lec03/lec03.html 17./lec19/lec19.html 16./lec26/lec26.html 15./lec09/lec09.html 14./lec17/lec17.html 14./lec11/lec11.html 13./lec12/lec12.html 13./lec04/lec04.html 10 of 15 01/05/15 09:57

11 12./lec23/lec23.html 7. Are we done yet? This script does pretty much what we wanted, but not quite. The initial requirements said list the ten biggest les in the user's area, but this script lists the ten biggest in the current directory and all its subdirectories. Running it from the user's home directory will do the job required, but it's worth thinking about. Why not extend it to allow the user to specify a directory on the command line, but still use the current directory as a default? It would also be nice to be able to change the number 10 to something else. To do this, we're going to have to read arguments from the command line. In the previous lecture we learned about the special variables which give access to the command line arguments. Now we're going to use them in a more sophisticated way. For this we need two new bash commands. The rst is shift. This throws away the rst command line argument (${1}) and moves all the others down one position, decrementing the value of ${#} at the same time. This is useful for loops which process one command argument at a time. The second new bash thing is the case construction, which is a bit like the inspect statement in Eiffel. It tries to match something against each of a list of regular expressions in turn, and carries out the action corresponding to the rst successful match. The regular expression syntax used is the same one as is used for le names by the ls command (which means it's not the same as the one for grep). (By the way, this expansion is called ``globbing''.) Here's the enhanced version of the bigles script. #!/bin/bash # bigles # List the n (or 10) largest les in the directory specied (or the # current directory by default), and all its subdirectories, together # with their sizes. # Version 5 # Ian Barnes, 1 February of 15 01/05/15 09:57

12 # Set default values debug= number="-10" # Parse command line arguments while [ ${#} -gt 0 ] do case ${1} in -*) # Assume that what's after the - is a number number=${1} ;; *) # It's a directory dir=${1} esac shift done [ $debug ] && echo number=${number} [ $debug ] && echo dir=${dir} # Get a list of all les (and directories etc) if [ ${dir} ] if [ -d ${dir} ] les=$(du -ka ${dir} cut -f2) else echo Error: ${dir} is not a directory! exit 1 else les=$(du -ka cut -f2) [ $debug ] && echo ${les} # Make sure we don't clobber any data tmp=".tmp_bigles" if [ -e ${tmp} ] echo Error: $tmp already exists! exit 1 # Write list of all regular les with sizes to $tmp 12 of 15 01/05/15 09:57

13 for f in ${les} do if [ -f ${f} ] du -ka ${f} >> ${tmp} done [ $debug ] && cat ${tmp} # Sort the list and print the top -${number} sort -rn ${tmp} head ${number} # Remove the temporary le rm -f ${tmp} [Richard's note: can the code duplication (the du cut line) be avoided?] There's one other new thing there. I have created a new variable debug, which is initially assigned the empty string (meaning ``False''). If I change the assignment statement at the start of the script to assign any non-empty value, the script will print a whole lot of helpful information about its progress. The notation statement1 && statement2 means ``Execute statement1. If the return code is `True' (= `success' = `zero') execute statement 2.'' In other words, it is shorthand for if statement1 statement2 Similarly you can write statement1 statement2 which means ``Execute statement1. If the return code is `False' (= `failure' = `non-zero') execute statement 2.'' This is particularly useful for tasks that might fail, and which should terminate the script if they do. do something dubious { echo Failed; exit 1; } [Richard's note: watch the spacing and punctuation. The space after { 13 of 15 01/05/15 09:57

14 is absolutely necessary, as is the ; just before the }. But in this case you don't need the space before the { or the space before the }, because here they are immediately preceded by other shell punctuation. It does makes sense (once you get used to it), but remember that these little things make the difference between your script working or not working.] What I've done is simpler: the test instruction, if given a single string, returns true if the string is non-empty, false otherwise. Exercises Exercise 1: Modify the script so that debug mode can be turned on by giving the script a -d option on the command line like this: bigles -d -17 ~. Exercise 2: Modify the script so that it prints a usage message and quits if the user gives the option -h or if there is more than one directory given, or if an unknown option is supplied (i.e. a minus sign followed by something which is not a number, `d' or `h'). Exercise 3*: Modify the script so that it uses sizes in bytes (not K) for sorting and output. (Hint: If you do ls -l le you will get a long line containing all the information you want, but a lot of other stuff too. The cut program won't do it the way we've used it so far, because the separators aren't tabs or single spaces, but sequences of varying numbers of spaces, which line the output up in columns 8 spaces wide. There are two possibilities I can think of. One is to use cut, but nd out about how to cut at particular character (= byte) positions rather than by elds. The other possibility is to learn enough about awk or sed to pull out just the parts you want.) Richard's notes for those who've made it this far Ian's notes show you the hard way to do it. Let's see the easy way. The nd command does all of the hard work. The following works on GNU/Linux: nd. -type f -printf "%k\t%p\n" sort -nr head -10 [Careful here: there's the `good old' version of nd that you'll nd on Solaris, and there's GNU nd, which has lots more options. You can do 14 of 15 01/05/15 09:57

15 the same thing with the old nd -- check out the -ls option -- but GNU nd has that extra printf and %k stuff that's so useful. Just be aware that using the options that are GNU-nd-specic makes your script less portable. As an exercise, use nd's -ls option (together with the cut command) to solve the problem in a way that works portably across Solaris and GNU/Linux.] What are the advantages and disadvantages of doing it my way? [ANU] [DCS] [COMP2100/2500] [Description] [Schedule] [Lectures] [Labs] [Homework] [Assignments] [COMP2500] [Assessment] [PSP] [Java] [Reading] [Help] Copyright 2005, Ian Barnes & Richard Walker, The Australian National University Version , Monday, 4 April 2005, 14:18: Feedback & Queries to comp2100@cs.anu.edu.au 15 of 15 01/05/15 09:57