CS 25200: Systems Programming Lecture 11: *nix Commands and Shell Internals Dr. Jef Turkstra 2018 Dr. Jeffrey A. Turkstra 1
Lecture 11 Shell commands Basic shell internals 2018 Dr. Jeffrey A. Turkstra 2
man Interface to reference manuals man [options] [section] file Examples man cp # manual page for copy cmd man -k printf # search pages for keyword man 3 exec # man page from section 3 2018 Dr. Jeffrey A. Turkstra 3
Sections 1: Executable programs or shells 2: System calls 3: Library calls 4: Special fles (usually in /dev) 5: File formats and coventions 6: Games 7: Miscellaneous 8: System administration commands 9: Kernel routines (non standard) 2018 Dr. Jeffrey A. Turkstra 4
ls List directory contents ls [options] [file(s)] Examples... ls # list files in pwd ls -al # hidden (all) files, long listing ls -R # recursive 2018 Dr. Jeffrey A. Turkstra 5
mkdir Make directories mkdir [options] directory Examples... mkdir bob mkdir -p bob/is/deep # leading path mkdir -m 700 bob # set the mode 2018 Dr. Jeffrey A. Turkstra 6
cp Copy fles and directories cp [options] src dest Examples... cp bob joe cp -r dir1 dir2 # recursive cp -rp dir1 dir2 # preserve perms/times/etc cp -x dir1 dir2 # do not cross filesystems 2018 Dr. Jeffrey A. Turkstra 7
mv Move (rename) fles mv [options] src dest Examples... mv bob mike mv dir1 dir2 mv -n bob exists # do not overwrite mv -i bob exists # interactive 2018 Dr. Jeffrey A. Turkstra 8
rm Remove fles or directories rm [options] file(s) Examples... rm a.txt b.txt rm -f a.txt # force, no error message rm -r dir1 # recursive rm -rf dir1 # be careful rm -rf ~ /some/path 2018 Dr. Jeffrey A. Turkstra 9
whereis Locates the binary, source, and manual fles for the given command whereis [options] name Only works if the requisite db has been generated Example whereis ls 2018 Dr. Jeffrey A. Turkstra 10
which Shows the full path to a command which [options] programname Examples which ps which ls which sudo 2018 Dr. Jeffrey A. Turkstra 11
tee Check out the unix tee command any_command tee save_out Saves a copy of all output (sent to standard out) in the fle save_out tee save_in any_command Saves a copy of all input (sent to standard in) in the fle save_in 2018 Dr. Jeffrey A. Turkstra 12
head Collects the frst n lines of a fle with n defaulting to 10 if unspecifed head [-n] file Examples head -30 yuk # top 30 lines head yuk # top 10 lines head * # top 10 lines of every file 2018 Dr. Jeffrey A. Turkstra 13
tail tail [+/-[n] [b c l] [-f]] file delivers n units from the fle +n counting from the top -n counting from the end n defaults to -10 if unspecifed counting by b blocks c characters l lines (default) -f means follow - infnite trailing output (use ctrl-c to stop) Has bufer limitations - see the man page 2018 Dr. Jeffrey A. Turkstra 14
Examples tail +10 yuk # all lines beyond line 10 tail -30 yuk # last 30 lines tail -30c yuk # last 30 characters tail -30f yuk # last 30 lines, continuing outputting any added lines tail -30 * # last 30 lines of all fles 2018 Dr. Jeffrey A. Turkstra 15
cut Used to make vertical cuts across a fle cut -flags columns or field filename Useful fags -c characters -d feld delimiter -f felds See man page for more information 2018 Dr. Jeffrey A. Turkstra 16
Examples Given a fle data, 12345 7890 abcd efgb This is line one this is no big deal and this script, #! /bin/bash cut -c1-5,8- data echo '------' cut -d' ' -f2-3 data exit 0 We get this output: 12345890 abcd efgb This line one this no big deal ------ 7890 abcd is line is no 2018 Dr. Jeffrey A. Turkstra 17
Another example Here's another example: #! /bin/bash DAY_OF_WEEK="$(date cut -d' ' -f1-1)" MONTH="$(date cut -d' ' -f2-2)" DAY="$(date cut -d' ' -f3-3)" YEAR="$(date cut -d' ' -f6-)" echo "Date: $(date)" echo "Month: ${MONTH}" echo "Day: ${Day}" echo "Year: ${Year}" echo "Day of the week: ${DAY_OF_WEEK}" exit 0 Which outputs Date: Mon Jul 22 16:01:17 EST 1996 Month: Jul Day: 22 Year: 1996 Day of the week: Mon 2018 Dr. Jeffrey A. Turkstra 18
paste Used to combine lines from two fles together paste [-dlist] file1 file2 By default concatenates corresponding lines of the fles together using a tab as the separator Example: paste -d" " x y z concatenates the corresponding lines of the fles x, y, and z together using the list of separators circularly. In this case the list only contains a single space. 2018 Dr. Jeffrey A. Turkstra 19
More examples paste -s [-d list] file1 file2 merges lines together serially (one fle at a time) paste -s -d" \n" yuk pastes each pair of lines in the fle yuk together the list specifed with -d is a space followed by a newline See man page for more options and information 2018 Dr. Jeffrey A. Turkstra 20
wc Word count wc -[c w l] file Used to count -c characters -l lines -w words (separated by whitespace) Default is all three 2018 Dr. Jeffrey A. Turkstra 21
Example #! /bin/bash Output: wc x.c wc -l x.c 301 878 8382 x.c wc -w x.c 301 x.c wc -c x.c 878 x.c NL=$(wc x.c) 8382 x.c echo ${NL} 301 878 8382 x.c echo "\"${NL}\"" " 301 878 8382 x.c" LL=$(wc -l < x.c) " 301" echo "\"${LL}\"" 301 echo ${LL} exit 0 2018 Dr. Jeffrey A. Turkstra 22
A quick look at sort Read the man page for more information A few very useful fags: -u unique lines only -txfeld separator x (default is whitespace) -b ignore leading blanks -r reverse sort -n numbers not characters -k sort on felds (up to 10 -k options allowed) Note: feld numbers begin with 1 2018 Dr. Jeffrey A. Turkstra 23
sort example This example, #! /bin/bash cat data echo sort data exit 0 Yields this output: 1 a 5 0 b 3 2 b 4 1 a 5 3 a 4 1 a 5 1 b 4 1 b 4 0 b 3 2 b 4 1 a 5 3 a 4 2018 Dr. Jeffrey A. Turkstra 24
Another example This example, #! /bin/bash cat data echo sort -u data exit 0 Yields this output: 1 a 5 0 b 3 2 b 4 1 a 5 3 a 4 1 b 4 1 b 4 2 b 4 0 b 3 3 a 4 1 a 5 2018 Dr. Jeffrey A. Turkstra 25
-u and -k example This example, #! /bin/bash cat data echo sort -u -k 2 data exit 0 Yields this output: 1 a 5 3 a 4 2 b 4 1 a 5 3 a 4 0 b 3 1 b 4 1 b 4 0 b 3 1 a 5 2018 Dr. Jeffrey A. Turkstra 26
Another -u and -k example This example, #! /bin/bash cat data echo sort -u -k 2,2 data exit 0 Yields this output: 1 a 5 1 a 5 2 b 4 0 b 3 3 a 4 1 b 4 0 b 3 1 a 5 2018 Dr. Jeffrey A. Turkstra 27
Specifying feld order This example, #! /bin/bash cat data echo sort -ur -k 2,2 -k 3,3 -k 1,1 data exit 0 Yields this output: 1 a 5 2 b 4 2 b 4 1 b 4 3 a 4 0 b 3 1 b 4 1 a 5 0 b 3 3 a 4 1 a 5 2018 Dr. Jeffrey A. Turkstra 28
Beating the dead horse This example, #! /bin/bash cat data echo sort -k 3,3 -k 1,1 -k 2,2 data echo sort -k 3bn,3 -k 1bn,1 -k 2b,2 data exit 0 Yields this output: 1 a 5 3 a 14 11 b 4 11 b 4 11 b 4 1 a 5 12 c 40 12 c 40 3 a 14 2 a 40 2 a 40 2 a 40 21 c 51 1 a 5 12 c 40 3 a 14 21 c 51 21 c 51 2018 Dr. Jeffrey A. Turkstra 29
awk Pattern scanner and processing language Sequence of rules... pattern {action} default {action} Pattern is a regular expression Action is a sequence of statements to execute when pattern is matched 2018 Dr. Jeffrey A. Turkstra 30
awk Good for simple text manipulation awk {print $1} somefile.txt Google awk for some good tutorials 2018 Dr. Jeffrey A. Turkstra 31
sed Stream editor sed s/current/new/g somefile > newfile Replace all instances of current with new, output to newfle 2018 Dr. Jeffrey A. Turkstra 32
find Search for fles in a directory hierarchy find [options] starting_dir expression Search directory tree for expression, applying optional tests and actions Examples find. -name *.conf -print find. -name *.conf -exec chmod o+r {} \; 2018 Dr. Jeffrey A. Turkstra 33
UNIX Organization *nix has multiple components Scheduler decides when and for how long a process should run File system provides a persistent storage mechanism Virtual memory address space isolation Networking Windowing system Shells and applications 2018 Dr. Jeffrey A. Turkstra 34
Shells A shell is basically a command interpreter. It provides an interface between the user and the computer (operating system). Shells may be graphical (explorer.exe, for instance) or text- based - often times called a CLI or command line interface 2018 Dr. Jeffrey A. Turkstra 35
Shells cont When we write a shell script, the frst line of the fle tells the operating system which shell to use. Some common *NIX shells include: #! /bin/sh Bourne shell #! /bin/csh C-Shell #! /bin/ksh KornShell (more powerful) #! /bin/bash Bourne-Again SHell 2018 Dr. Jeffrey A. Turkstra 36
Shells A shell is a program. We interact with an instance of it a process. Just like malloc() or anything other software, we can write our own Basic shells are made up of a parser and an executor Most shells have other subsystems as well 2018 Dr. Jeffrey A. Turkstra 37
Parser The process of reading a string of symbols conforming to a grammar For our shell, it results in a command table: $ ls -al grep me > file1 ls grep In:dft -al me Out:fle1 Err:dft 2018 Dr. Jeffrey A. Turkstra 38
Executor Creates processes for the command table entries Juggles fle descriptors to create pipes between process inputs and outputs Also redirects stdin, stdout, and stderr $ A b c d > out < in 2018 Dr. Jeffrey A. Turkstra 39
Other subsystems Environment variables Setting, printing, expanding Wildcards E.g., expanding a*a Subshells Backticks ` ` and $( ) 2018 Dr. Jeffrey A. Turkstra 40
Shell project Three parts Parser read command line and print command table Executor create processes and connect them with pipes Also handle I/O redirection Other subsystems wildcards, environment vars, subshells 2018 Dr. Jeffrey A. Turkstra 41
Questions? 2018 Dr. Jeffrey A. Turkstra 42