Overview Shell programming is a way of taking several command line instructions that you would use in a Unix command prompt and incorporating them into one program. There are many versions of Unix. Some of the major ones include: AIX (IBM) HP/UX (Hewlett Packard) Linux (free and produced by numerous organizations including RedHat, Ubuntu and others) Solaris (Sun Microsystems) The most common way of accessing a Unix environment is to use a software tool to telnet or ssh into a server via a terminal. A terminal is similar to a Windows command prompt, however it is a command prompt to a remote computer (Unix server). The software used creates the connection and launches the terminal. Some of the terminal software products used are: PuTTy (shown below) Hummingbird Cygwin Remote terminal methods rlogin and rsh. These allow remote login and remote execution, but are not secure and do not provide a terminal. So, no further action can be taken telnet. Provides an unsecured terminal. ssh. Secure shell provides a secure (encrypted) terminal. Shell programming is referred to as scripting. Shell scripts, like all forms of scripting, are interpreted, not compiled. Unix does not have drives like Windows (C:, H:, etc.). Unix uses file systems, which are essentially the same thing as drives. The main difference is that in scripting you do not have to worry about drives, everything is relative to root. Page 1 of 19
Basic Shell s pwd cd <directoryname> cd / cd.. cd <directoryname/subdirname> cd /<directoryname> cd /<directoryname/subdirname> Present working directory. Answers the question, where am I? Change directory (folder). This is a relative change. If this directory does not exist where you currently are, then it will not work. Change to the root directory Go back (up) one directory Change to a directory (folder) more than one deep. This is a relative change. If this directory does not exist where you currently are, then it will not work. Change directory (folder). This is an absolute change. This directory must exist at the root. Change to a directory (folder) more than one deep. This is an absolute change. This directory must exist at the root. cd ~ Change to my home directory. Page 2 of 19
clear Clear the screen mkdir <directoryname> mkdir <dir1> <dir2> <dirn> ls Make one or more new directories List what s in this directory ls -l List in long format with more details Page 3 of 19
ls -lt ls -ltr List what s in this directory in long format in time order List what s in this directory in long format in reverse time order ls -R List this directory and its subdirectories. R stands for recursive. Page 4 of 19
rm <filename> Removes files. List multiple files (at the same level) if you want more than one file deleted. rm -r <directoryname> Removes recursively. This allows the removal of directories. cp <filename> <newfilename> cp R <dirname> <newdirname> Copies a file. If the file is copied to a different directory, the new file name is optional. Page 5 of 19
mv <filename> <newdirectory> mv <filename> <newfilename> Relocates a file. This can also be used to name the file in its new location. date Displays the date as a string (text). export <variable>=<value> echo <astring> echo $<avariable> Assigns a value to a variable. If the variable does not exist, it creates it and assigns the value. Export is NOT needed inside of a shell script. It is only used at the command line. Echoes (displays) the string or value of the variable. When using a Page 6 of 19
echo <astring> > <newfilename> echo $<avar> > <newfilename> echo <astring> >> <filename> echo $<avar> >> <filename> cat <filename> Echoes a string to a new file. If the file name already exists it replaces the file. Echoes and appends a string to a file. This means it adds it to the bottom Outputs the contents of a file to the screen. Shell Rules and Details Scripts (programs) are saved in a file. The file should have a.sh extension, however Linux will attempt to run anything you tell it to. The benefit of the extension is for user friendliness and organization. Relative vs. absolute paths. A path is a named location of a file or directory. If it does not include a drive letter or a starting \, it is a relative path. That means relative to the location it is referenced from. If it begins with a \, then it is absolute with respect to the drive I am currently located in. This means the path is relative to the root of the drive. Location and directional parameters Parameter Meaning / Directory separator or root directory. Current directory.. Parent directory See the examples below. Your Hypothetical Directory Structure Current Location Result /School cd homework Error. There is no homework directory here. /School/Math cd homework Changes to /School/Math/Homework /School/Math/Homework cd.. Goes back to /School/Math /School/Math/Homework cd /School/Science Changes to /School/Science Page 7 of 19
/School/Math/Homework cd School/Science Error. There is no School/Science directory here. /School/Math/Homework cd../ Goes back to /School/Math and down to /School/Math/ /School/Math/Homework cd / Error. There is no / directory. Wild Cards. Wild cards are like substitutions for characters. There are two wildcards: * means any number of characters? means exactly 1 character Wild cards can be used to make operations very efficient when naming conventions are used in conjunction with them. For instance, consider a situation where all homework filenames start with hwk whether they are Excel, Powerpoint or Word, and all report filenames start with rpt whether they are Excel, Powerpoint or Word. Furthermore, Imagine as part of the naming convention, all files had 4 digits in the name representing the date as mmdd. So, for example file names would look like this: File Name hwk0922math.xlsx hwk0928english.docx hwk1109ss.docx rpt0108english.docx rpt0310science.pptx Math homework for September 22 nd done in Excel English homework for September 28 th done in Word Social Studies homework for November 9 th done in Word English report for January 8 th done in Word Science report for March 10 th done in Powerpoint Using wild cards in conjunction with the naming convention would allow for some of the following efficient and powerful commands. See the examples below. Purpose rm hwk*.* Delete all homework files rm hwk*.xlsx Delete all homework done in Excel mv *.docx..\backups Move all Word files to the backups directory cp hwk110?ss.pptx..\backups Copy all Social Studies homework files done in Powerpoint from the first 9 days of November to the backups directory rm rpt1???*.* Delete all reports files of any kind for any class from any day in October, November or December. Syntax. Syntax is the grammar, vocabulary and spelling of any programming language. If you don t type commands, parameters and arguments the right way, your instructions can t be executed. In shell programming, because it is an interpreted language, there is no way of getting a syntax check ahead of time. Therefore, all scripts should get tested in a safe place first. You don t want a script that accidentally deleted the wrong things. The shell interpreter is case sensitive. Shell Syntax Tips. Follow the rules below for proper syntax. Directory separators are forward slashes, not back slashes. E.g., cd /school/math/homework Parameters (also called switches) are like arguments, but they are built in optional arguments that change the way a command will work. They often use dashes, but can also use forward slashes. Piping is a common way to pass the output from one command to another. To pipe use the character. ls l grep i customer Output can be used to create new files or append (add after bottom) of existing files. Use one or two greater than signs (>) to create (or recreate) or append. Create new file: ls ltr > filelist.txt Append to existing file: echo John,Jones,Bardonia,NY,10954 >> customers.csv White Space is any space between characters including blank lines, tabs, and spaces. Arguments need space before them. E.g., cp hwk0919ss.docx hwk0919ssbackup.docx Arguments passed into scripts can be referred to inside the script in numerical order with dollar signs. For instance, if you name your script createbackup.sh and call it like this: createbackup.sh d Page 8 of 19
Then inside the script, $1 has the value -d. This can be used to run a script with different options. Filenames can only have letters, numbers, underscores, dashes and a handful of other characters in them. The following characters are NOT allowed in filenames: < (less than) > (greater than) : (colon) " (double quote) / (forward slash) \ (backslash) (vertical bar or pipe)? (question mark) * (asterisk) ` (back quote or tic) Variables. When assigning a value to a variable on the command line we use the export command. From within a script export is not needed. line: export mytempvar=22 Script: mytempvar=22 When referencing a variable use $. See below. echo $mytempvar Special Characters. Some characters or strings have special meaning. New line: \n Tab: \t Wild cards: *?. Separators: \ / Delimiters: Pipes and output: > Beginning of Line: ^ End of Line: $ wrapper: ` In order to refer to these characters rather than have them be interpreted for what they mean, place a back slash before them. This is called escaping. See below. In this example a file contains names of customers separated by commas. If you wanted to display the list with slashes instead of commas you would need to escape the slashes because they are special characters which normally serve as separators. If they are not escaped they will give undesired results. The first time the command is run the / is properly escaped. In the second instance, it is not. See what happens. Page 9 of 19
Advanced Shell s date + <yourformat> Displays the date in a format of your choosing. %y represents year as 2 digits %Y represents year as 4 digits %m represents month as 2 digits %h represents month as a 3 letter abbreviation %d represents date as 2 digits %%H represents the hour as 2 digits %%M represents the minute as 2 digits %%S represents the second as 2 digits # Comments are created in shell scripts by beginning them with a #. Everything following the # on that line is ignored by the compiler. Page 10 of 19
cut f <fieldnumber> -d <delimiter> This must be used in conjunction with another command such as cat. The other command is piped into cut. The cut command treats each line it receives like a record in a database table. The f parameter provides the field you want cut out of the record by number. The d parameter tells cut how you want to delimit the fields. A delimiter is a character that separates data into fields. In examples where there are commands or files that when output to the screen produce rows and columns that use space to separate data of varying sizes, cut is not practical. In this case, use the awk command. This is explained later. In the example below, a file is displayed with cat. In the second window the cut command is used to separate the data with a comma delimiter and show only the state (5 th ) field. Page 11 of 19
sort sort r sort k <keyfieldnumber> -t <delimiter> The sort command is used in conjunction with other commands via a pipe. This sorts output in alphanumerical order. Numbers will come before letters. The r option sorts in reverse order. The k option allows you to pick a field by number as a key to sort by. The default is for the interpreter to assume the fields are separated by space. Use the t option to define your own delimiter to separate fields. uniq Provides a uniq version of the output. This is typically used with sort. You pipe into uniq. Page 12 of 19
grep <text> <filename> grep i <text> <filename> <command> grep <text> <command> grep v <text> The grep command is a filter used to pull lines out of files or command output. The i option makes it ignore case. The v option excludes text from the output. It can be used with wild cards. See below. Get all states starting with new or New from a file: grep -i new.* states.csv Exclude all blank lines from a file from a file: grep -v ^$ data.csv <command> sed s/<original>/<new>/g Sed is a powerful tool that can do many things. The most common use is to substitute one string for another. Everything between the first forward slash and the second forward slash will be replaced by everything between the second forward slash and the third forward slash. The s at the start tells sed that you want to substitute. The g tells sed to do this globally (all occurrences on the line). <command> awk print {$<fieldnumber>} Awk is another tool that can do many things. The most common use is to extract fields from output based on white space. It can pull pieces of data out of a command that produces columns and rows with varying sizes of data. This is more practical than cut in these situations. Page 13 of 19
wc wc -l ls l wc -l cat <filename> wc -l The wc command is a word counter. Words are anything separated by white space (blanks, tabs, etc.). The l option makes it a line counter. You can use this to get the number of lines in a file or the number of files in a directory or the number of rows of output. <command> bc `expr <somemathmaticalexpression>` The bc command converts text into a number which is critical when assigning values to a variable that may be used for calculation. You can also use bc to pipe in a mathematical expression that uses decimals. The syntax requires an echo of the expression as a string preceded by assigning an environment variable called scale a value equivalent to the number of decimal places. See the examples below. answer=`echo scale=2; 3.88 / 2.2 bc` answer=`echo scale=4; $x *.75 bc` The expr (expression) command is used to calculate using basic mathematical operators and integers only. This is often used to assign a value to a variable. The entire command and its arguments needs to be in back quotes. Page 14 of 19
id Displays the ID of the user running the command. who Displays a list of all instances of all users connected along with the time they logged in. netstat -n The netstat command provided network statistics. The n option provides this in numeric form. This means ip addresses instead of hostnames. The output is (from left to right): Proto This is the protocol of the connection. For example, tcp is the most common. Recv-Q This is the number of bytes received by the socket on this connection. Send-Q This is the number of bytes sent from the socket on this connection. Local Address This is the ip address of this host (server) and the socket. Foreign Address This is the ip address of the remote host connected to this host (server) and the socket. State This is the state of the connection. It could be waiting, closing, established, etc. iostat The iostat command provided io (input and output) statistics. IO is the reading and writing of data into and out of memory, disk, printers and other devices. Page 15 of 19
mpstat mpstat P <#> mpstat P ALL Displays statistics on multiple processors. You can choose a specific processor number from 0 to n where n is one less than the number of processors on the server. Alternatively use ALL to include all processer statistics. df -k The df command reports file system disk space usage. The k option shows usage in KB. free The free command reports the amount of total, used and free memory and swap space. Swap space is memory borrowed from disk temporarily. A computer generally uses this for items that are still open, but not in use in order to free up RAM (which is much faster) for active processes. Page 16 of 19
tail -<somenumberoflines> <filename> <command> tail -<somenumberoflines> <command> head -<somenumberoflines> head -<somenumberoflines> <filename> The tail command shows only the last n lines from your output. You can display the end of a file or pipe a command into tail. The head command shows only the first n lines from your output. You can display the top of a file or pipe a command into head. sleep <somenumberofseconds> if [<TrueFalseExpression>] then <command1> <commandn> else fi Pauses the shell script process for a defined number of seconds. This is a great way to run a script that monitors and writes output to a log. Use this to only execute certain commands when a condition is met. Use == when comparing strings. Use gt, -lt, -eq for >, <, == when comparing numbers. while [<TrueFalseExpression>] do <command1> <commandn> done Use this to only execute certain commands when a condition is met. Use == when comparing strings. Use gt, -lt, -eq for >, <, == when comparing numbers. Page 17 of 19
Examples Systems programmers are often tasked with producing reports and logs that help support and monitor the servers and applications an organization uses. Below are some examples of the tricks they employ to get information from the server. Get the percentage of CPU time currently being taken by users, converted from a string to a number. Get the number of logged in users, converted from a string to a number. Get the IP address of all remote tcp connections, the port of all remote tcp connections, and the local ports in use. Page 18 of 19
Get the number of bytes still available in the root file system. Page 19 of 19