Lecture 03 Shell programming Part 1 CSC 320: Systems Programming
What is Shell? Shell is a UNIX term for the interactive user interface with an operating system. The shell is the layer of programming that understands and executes the commands a user enters. In some systems, the shell is called a command interpreter. A shell usually implies an interface with a command syntax (think of the DOS operating system and its "C:>" prompts and user commands such as "dir" and "edit"). Spring 2011-2012 Dr. Safwan Qasem 2
Linux Shell Linux has a variety of different shells: sh => Bourne shell, csh => C shell, ksh => Korn shell, tcsh => TC shell, bash => Bourne Again shell. Many Shells can exist in the same system The most popular and default shell in Linux is bash : Bash is the shell appearing in the GNU operating system. Bash is an sh-compatible shell that incorporates useful features from the Korn shell (ksh) and C shell (csh). Bash offers functional improvements over sh for both programming and interactive use. The GNU Project was launched in 1984 to develop a complete UNIX-like operating system. Spring 2011-2012 Dr. Safwan Qasem 3
Linux Shell, Cont. bash is not only an excellent command line shell, but a scripting language in itself. Shell scripting allows users to use the shell's abilities and to automate a lot of tasks that would otherwise require a lot of commands. For more details: A bash guide for beginners http://tille.garrels.be/training/bash/ List of bash commands http://ss64.com/bash/ Spring 2011-2012 Dr. Safwan Qasem 4
Programming vs. scripting. Programming languages are generally a lot more powerful and a lot faster than scripting languages. Programming languages generally start from source code and are compiled into an executable. This executable is not easily ported into different operating systems. A scripting language also starts from source code. An interpreter reads the instructions in the source file one by one and executes them sequentially, even if. No prior compilation is needed. Interpreted programs are generally slower than compiled programs. The main advantage is that they can be easily ported the source file to any operating system. bash is a scripting language. Other examples of scripting languages are Perl, Lisp, and Tcl. Spring 2011-2012 Dr. Safwan Qasem 5
Bash Programming Make sure you have UNIX username and password (Contact the Computer Center). Spring 2011-2012 Dr. Safwan Qasem 6
Log in! Bash programming 3 tries to get valid username and password right Show who is logged in w or who finger Logout! exit CTRL-D Spring 2011-2012 Dr. Safwan Qasem 7
ls, cd and pwd Start a Terminal: Menu: Application => Accessories => Terminal SQ@Linux:~$> pwd /home/sqasem SQ@Linux:~$> ls.... SQ@Linux:~$> cd.. SQ@Linux:~$> pwd /home SQ@Linux:~$> ls.... pwd: present work directory ls : list directory cd: Change directory Spring 2011-2012 Dr. Safwan Qasem 8
ls, cd and pwd, cont. SQ@Linux:~$> cd SQ@Linux:~$> pwd /home/sqasem SQ@Linux:~$> cd.. SQ@Linux:~$> pwd /home SQ@Linux:~$> cd. SQ@Linux:~$> pwd /home SQ@Linux:~$> cd ~ SQ@Linux:~$> pwd /home/sqasem SQ@Linux:~$> cd /usr/src SQ@Linux:~$> ls.... Spring 2011-2012 Dr. Safwan Qasem 9
Create and delete directory SQ@Linux:~$> mkdir./develop SQ@Linux:~$> ls.... (check that develop exists) SQ@Linux:~$> cd./develop SQ@Linux:~$> pwd /home/sqasem/develop SQ@Linux:~$> ls.... (check the contents of develop) mkdir: make directory rmdir: remove directory.... (notice entries. and file.. ) SQ@Linux:~$> cd.. SQ@Linux:~$> rmdir./develop SQ@Linux:~$> ls....(check that develop does not exist) SQ@Linux:~$> mkdir./develop Create the directory./develop again Spring 2011-2012 Dr. Safwan Qasem 10
Filenames rules File names that begin with a period character are hidden from ls, but are available to all other commands including ls -a. Many configuration files are hidden including shell configuration files in the home directory. SQ@Linux:~$> cd ~ SQ@Linux:~$> ls -a.... (check the contents of home directory) File names in Linux are case sensitive. The file names Toto" and toto" are different files. Linux does not recognize any "file extension". Ex: filename.exe can be text file, and filename.txt can be an executable. In file names avoid spaces. filename, file_name, file-name and file.name are way better than file name. Spring 2011-2012 Dr. Safwan Qasem 11
Access to documentation Commands are generally documented using the command man. SQ@Linux:~$> man man.... man will present the manual page of the specified entry using more or less. In Linux, the default is less, but can be overridden. man pages are subdivided into various sections: SQ@Linux:~$> man time.... SQ@Linux:~$> man 3 time No manual entry for time in section 3 See 'man 7 undocumented' for help when manual pages are not available. Spring 2011-2012 Dr. Safwan Qasem 12
Access to documentation The table below shows the section numbers of the manual followed by the types of pages they contain: 1 Executable programs or shell commands 2 System calls (functions provided by the kernel) 3 Library calls (functions within program libraries) 4 Special files (usually found in /dev) 5 File formats and conventions eg /etc/passwd 6 Games 7 Miscellaneous (including macro packages and conventions), e.g. man(7), groff(7) 8 System administration commands (usually only for root) 9 Kernel routines [Non standard] Spring 2011-2012 Dr. Safwan Qasem 13
File creation & output redirection SQ@Linux:~$> cd./develop SQ@Linux:~$> touch./myfirst_file.txt SQ@Linux:~$> ls -a.... [notice the file size 0] SQ@Linux:~$> ls a /usr/bin >./myfirst_file.txt.... [notice No output on the screen. Where is it??] SQ@Linux:~$> ls -a.... [notice the file size > 0] SQ@Linux:~$> wc l./myfirst_file.txt.... [notice the number of lines] SQ@Linux:~$> ls a >./myfirst_file.txt SQ@Linux:~$> wc l./myfirst_file.txt.... [notice the number of lines did not change] SQ@Linux:~$> ls a >>./myfirst_file.txt SQ@Linux:~$> wc l./myfirst_file.txt.... [notice the number of lines doubled. Why?] Spring 2011-2012 Dr. Safwan Qasem 14
file, more and less SQ@Linux:~$> more./myfirst_file.txt... SQ@Linux:~$> less./myfirst_file.txt... What are the differences? Use man to learn about them? [Reading Assignment] SQ@Linux:~$> file /etc/bash.bashrc... [What do you get?] SQ@Linux:~$> file /usr/bin/make... [What do you get?] SQ@Linux:~$> file /lib/libx86.so.1... [What do you get?] Spring 2011-2012 Dr. Safwan Qasem 15
ls and its options SQ@Linux:~$> file /etc/bash.bashrc... [What do you get?] Command $>ls Effect $>ls -a $>ls A $>ls -1 $>ls -l $>ls -g $>ls -G Show the hidden files (starting with `.` Like above without `.`. And `..` (-one) List one file per line (-L small letter) show details of files like -l, but do not list owner --no-group. In a long listing (-l), don't print group names. $>ls -h --human-readable. With -l, print sizes in human readable format (e.g., 1K 234M 2G) $>ls -d $>ls -s $>ls -S $>ls -r list directory entries instead of contents, and do not dereference symbolic links --size. Print the allocated size of each file, in blocks sort by file size --reverse. Reverse order while sorting $>ls -R --recursive. List subdirectories recursively Spring 2011-2012 Dr. Safwan Qasem 16
ls l output -rwxr-xr-x 1 sqasem sqasem 338 2011-12-09 11:51 shell_exam_fall_2011.sh -rw-r--r-- 1 sqasem sqasem 70 2011-12-09 16:21 test1.txt -rwxr-xr-x 1 sqasem sqasem 848 2011-12-09 11:04 testfile.sh -rwxr-xr-x 1 sqasem sqasem 269 2011-09-28 12:20 testnumbers.sh -rw-r--r-- 1 sqasem sqasem 0 2011-12-09 16:21 test.txt ---------- - ------ ------ --- ---------------- ----------------------- File Name +------- Modification Time +------------------ Size (in bytes) +----------------------- Group +-------------------------------- Owner +---------------------------------------------- File Permissions Spring 2011-2012 Dr. Safwan Qasem 17
Summary of less Commands Commands marked with * may be preceded by a number, N. Notes in parentheses indicate the behavior if N is given. Command Effect h H Display this help. q :q Q :Q Exit. e CR * Forward one line (or N lines). y * Backward one line (or N lines). f SPACE * Forward one window (or N lines). b * Backward one window (or N lines). /pattern * Search forward for (N-th) matching line?pattern * Search backward for (N-th) matching line. n * Repeat previous search (for N-th occurrence). N * Repeat previous search in reverse direction. Spring 2011-2012 Dr. Safwan Qasem 18
Directories found in UNIX systems Directory / Root of the file system /boot /bin /usr/bin Role and use All the files required for booting Linux on a system. Binaries which are absolutely essential to run Linux. Binaries for use by the system users /sbin & /usr/sbin The system-administration tools. mostly for use by the superuser. /lib /home /root /tmp /mnt, /media /opt /lost+found The libraries required by system-applications. (Like DLLs) All users home directories. User: smith, Directory /home/smith The home-directory for the super-user: root Applications can write temporary files here. peripherals and other file-systems are mounted here. The directory where optional software are installed. When a disk-check finds files which are damaged or which are not linked to any directory, they are recovered to this directory. Spring 2011-2012 Dr. Safwan Qasem 19
Directories found in UNIX systems /usr Everything related to users applications /usr/include The header-files required by programs for compilation. /usr/lib The libraries required by user-applications. /usr/src The source-code for the Linux kernel. /usr/local Files and applications for use on this particular machine. /usr/share Information that can be shared by most users. /usr/share/x11 Support files for the X Windows system /usr/share/dict Dictionaries for the spelling checker. Bet you didn't know that Linux had a spelling checker. See look and ispell. /usr/share/doc Various documentation files in a variety of formats. /usr/share/man The man pages are kept here. /usr/x11r6 Files needed by the X Window system. /var Files whose contents vary duriong run-time: /var/log The log-files of the system. /var/spool Directories for mail, news, printing and other queued work. Spring 2011-2012 Dr. Safwan Qasem 20
Directories found in UNIX systems /etc All the configuration files for the various software are stored here. /etc/passwd The passwd file contains the essential information for each user. It is here that users are defined. /etc/fstab The fstab file contains a table of devices that get mounted when your system boots. This file defines your disk drives. /etc/hosts This file lists the network host names and IP addresses that are intrinsically known to the system. /etc/init.d This directory contains the scripts that start various system services typically at boot time. Spring 2011-2012 Dr. Safwan Qasem 21
/dev Directories found in UNIX systems In Unix, the system devices are represented as entries in the filesystem in the special directory /dev. Every entry in this directory corresponds to one of the system devices (peripherals). It is possible to read from and write to devices the same way as for files. For example /dev/fd0 is the first floppy disk drive, /dev/sda (/dev/hda on older systems) is the first IDE hard drive. Spring 2011-2012 Dr. Safwan Qasem 22
Directories found in UNIX systems /proc The /proc directory is a virtual directory as it does not have representation in the file system. The /proc directory gives hooks to access the kernel status and configure it on runtime. It shows a set of numbers and some entries: The numbers correspond to the process-ids of the processes running on the system. The entries (like filesystems, consoles, cpuinfo, diskstats, version, ) permit access to the current configuration of the system. SQ@Linux:~$> /proc/cpuinfo Spring 2011-2012 Dr. Safwan Qasem 23
mv, cp and rm SQ@Linux:~$> cd ~/develop SQ@Linux:~$> mv./myfirst_file.txt./my_1st_file.txt SQ@Linux:~$> ls -a.... [notice the filename changed] SQ@Linux:~$> cp./my_1st_file.txt./my_2nd_file.txt SQ@Linux:~$> ls -a.... [notice: Now we have two files of the same size] Copy the file my_2nd_file.txt to your home directory? And check its existence. SQ@Linux:~$> cp./my_2nd_file.txt ~ SQ@Linux:~$> ls a ~.... [notice: Now we have two files of the same size] SQ@Linux:~$> rm ~/my_2nd_file.txt SQ@Linux:~$> ls a ~.... [notice: The file has been removed] Spring 2011-2012 Dr. Safwan Qasem 24
Cmd. Completion & history SQ@linux:~$ ls /bi<tab> SQ@linux:~$ ls /bin/<tab> Display all 134 possibilities? (y or n) n SQ@linux:~$ ls /bin/le<tab> SQ@linux:~$ ls /bin/less<tab> less lessecho lessfile lesskey lesspipe SQ@linux:~$ fire<tab> Use [Arrow up] at the prompt to move to previous commands Command history prints a numbered list of previous commands:!n Refer to command line n.!-n Refer to the current command minus n.!! Refer to the previous command. This is a synonym for `!-1'.!string Refer to the most recent command preceding the current position in the history list starting with string. Spring 2011-2012 Dr. Safwan Qasem 25
Commands: try & understand 1 http://ss64.com/bash/ (man pages) A alias Create an alias apropos Search Help manual pages (man -k) aspell Spell Checker awk Find and Replace text, database sort/validate/index B basename Strip directory and suffix from filenames bash GNU Bourne-Again SHell bc Arbitrary precision calculator language bg Send to background bzip2 Compress or decompress named file(s) C cal Display a calendar cat Concatenate and display the content of files cd Change Directory chgrp Change group ownership chmod Change access permissions chown Change file owner and group chroot Run a command with a different root directory clear Clear terminal screen cmp Compare two files comm Compare two sorted files line by line command Run a command - ignoring shell functions cp Copy one or more files to another location cron Daemon to execute scheduled commands crontab Schedule a command to run at a later time csplit Split a file into context-determined pieces cut Divide a file into several parts D date Display or change the date & time dc Desk Calculator declare Declare variables and give them attributes df Display free disk space diff Display the differences between two files diff3 Show differences among three files dir Briefly list directory contents dirname Convert a full pathname to just a path du Estimate file space usage Spring 2011-2012 Dr. Safwan Qasem 26
Commands: try & understand 2 E echo Display message on screen env Environment variables eval Evaluate several commands/arguments exit Exit the shell export Set an environment variable expr Evaluate expressions F fg Send job to foreground fgrep Search file(s) for lines that match a fixed string file Determine file type find Search for files that meet a desired criteria free Display memory usage ftp File Transfer Protocol fuser Identify/kill the process that is accessing a file G grep Search file(s) for lines that match a given pattern groups Print group names a user is in gzip Compress or decompress named file(s) H head Output the first part of file(s) help Display help for a built-in command history Command History hostname Print or set system name I id Print user and group id's ifconfig Configure a network interface J jobs List active jobs join Join lines on a common field K kill Stop a process from running killall Kill processes by name L less Display output one screen at a time let Perform arithmetic on shell variables ln Make links between files local Create variables locate Find files logname Print current login name logout Exit a login shell look Display lines beginning with a given string ls List information about file(s) lsof List open files Spring 2011-2012 Dr. Safwan Qasem 27
Commands: try & understand 3 M make Recompile a group of programs man Help manual mkdir Create new folder(s) mkfifo Make FIFOs (named pipes) more Display output one screen at a time mount Mount a file system mv Move or rename files or directories N netstat Networking information nice Set the priority of a command or job nl print file with line numbers nslookup Query Internet name servers interactively O open Open a file in its default application op Operator access P passwd Modify a user password paste Merge lines of files pathchk Check file name portability ping Test a network connection pkill Stop processes from running printenv Print environment variables ps Process status pwd Print Working Directory R read Read a line from standard input rename Rename files rev Reverse lines of a file rm Remove files rmdir Remove folder(s) S sdiff Merge two files interactively sed Stream Editor seq Print numeric sequences set Manipulate shell variables and functions sleep Delay for a specified time sort Sort text files split Split a file into fixed-size pieces ssh Secure Shell client (remote login program) sudo Execute a command as another user sum Print a checksum for a file suspend Suspend execution of this shell symlink Make a new name for a file sync Synchronize data on disk with memory Spring 2011-2012 Dr. Safwan Qasem 28
Commands: try & understand 4 T tail Output the last part of files tar Tape ARchiver tee Redirect output to multiple files test Evaluate a conditional expression time Measure Program running time times User and system times touch Change file timestamps top List processes running on the system tr Translate, squeeze, and/or delete characters type Describe a command U umask Users file creation mask umount Unmount a device unalias Remove an alias uname Print system information uniq Uniquify files unset Remove variable or function names users List users currently logged in V vi Text Editor W wait wait for a process to complete watch Execute/display a program periodically wc Print byte, word, and line counts whereis Search the user's $path, man pages and source files for a program which Search the user's $path for a program file while Execute commands who Print all usernames currently logged in whoami Print the current user id and name (`id -un') Wget Retrieve web pages or files via HTTP, HTTPS or FTP write Send a message to another user X xargs Execute utility, passing constructed argument list(s) xdg-open Open a file or URL in the user's preferred application. yes Print a string until interrupted. Run a command script in the current shell Spring 2011-2012 Dr. Safwan Qasem 29
History & cmd. completion SQ@Linux:~$> his<tab> SQ@Linux:~$> history... 4 cp./my_1st_file.txt./my_2nd_file.txt 5 ls a 6 cp./my_2nd_file.txt ~ 7 ls a ~ 8 rm ~/my_2nd_file.txt 9 ls a ~ 10 mv./myfirst_file.txt./my_1st_file.txt 11 ls a SQ@Linux:~$>!4.... [notice what happened] SQ@Linux:~$>!cp.... [notice what happened] Spring 2011-2012 Dr. Safwan Qasem 30
Output redirection Most command line programs display results on the screen, by using a facility called standard output. By default, standard output directs its contents to the display. To save data for further processing, standard output can be redirected to a file by using the ">" or ">>" characters: SQ@Linux:~$> ls a >./myfirst_file.txt Using >, the destination file is erased every time before writing new content. Using >> the new content is appended at the end of the destination file. Spring 2011-2012 Dr. Safwan Qasem 31
Output redirection >filename redirects just standard output to filename (Overwrite). >>filename redirects just standard output to filename (Append). 2>filename redirects just standard error to filename. >&filename redirects the standard output and error to the file called filename: cmd > f1 2> f2 redirects the standard output to f1 and standard error to f2. cmd tee f1 redirects the output to f1 and to the standard output. SQ@Linux:~$> last grep ^sqasem tee root-logins.txt SQ@Linux:~$> less root-logins.txt Spring 2011-2012 Dr. Safwan Qasem 32
Input redirection Many commands can accept input from the keyboard by using a facility called standard input. By default, standard input gets its contents from the keyboard. Like standard output, it can be redirected to read input from a file by using the <" character: SQ@Linux:~$> sort <./myfirst_file.txt It is possible to combine input and output redirection SQ@Linux:~$> sort < file_list.txt > sorted_file_list.txt Spring 2011-2012 Dr. Safwan Qasem 33
Input redirection SQ@linux:~$ sort <<endoft > Ali > taha > mohammad > barmuda > calife > zebra > imam > aamam > endoftext > endoft aamam Ali barmuda calife endoftext imam mohammad taha zebra The subsequent text up to the word endoft is used as input to the command in this case sort. Spring 2011-2012 Dr. Safwan Qasem 34
pipes user@host$> Cmd1 cmd2 Standard output of cmd1 is used as standard input of cmd2 SQ@linux:~$ ls al /usr/bin.... [notice what happened] SQ@linux:~$ ls al /usr/bin less.... [notice what happened] SQ@linux:~$ ls lt ~ head.... [Displays the 10 newest files in the home directory] SQ@linux:~$ du sort -nr.... [Displays a list of directories and how much space they consume, sorted from the largest to the smallest] SQ@linux:~$ find. -type f -print wc l.... [Displays the total number of files in the current working directory and all of its subdirectories] Spring 2011-2012 Dr. Safwan Qasem 35
Practice Bash commands SQ@linux:~$ apropos time > tzselect SQ@linux:~$ less./ tzselect... [What is the function of apropos?] SQ@linux:~$ ls -l awk '{ print "directory/file " $8 " has permission " $1}... [Notice the new formatting? Compare with ls l alone] SQ@linux:~$ df -h sort -rnk 5 head -3 awk '{ print "Partition " $6 "\t: " $5 " full!" }... [Notice the new formatting? Try the command part by part. Use history] SQ@linux:~$ Cal... [Notice the output] SQ@linux:~$ cal -3... [Notice the output] SQ@linux:~$ cal -m3... [Notice the output] SQ@linux:~$ cal y... [Notice the output] SQ@linux:~$ cal 1990... [Find your birthday in gregorian] Spring 2011-2012 Dr. Safwan Qasem 36
Practice Bash commands SQ@linux:~$ ls -l -h /bin cut -d' ' -f1,4,8 SQ@linux:~$ dc 100 0.5 * P SQ@linux:~$ yes I am coming in few minutes SQ@linux:~$ ls xargs more Spring 2011-2012 Dr. Safwan Qasem 37
Variables and quotes SQ@linux:~$ MY_NAME= Mohammad Ali' SQ@linux:~$ echo $MY_NAME Mohammad Ali SQ@linux:~$ echo Hello $MY_NAME " Hello Mohammad Ali SQ@linux:~$ echo Hello $MY_VAR' Hello $MY_VAR SQ@linux:~$ echo $HOME... [What do you get] SQ@linux:~$ echo $USER... [What do you get] Spring 2011-2012 Dr. Safwan Qasem 38
First Bash Script Start xemacs or emacs and type the following inside it: #!/bin/bash echo This is my first script Save the file Ctrl-x Ctrl-s Select the right directory and give a name, like: my_first_script.sh Try: SQ@linux:~$./my_first_Script.sh bash:./first_script.sh: Permission denied Spring 2011-2012 Dr. Safwan Qasem 39
First Bash Script Run the script using /usr/bash SQ@linux:~$/usr/bash./my_first_Script.sh This is my first script Make the script executable SQ@linux:~$chmod 700./my_first_Script.sh Check the script access permission SQ@linux:~$ls l./my_first_script.sh -rwxr-xr-x 1 userid userid 338 2011-12-09 11:51 my_first_script.sh Run the script directly SQ@linux:~$my_first_Script.sh...[What do you get?] my_first_script.sh : command not found SQ@linux:~$./my_first_Script.sh This is my first script Spring 2011-2012 Dr. Safwan Qasem 40