Course Overview and Introduction to the UNIX Shell

Transcription

1 Course Overview and Introduction to the UNIX Shell CSC209: Software Tools and Systems Programming (Winter 2019) Furkan Alaca & Paul Vrbik University of Toronto Mississauga Week 1 1 / 54

2 Overview Course Objectives & Outcomes Course Evaluation Scheme and Workflow Academic Integrity and Effective Ways of Getting Help Preparing for CSC209 What is UNIX? Using a UNIX Shell to Run Programs and Access the File System Acknowledgement: These slides are derived, in part, from content written by Andrew Petersen and Dan Zingaro. 2 / 54

3 What is this Course About? Using a UNIX shell The shell is the programmable interface for interacting with a computers core services and kernel (i.e. the computers operating system). Developing more efficiently By creating programming tools for 1. debugging (gdb, valgrind), 2. building and automating (make), and 3. version control (git). 3 / 54

4 What is this Course About? Systems programming Interacting with the OS through 1. the file system, 2. system calls, 3. processes, and 4. communication. The C programming language will be our tool of choice but other systems-level languages exist (e.g. Go). 4 / 54

5 After taking this course, you should be able to... Diagram and explain the memory usage of an arbitrary C-program. Read/use/understand the man pages of unfamiliar system call or library function. Write command-line tools adhering to the UNIX philosophy that leverage: system calls, inter-process communication, the standard library (libc), and the network stack. Effectively use standard UNIX development tools (e.g. gdb, make, git, valgrind, gdb, bash) and command-line utilities (e.g. cat, less, grep). 5 / 54

6 Evaluation Scheme 5% Weekly lecture prep (due 10pm on Sundays) 10% Weekly tutorial exercises (due 10pm on Fridays) 5% Assignment 1 (C Basics and Pointers) 10% Assignment 2 (System Calls and Dynamic Memory) 10% Assignment 3 (Processes) 10% Assignment 4 (Communication) 10% Midterm Exam (Fri. Feb. 15, 5pm) 40% Final Exam (Min. 40% grade required to pass the course) 6 / 54

7 A Typical Week in CSC209 Prepare for class 1. Watch videos on PCRS. 2. Solve simple exercises (worth marks). Attend class 1. Solve more advanced problems in a supervised setting. 2. Ask questions during brief lectures. Execute Put your knowledge into action with assignments and weekly tutorials. 7 / 54

8 Assignment and Tutorial Submission Assignments will be submitted over git. 1. Repositories will be managed with MarkUs: 2. Each assignment or tutorial will have its own directory in your repository. 3. Starter code will be pushed to your repositories. 4. Marks will be made available through the MarkUs web interface. 8 / 54

9 Assignment and Tutorial Submission Marking will be automated. 1. Be extremely deligent with required file names, directory structure, and output format. 2. Ensure that your code compiles without warnings or errors, and can be run without crashing the lab PCs. 9 / 54

10 Code submissions that do not compile will receive a grade of ZERO. 10 / 54

11 Assignment Resubmission You will receive feedback in the form of test results soon after the submission deadline and be permitted a resubmit. 1. The feedback is not your mark: Some tests are worth more than others. 2. Some tests might not even be worth marks. 3. Do your best to fix the errors that we have identified. 11 / 54

12 Assignment Resubmission You will have 48 hours after the feedback is transmitted to correct errors and resubmit. Your assignment will receive the higher of 1. the grade on the original submission and 2. the grade on the resubmission, with a 10% deduction 12 / 54

13 Academic Integrity The work you submit must be your own, done without participation by others. It is an academic offence to hand in anything written by someone else without acknowledgement. Copying We have software which determines if two listings are isomorphic in some way (i.e. variable renaming or line permuting). Any code that you use that is not your own must have correct attribution. 13 / 54

14 Not Academic Offense It is OK to help a friend by 1. Explaining and/or clarifying important concepts. 2. Reviewing/modifying/practicing exercises from PCRS, lectures, and previous weeks tutorials together. 3. Helping each other understand error messages and man pages. 14 / 54

15 Other Resources Discussion Board mcs.utm.utoronto.ca/forum/ Facilitated Study Groups utm.utoronto.ca/asc/facilitated-study-groups-fsgs TAs Ask the TAs questions only after genuinely trying yourself. Do not ask them How do I do the assignment? They can help you understand tools/concepts that you need for completing the assignment, common mistakes, helpful techniques for debugging your code, etc. 15 / 54

16 Linux This course assumes some familiarity with Linux so 1. Get accustomed to using a text editor on Linux (vi, vim, pico). 2. Familiarize yourself with basic shell commands/utilities. 3. Learn to ssh into the lab machines to work remotely $ssh vrbikpau@dh2026pc01.utm.utoronto.ca (windows users install putty) 4. Install a Linux virtual machine on your PC or, 5. Dual-boot. 16 / 54

17 Figure: This is how cool you will feel after taking CSC / 54

18 C Programming This is not a C-programming course. You are expected to learn the syntax yourself. There are many resources. 18 / 54

19 Upcoming This week UNIX shell and compiling C programs 1. Tutorial 1 is due on Friday (Jan. 11) Next week Arrays and pointers 1. Lecture prep is due on Sunday Jan Tutorial is due Friday Jan Assignment 1 posted. 19 / 54

20 The UNIX Philosophy A simple summary of the UNIX philosophy (from A Quarter-Century of Unix by Peter H. Salus, 1994): 1. Write programs that do one thing and do it well. 2. Write programs to work together. 2.1 Expect that the output from your program will be used as input for another (e.g., by piping) 2.2 Do not require interactive input 3. Write programs that handle text streams, because that is a universal interface. 20 / 54

21 UNIX vs. Linux vs. UNIX-like UNIX is a proprietary OS developed by AT&T in 1969 which quickly gave rise to both free and commercial imitations such as BSD (with derivative macos), Linux, Solaris... When we say UNIX we generally refer to these UNIX-like OSs, which are often referred to colloquially as *nix. Linux is the most widely-used OS in the UNIX-like family: It runs on all sorts of devices: PCs, smartphones, printers, security cameras, wireless routers... We use Linux in this course. 21 / 54

22 Time System III & V family BSD (Berkeley Software Distribution) Bill Joy BSD family SunOS Xenix OS Microsoft/SCO GNU Richard Stallman CommercialUNIX HP-UX Minix Research UNIX 10.5 Bell Labs: Ken Thompson, Dennis Ritchie, et al. AT&T FreeBSD NetBSD NextStep 3.3 GNU/Hurd Linux 4.4 Linus Torvalds Andrew S. Tanenbaum Solaris Sun/Oracle OpenBSD Theo de Raadt Darwin UnixWare Univel/SCO macos Apple DragonFly BSD 4.8 Matthew Dillon i v 3 AIX IBM IRIX SGI Timeline of UNIX-like operating systems. 22 / 54

23 Examples of Common UNIX Tools 1. cd 2. mkdir 3. ls 4. cp, mv, rm 5. touch 6. who 7. diff 1. head, tail 2. cat 3. sort 4. grep 5. wc 6. tr (The commands in this column are for text filtering) 1. stdin (standard input) 2. stdout (standard output) 3. stderr (standard error) 4. pipes All of the above in magenta have a man page. 23 / 54

24 How to Learn Linux Learn by doing. Do not worry about memorizing stuff but rather look things up as needed. Resources 1. man name-of-command to learn how a command works. 2. man intro or man gittutorial to learn about topics 3. Arch wiki: (very useful even for users of other Linux flavours such as Ubuntu) 4. Our ultimate goal is to teach you how to learn on your own. 24 / 54

25 Figure: How you will feel when you find an answer in the man pages. 25 / 54

26 Figure: Even the man command has its own man page: Check man man. 26 / 54

27 Consider the following shell command: $ gcc -o hello hello.c The $ is a prompt, i.e., prompting the user for input. We are using the bash shell, but there are others too. A command can be 1. built-in to the shell (cd, ls, type, man builtins), or 2. provided by an executable file 2.1 specified by file path, or 2.2 located by the shell after searching all directories listed in the PATH environment variable. 27 / 54

28 The Shell: Running a Program $ gcc -o hello hello.c $./hello Line 1 compiles the C program hello.c into an executable file hello Line 2 loads the hello program into memory and hands it to the OS to run the program Recall from CSC207: Java programs are compiled into bytecode; C programs are compiled into native code. 28 / 54

29 Memory Model Memory is divided into segments. The program itself is loaded into the bottom segments: 1. Read/write data. 2. Read-only code and data. 29 / 54

30 Question Why in $gcc - o hello hello. c $./ hello is there a./ in front of hello but not gcc? Answer The current directory is typically not in the PATH, so executing command is done by $./command, where. represents the current directory. Type echo $PATH to output your PATH variable. 30 / 54

31 Please ask questions if you spot something unfamiliar. In this course, a single missing or extra character can make a big difference (remember this when we introduce you to our good friends *, **, and & from the C language). Cannot say it is just one character if that character holds important conceptual meaning, compared to perhaps a simple ; which just terminates a statement. 31 / 54

32 / "root" /bin "essential user command binaries" bash cat chmod cp date echo grep gunzip gzip hostname kill less ln ls mkdir more mount mv nano open ping ps pwd rm sh su tar touch umount uname /etc "configuration files for the system" crontab cups fonts fstab host.conf hostname hosts hosts.allow hosts.deny init init.d issue machine-id mtab mtools.conf nanorc networks passwd profile protocols resolv.conf rpc securetty services shells timezone /sbin "essential system binaries" fdisk fsck getty halt ifconfig init mkfs mkswap reboot route /usr /var /dev "read-only user application "variable data files" "device files support data & binaries" incl. /dev/null" /var/cache /usr/bin /home "application "most user "user home cache data" commands" directories" /var/lib /lib /usr/include "data modified as "libraries & "standard include programmes run" kernel modules" files for 'C' code" /var/lock /mnt /usr/lib "lock files to track "mount files for "obj, bin, lib resources in use" temporary files for coding filesystems" & packages" /var/log "log files" /opt /usr/local "optional software /var/opt applications" "local software" "variable data for /usr/local/bin installed packages" /proc /usr/local/lib "process & kernel /usr/local/man /var/spool information files" /usr/local/sbin "tasks waiting to /usr/local/share be processed" /root /var/spool/cron "home dir. for /usr/share /var/spool/cups the root user" "static data sharable /var/spool/mail accross all architectures" /var/tmp /usr/share/man "temporary files saved "manual pages" between reboots" Figure: The UNIX File System Hierarchy 32 / 54

33 The following is the path to the executable file bash (our shell) /usr/bin/bash 1. The leading / represents the root directory. 2. usr is a subdirectory of /. 3. bin is a subdirectory of usr. 4. bash is a file located in bin. 5. A directory is a file that contains directory entries. 6. Each directory entry maps a file name to an inode. 7. An inode is a data structure that contains information about a file, such as its physical address on the disk (CSC369). 33 / 54

34 Directory Entries and inodes Directory Entry afile Inode size owner UID, GID access time modified time creation time link and block counts permissions direct pointers to file blocks single indirect pointer double indirect pointer triple indirect pointer pointers to next file blocks 34 / 54

35 File System Hierarchy on the Lab PCs On most UNIX systems, a user bob s home directory is /home/bob On the lab PCs, it is /student/bob 1. Your home directory is located on a networked file system that is mounted to the local directory /student on every lab PC. 2. Type df to see a list of devices, network locations, etc. that are mounted to directories in your file system. Your home directory is also represented by the shortcut ~ 1. cd ~ changes your current working directory to your home directory. 2. Type pwd to see what your current working directory is. 35 / 54

36 Everything is a file. This is the defining feature of UNIX. It even has its own Wikipedia page en.wikipedia.org/wiki/everything_is_a_file! 36 / 54

37 Files in UNIX UNIX provides a file interface (i.e., open, read, write, and close) for all Input/Output (I/O) 1. Regular files. 2. Directories. 3. Special files (e.g., /dev/null, /dev/urandom). 4. Devices (e.g., keyboard, mouse, printer, video output). Try: cat /dev/urandom padsp tee /dev/audio > /dev/null 5. Pipes for inter-process communication (can be named or unnamed*). 6. Network sockets (sometimes). 37 / 54

38 Redirecting Output By default, programs read from stdin (default: keyboard), write results to stdout (default: screen), and write errors to stderr (default: screen). 1. Use > to redirect stdout, and 2> to redirect stderr. 2. > overwrites the output file; use >> to append and 2 are the file descriptors of stdout and stderr respectively. 38 / 54

39 Example To list the files in the current directory and save (i.e. redirect) the output to myfiles.txt do: $ls > myfiles.txt 39 / 54

40 Pipes and Process Substitution Pipes transfer output from one command as input to another command. Example To count and output the number of lines of text that were generated by the ls command do $ls wc -l 40 / 54

41 Pipes and Process Substitution You can also feed the output of one or more processes to stdin of another process with <; this is called process substitution. Example The following accomplishes the same task as the previous example. $wc -l < ls 41 / 54

42 Pipes and Process Substitution Technicality Whereas a pipe redirects stdin a process substitution saves the output of a command into a temporary file and passes the descriptor of that file to the other process. See examples here unix.stackexchange.com/questions/17107/ process-substitution-and-pipe 42 / 54

43 Directories and Links $ ls file1 file2 file3 test1 test2 $ ls -a... file1 file2 file3. hidden test1 test2 $ ls -la total 16 drwxr - xr - x 4 bob staff 4096 Jan 6 20:18. drwxr - xr - x 3 bob staff 4096 Jan 6 20:18.. -rw -r--r-- 1 bob staff 0 Jan 6 20:16 file1 -rw -r--r-- 1 bob staff 0 Jan 6 20:17 file2 lrwxrwxrwx 1 bob staff 5 Jan 6 20:17 file3 -> file2 -rw -r--r-- 1 bob staff 0 Jan 6 20:18. hidden drwxr - xr - x 2 bob staff 4096 Jan 6 20:16 test1 drwxr - xr - x 2 bob staff 8192 Jan 6 20:16 test2 43 / 54

44 Directories and Links drwxr-xr-x 2 bob staff 4096 Jan 6 20:16 test1 First column is a permission string. First character indicates whether it is a directory (d), link (l), or a regular file (-) 1. A symbolic link is a file that contains a reference to another file 2. A hard link is a regular file that references an inode (doesn t actually appear to ls as a link having multiple hard links to an inode essentially means referencing the same inode by multiple file names) 44 / 54

45 Directories and Links First column is a permission string like drwxr-xr-x The rwx flags represent the read, write, and execute privileges on the file possessed by the user (file owner), group, and other users For a directory, r allows you to list its contents (ls), w allows you to create and delete directory entries in it, and x allows you to enter the directory (cd) 45 / 54

46 Directories and Links drwxr-xr-x 2 bob staff 4096 Jan 6 20:16 test1 Second column is the number of hard links to the inode referenced by the file. 1. Note that. and.. are hard links that are present in every directory 2. If you rm a file name with only one hard link, and if the file is not currently opened by any running process, the file is deleted. Otherwise, only the specified file name is deleted, and the hard link count is decremented 46 / 54

47 Directories and Links drwxr-xr-x 2 bob staff 4096 Jan 6 20:16 test1 Third column is the user (owner) of the file. Forth column is the group of the file. Fifth column is the size (in bytes) of the file. Sixth column is the last modified date of the file. Last column is the file name. 47 / 54

48 Changing File Permissions If you are the file owner (or root), you can change a file s permissions with chmod Example $ chmod 754 file.txt changes the permissions of file.txt to grant: 7: rwx to the user (owner) 5: rx to the group 4: r to all other users Note that in base two 7 = 111, 5 = 101, 4 = 100 which correspond to rwx, r x, r. 48 / 54

49 Changing File Permissions Other ways of using chmod: 1. chmod u+rwx 2. chmod go-x 3. Exercise: See man chmod to find out what the above commands do 49 / 54

50 Globbing 1. Glob patterns are strings that expand to match multiple file names (similar, but simpler, than regular expressions) 2. Check man 7 glob (this displays the contents glob(7), i.e., the glob entry from section 7 of the man-pages) 3. A? matches any single character 4. A * matches any string, including the empty string 5. An expression [list of characters] matches a single character inside the list, e.g., [abc], [a-z], [ax-z] 50 / 54

51 Globbing Example 1. $rm *.log: Remove all files ending in.log from the current directory. 2. $ls *.pdf: List all files in the current directory ending in.pdf 51 / 54

52 Job Control Definition (Job) A job is a program that the user has executed. 1. When a program is running, hit ctrl+z to suspend it 2. jobs gives you the list of jobs, each with an associated job number 52 / 54

53 $fg num Brings job number num to the foreground and resumes execution. The foreground job has control of the terminal. That is, all user input is captured by that job $bg num Brings job number num to the foreground and resumes execution. The background job does not control the terminal, but will be stopped if the terminal is closed. Any command can be started in the background by appending & to the command (e.g., sleep 10&) $kill %num Kills job number num 53 / 54

54 Next Time 1. Arrays and pointers. 54 / 54