Housekeeping (Lecture 12-6/26/2018)

Operating Systems - CSCI 42 Housekeeping (Lecture 12-6/26/218) Cheating on roll sheet signing constitutes academic dishonesty and will result in an F in the class Kernel 1 is due at 11:45pm this Tuesday, 6/26/218 if you have code from current or a previous semester, do not look at/copy/share any code from it it s best if you just get rid of it Grading guidelines is the ONLY way we will grade read it carefully and send me questions if you are not sure don t run kshell in a separate process tests in sections (C) and (D) of the grading guidelines must run in the "foreground" and show correct printout if you are confused about "SELF-checks", send me e-mail After submission, make sure you Verify Your Kernel Submission Do GDB Assignment #2 you need to be familiar with how to debug the kernel 1

Midterm exam is Thursday, next week, on 7/5/218 (DEN section) 9:3am - 1:5am in SGM 124 (PM section) 12:45pm - 2:5pm in KAP 144 midterm is worth a lot, so you should study Kernel 2 implementation timeline read and step into the code in ramfs to understand polymorphism before next Wednesday, implement basic functionalities read comment blocks above functions, read Linux man pages keep in mind that weenix and Linux are not the same thing keep in mind that what s in comment blocks may not be our requirements our requirements are the spec and the grading guidelines after next Wednesday, fix your code to pass all tests in "vfstest.c" (and then "faber_fs_test.c") make sure you don t have any reference counting bugs Operating Systems - CSCI 42 Housekeeping (Lecture 12-6/26/218) 2

Look Up Inode Number Of A Path Operating Systems - CSCI 42 Ex: how do figure out the inode number of "/home/bc/foo.c"? this process is called pathname resolution / /home home 18 bc? /home/bc foo.c? 3

Operating Systems - CSCI 42 Why link() cannot be used on a directory? to avoid cycles Unix directory hierarchy can be viewed as a directed acyclic graph (DAG) straight-forward algorithm to traverse directory hierarchy efficiently Directory Hierarchy Unix and many other OSes allow limited deviation from trees hard links reference to a file (not a directory) in one directory that also appears in another using the link() system call or the "ln" shell command soft links or symbolic links a special kind of file containing the name of another file or directory using the symlink() system call or the "ln -s" shell command 4

% ln /unix /etc/image create "image" in "/etc" to link to "/unix" unix etc home proc dev motd bc Hard Links... unix etc home proc dev Operating Systems - CSCI 42 1 1 117 4 18 36 93 unix... directory inode lecture1 lecture2. 4.. 1 motd 33 file inode 5

% ln /unix /etc/image create "image" in "/etc" to link to "/unix" unix etc home proc dev image motd bc Hard Links... unix etc home proc dev Operating Systems - CSCI 42 1 1 117 4 18 36 93 unix... directory inode file inode lecture1 lecture2... 4 1 motd 33 image 117 6

Soft Links % ln -s /unix /home/bc/mylink create "mylink" in "/home/bc" to soft-link to "/unix" Operating Systems - CSCI 42 unix etc home proc dev image motd bc unix... directory inode lecture1 lecture2 file inode 7

Soft Links % ln -s /unix /home/bc/mylink create "mylink" in "/home/bc" to soft-link to "/unix" Operating Systems - CSCI 42 unix etc home proc dev image motd bc unix... mylink directory inode lecture1 lecture2 "/unix" file inode 8

Soft Links Operating Systems - CSCI 42 % ln -s /unix /home/bc/mylink % ln -s /home/bc /etc/bc create "bc" in "/etc" to soft-link to "/home/bc" unix etc home proc dev image motd bc unix... mylink directory inode lecture1 lecture2 "/unix" file inode 9

Soft Links Operating Systems - CSCI 42 % ln -s /unix /home/bc/mylink % ln -s /home/bc /etc/bc create "bc" in "/etc" to soft-link to "/home/bc" unix etc home proc dev image motd bc bc "/home/bc" unix... mylink directory inode lecture1 lecture2 "/unix" file inode 1

Soft Links Operating Systems - CSCI 42 % ls -l /etc/bc/unix/lecture1 same as "ls -l /home/bc/unix/lecture1", or is it? yes for the "root" account, may be no for the "bc" account see "access protection" later unix etc home proc dev image motd bc bc "/home/bc" unix... mylink directory inode lecture1 lecture2 "/unix" file inode 11

Working Directory Operating Systems - CSCI 42 When you ssh to nunki.usc.edu and you type "ls" how does the login shell know what directory content to list? Working Directory: maintained in kernel for each process paths not starting from "/" start with the working directory get by using the getcwd() system call set by using the chdir() system call displayed (via shell) using "pwd" 12

Kernel 2 Operating Systems - CSCI 42 Now you have everything you need to complete kernel 2 if you are not familiar with a function, look at Linux man page keep in mind that weenix is not Linux your goal is to pass all the tests in the grading guidelines New things in kernel 2 reference counting (not as easy as it sounds) if you have a reference counting bug, will get a kernel panic C++ polymorphism implemented in C the VFS layer is AFS-independent Can the VFS layer read data from disk? no way it needs to ask AFS to do it because only AFS knows what data structure is used on disk need to invoke AFS in a FS-independent way read the ramfs code read kernel 2 FAQ 13

Access Protection Operating Systems - CSCI 42 OS needs to make sure that only authorized processes are allowed access to system resources various ways to provide this Unix (and many other systems, such as Windows) associates with files some indication of which security principals are allowed access along with what sort of access is allowed A security principal is normally a user or a group of users a file typically contains two pieces of information which user owns the file (uid) which group owns the file (gid) each running process can have several security principals associated with it for Sixth-Edition Unix, one user ID and one group ID a process in some other OSes can have more than one group IDs 14

Access Protection Operating Systems - CSCI 42 Each file has associated with it a set of access permissions there are 3 classes of security principals: user: owner of the file group: group owner of the file others: everyone else for each of the 3 classes of principals, specify what sorts of operations on the file are allowed the operations are grouped into 3 classes: read: can read a file or directory write: can write a file or directory execute: one must have execute permission for a directory in order to follow a path through it Rules for checking permissions 1) determines the smallest class of principals the requester belongs to (user being smallest and others being largest) 2) then it checks for appropriate permissions with that class 15

Permissions Example Operating Systems - CSCI 42 % ls -lr.: total 2 drwxr-x--x 2 bill adm 124 Dec 17 13:34 A drwxr----- 2 bill adm 124 Dec 17 13:34 B./A: total 1 -rw-rw-rw- 1 bill adm./b: total 2 -r--rw-rw- 1 bill adm -rw----rw- 1 trina adm 593 Dec 17 13:34 x 446 Dec 17 13:34 x 446 Dec 17 13:45 y Suppose that bill and trina are members of the adm group and andy is not 1) Q: May andy list the contents of directory A? 16

Permissions Example Operating Systems - CSCI 42 % ls -lr.: total 2 drwxr-x--x 2 bill adm 124 Dec 17 13:34 A drwxr----- 2 bill adm 124 Dec 17 13:34 B./A: total 1 -rw-rw-rw- 1 bill adm./b: total 2 -r--rw-rw- 1 bill adm -rw----rw- 1 trina adm 593 Dec 17 13:34 x 446 Dec 17 13:34 x 446 Dec 17 13:45 y Suppose that bill and trina are members of the adm group and andy is not 3) Q: May trina list the contents of directory B? 2

Permissions Example Operating Systems - CSCI 42 % ls -lr.: total 2 drwxr-x--x 2 bill adm 124 Dec 17 13:34 A drwxr----- 2 bill adm 124 Dec 17 13:34 B./A: total 1 -rw-rw-rw- 1 bill adm./b: total 2 -r--rw-rw- 1 bill adm -rw----rw- 1 trina adm 593 Dec 17 13:34 x 446 Dec 17 13:34 x 446 Dec 17 13:45 y Suppose that bill and trina are members of the adm group and andy is not 3) Q: May trina list the contents of directory B? A: Yes 21

Open Operating Systems - CSCI 42 #include <sys/types.h> #include <sys/stat.h> #include <fcntl.h> int open(const char *path, int options [, mode_t mode]) options O_RDONLY open for reading only O_WRONLY open for writing only O_RDWR open for reading and writing O_APPEND set the file offset to end of file prior to each write O_CREAT if the file does not exist, then create it, setting its mode to mode adjusted by umask O_EXCL: if O_EXCL and O_CREAT are set, then open fails if the file exists O_TRUNC delete any previous contents of the file O_NONBLOCK don t wait if I/O cannot be done immediately some options are not compatible with other options 28

Setting File Permissions #include <sys/types.h> #include <sys/stat.h> int chmod(const char *path, mode_t mode) Operating Systems - CSCI 42 sets the file permissions of the given file to those specified in mode only the owner of a file and the superuser may change its permissions nine combinable possibilities for mode (read/write/execute for user, group, and others) S_IRUSR (4), S_IWUSR (2), S_IXUSR (1) S_IRGRP (4), S_IWGRP (2), S_IXGRP (1) S_IROTH (4), S_IWOTH (2), S_IXOTH (1) note: numeric prefix of means the number is in octal format 29

Creating a File The mode parameter helps specify the permissions of the newly created file permissions = mode & ~umask Operating Systems - CSCI 42 Use either open or creat open(const char *pathname, int flags, mode_t mode) flags must include O_CREAT creat(const char *pathname, mode_t mode) open is preferred 3

Umask Standard programs create files with "maximum needed permissions" as mode compilers: 777 editors: 666 Operating Systems - CSCI 42 Per-process parameter, umask, used to turn off undesired permission bits e.g., turn off all permissions for others, write permission for group: set umask to 27 compilers: permissions = 777 & ~(27) = 75 editors: permissions = 666 & ~(27) = 64 set with umask() system call or (usually) umask shell command 31

Midterm Exam Coverage Operating Systems - CSCI 42 Midterm exam covers everything from the beginning of the semester to this slide Ch 1 through Ch 4 only final exam coverage will not overlap midterm coverage since the topics covered by the final exam is not independent of the midterm coverage, we say the final exam "focuses" on Ch 5 plus everything beyond this slide 32

Operating Systems - CSCI 42 1.4 Beyond A Simple OS Extensions New Functionality 33

Operating Systems - CSCI 42 CS42 Exams Bill Cheng http://merlot.usc.edu/cs42-m18 34

Purpose Of Exams Operating Systems - CSCI 42 Some people think that an exam is a learning experience not really it s more like SAT, GRE, and TOFEL tests it s an assessment because I need to give you a fair grade you are being tested for materials you have learned in this class it s not about testing your general OS knowledge you are tested on knowledge + "critical thinking" 35

Calculation-type questions Question Types Operating Systems - CSCI 42 Short answers give the best answer for a question if the question says, "In N words or less,...", it s just a hint to tell you approximately how many words we expect you can write more words exception: when it s a fill-in-the-blank question (N would be small) then you must not use more than N words you get credit for including the right answer you get deduction for saying something wrong must not answer a general (high-level) question by giving an example must not answer a specific question by giving a general (high-level) answer 36

Question Types Operating Systems - CSCI 42 Multiple-choice unless explicitly stated, every multiple-choice question has one or more correct answers even if the question asks, "which statements are correct", it can have one or more correct answers please ignore minor grammatical and spelling mistakes if you select all the correct choices, you get 2 points if only one correct answer, it s worth 2 points if two correct answers, each is worth 1 point if three correct answers, first one is worth 1 point, the rest worth.5 point each if four correct answers, each is worth.5 point for every wrong answer you have selected, you lose 1 point best not to guess! please note that if a statement is inconsistent or incorrect, it should be considered to be a "false" statement Total points for a problem cannot go negative 37

Question Types Operating Systems - CSCI 42 Correct/best answers to most questions in the exam are the ones on lecture slides please understand that exams are about materials in this class it doesn t matter what you have read anywhere else if you disagree with what s on a lecture slide or my posts to the class Google Group, you need to complain to me now to set things straight once the exam starts, it s too late to say that you don t like the answers on the lecture slides or my posts I reserve the right to ask about things that I think you should know and not on lecture slides Therefore, you should focus on studying lecture slides although it should be clear that for multiple-choice questions, the "wrong answers" are not on lecture slides so you need to figure out why some answers are wrong you need to understand lecture slides 38

Midterm Exam Logistics Thu 7/5/218, 8 minutes DEN lecture will start at 11am (no roll sheets) Operating Systems - CSCI 42 Closed book, closed notes, and closed everything No calculators, cell phones, or any electronic gadgets are allowed bring a photo ID - will be collected at the beginning of the exam and will be returned to you when you turn in your exam There will be assigned seating you must sit in your assigned seat Covers Ch 1 through Ch 4 that have been covered in lectures so far Ch 5 is not part of midterm coverage also warmup1, warmup2, kernel1 (specs, FAQs, my post to class Google Group) will not ask you to write code, but can ask you about code in specs and FAQs 39

General Rules Read problem descriptions very carefully you may not receive any credit if you answer the wrong question if a question has multiple parts, make sure you answer every part and clearly label which answer is for which part We will give partial credit if applicable no guarantee that you will get partial credit Operating Systems - CSCI 42 Do not just draw pictures to answer questions (unless the question asked you to draw pictures) pictures will not be considered for grading unless they are clearly explained with words Weenix questions if a question does not mention weenix explicitly, you must not give a weenix-specific answer 4

General Rules Operating Systems - CSCI 42 During the exam, please only ask questions to clarify problems please do not ask for hints or ask if "it s okay to answer this way" if you ask a non-clarification type question, we are not responsible for the answer that you think we gave during regrade, you cannot say, "But the TA told me that it s okay to answer it this way" I will not answer questions regarding multiple choice questions because it gives answers away read the question carefully and choose carefully need to select all the right answers and not select any of the wrong answers 41

Rules About Regrades Operating Systems - CSCI 42 In order to be fair to all, we must stick to the 15-minute timeslot if you have been assigned a timeslot from X to Y, your appointment must be done at Y be on time and spend your time wisely again, this is not a "learning experience" We can only grade based on what you wrote it cannot be based on "what you meant" please do not argue about what you meant and focus on "what you wrote" Please keep the following in mind regarding regrades: regrades is about whether the TA has made a mistake or not it s not about arguing for points there is no point arguing, because: everyone who answered the same way got the same score you cannot argue about anything that s not written on your exam paper 42

Operating Systems - CSCI 42 Ch 6: File Systems Bill Cheng http://merlot.usc.edu/cs42-m18 43

Files Operating Systems - CSCI 42 Memory Disk Disk 44

Requirements Permanent storage resides on disk (or alternatives) survives software and hardware crashes (including loss of disk?) Operating Systems - CSCI 42 Quick, easy, and efficient satisfies needs of most applications how do applications use permanent storage? 45

Software development text editors linkers and loaders source-code control Applications Operating Systems - CSCI 42 Document processing editing browsing Web stuff serving browsing Program execution paging 46

Directories convenient naming fast lookup Needs Operating Systems - CSCI 42 sequential is very common! File access "random access" is relatively rare 47

Operating Systems - CSCI 42 6.1 The Basics of File Systems UNIX s S5FS Disk Architecture Problems with S5FS Improving Performance 48

S5FS A simple file system slow not terribly tolerant to crashes reasonably efficient in space no compression Operating Systems - CSCI 42 Concerns on-disk data structures file representation recall that AFS translates an inode number to disk address free space 49

S5FS Layout Operating Systems - CSCI 42 Boot block Superblock I-list Data Region A disk is simply an array of blocks of 1KB each (old Unix: 512B) a disk block is a logical unit (usually multiple of page size) A "linear view" (1-D array of blocks) of the disk 1 2... I-list Data Region 5

Operating Systems - CSCI 42 S5FS Layout Boot block Superblock I-list where are the "free space"? inodes can be free or not data block can be free or not Data Region The superblock describes the layout of the rest of the file system contains the head of the free list The i-list is an array of index nodes (inodes) each representing a file 51

S5FS: Inode Operating Systems - CSCI 42 Device Inode Number File Type Mode (Protection) Link Count Owner, Group Size missing from textbook (but in weenix) Disk Map 52

1 2 3 4 5 6 7 8 9 1 11 12 Disk Map Operating Systems - CSCI 42 assuming blocksize = 1KB Disk Map contains 13 disk block pointers a disk block points is just a block number 53

1 2 3 4 5 6 7 8 9 1 11 12 Disk Map Operating Systems - CSCI 42 assuming blocksize = 1KB up to 1KB Disk Map contains 13 disk block pointers a disk block points is just a block number Some blocks in the data region contains data 54

1 2 3 4 5 6 7 8 9 1 11 12 256 max entries Disk Map Operating Systems - CSCI 42 assuming blocksize = 1KB up to 1KB+256KB Disk Map contains 13 disk block pointers a disk block points is just a block number Some blocks in the data region contains data some blocks in the data region contains data structures 55

1 2 3 4 5 6 7 8 9 1 11 12 256 max entries Disk Map 256 max entries Operating Systems - CSCI 42 assuming blocksize = 1KB up to 1KB+256KB+64MB 256 max entries 256 max entries 56

1 2 3 4 5 6 7 8 9 1 11 12 256 max entries Disk Map 256 max entries Operating Systems - CSCI 42 assuming blocksize = 1KB up to 1KB+256KB+64MB+16GB limit set at 2GB 256 max entries 256 max entries 256 max entries 256 max entries 256 max entries 256 max entries 57

Disk Map Operating Systems - CSCI 42 1 2 3 4 5 6 7 8 9 1 11 12 File "Metadata" 256 max entries 256 max entries Content of File (i.e., File "Data") 256 max entries 256 max entries 256 max entries 256 max entries 256 max entries 256 max entries 58

Operating Systems - CSCI 42 1 2 3 4 5 6 7 8 9 1 11 12 File "Metadata" 256 max entries 256 max entries Disk Map Content of File (i.e., File "Data") 256 max entries 256 max entries 256 max entries for a directory inode, the lookup table is the "content of file" 256 max entries 256 max entries 256 max entries 59

S5FS Free List Operating Systems - CSCI 42 free list 1 97 98 99 Free Disk Block Super Block The superblock contains the addresses of up to 1 free disk blocks the last of these disk blocks contains 1 pointers to additional free disk blocks, etc. can find all the free disk blocks this way 97 98 99 6

S5FS Free List Operating Systems - CSCI 42 free list 1 97 98 99 Free Disk Block Super Block Need a free block return first available block in first node of the free list (in superblock) 97 98 99 61

S5FS Free List Operating Systems - CSCI 42 return this block free list 1 97 98 99 Free Disk Block Super Block Need a free block return first available block in first node of the free list (in superblock) 97 98 99 62

S5FS Free List Operating Systems - CSCI 42 free list NULL NULL NULL NULL 99 Free Disk Block Super Block Need a free block return first available block in first node of the free list (in superblock) if no free block in first node first copy 1 pointers into the superblock with one disk access 97 98 99 63

Free Me S5FS Free List Operating Systems - CSCI 42 free list NULL 1 97 98 99 Free Disk Block Super Block When a disk block is freed that block s address is added to the list of free blocks in the superblock if the list is full write it out and update superblock 97 98 99 64

Free Disk Block S5FS Free List Operating Systems - CSCI 42 free list 1 97 98 99 Free Disk Block Super Block When a disk block is freed that block s address is added to the list of free blocks in the superblock 97 98 99 65

Free Me S5FS Free List Operating Systems - CSCI 42 free list 1 97 98 99 Free Disk Block Super Block When a disk block is freed that block s address is added to the list of free blocks in the superblock if the list is full write it out and update superblock 97 98 99 66

S5FS Free Inode List Operating Systems - CSCI 42 inode cache 4 12 6 11 13 Super Block Inodes (in the i-list) are marked free or not free I-list use a "bitmap" to store this no additional organization in the i-list the superblock caches free inodes (i.e., in the inode cache) 1 2 3 4 5 6 7 8 9 1 11 12 13 14 15 16 67

S5FS Free Inode List Operating Systems - CSCI 42 inode cache 4 12 6 11 13 Super Block The inode cache to allocate an inode, simply I-list mark it not free and remove it from the inode cache to free an inode, simply mark it free and add to the inode cache if there is room 1 2 3 4 5 6 7 8 9 1 11 12 13 14 15 16 68

S5FS Free Inode List Operating Systems - CSCI 42 inode cache 4 12 6 11 13 Super Block If the inode cache is empty scan the i-list to refill it 1 2 3 4 5 6 7 8 9 1 11 12 13 14 15 16 I-list to help out with the scan, the inode cache contains the index of the first free inode in the i-list need to maintain this entry when necessary 7

S5FS Free Inode List Operating Systems - CSCI 42 inode cache 4 12 6 11 13 Super Block To create a file get a free block update free list 1 2 3 4 5 6 7 8 9 1 11 12 13 14 15 16 get a free inode update i-list and inode cache I-list 71

S5FS Free Inode List Operating Systems - CSCI 42 inode cache 4 12 6 11 13 Super Block To delete a file add disk block(s) to free list I-list mark inode free in i-list and may be update inode cache 1 2 3 4 5 6 7 8 9 1 11 12 13 14 15 16 72

S5FS Summary Operating Systems - CSCI 42 In designing a file system, one tries to minimize the number of disk operations read vs. write sequential access vs. random access S5FS gives O(1) number of disk operations for random access 73