According to this plan, you should have started with Lab 2 now Around 50% have passed Lab 1. The environment (1)

Transcription

1 TDDB68 Lesson 2 Pintos Assignments 2 & 3 Mattias Eriksson 2009 mater@ida.liu.se some slides by Viacheslav Izosimov Comments on muddy card evaluation Environment to build, run and test on Windows/ cygwin? /... on Linux? Appendix G in the Pintos manual. Known to work on Unix-like systems. I don't know about Cygwin. You can also ssh to astmatix Disable VGA (run qemu in terminal): pintos v C c won't stop Pintos now, use C a x to terminate Qemu 4 Instead of Motivation Plan for something like this: Lab 00, 0 Should be finished now Lab 1 10 th September (1 week) Lab 2 23 th September (2 weeks) Lab 3 9 th October (2 weeks) Lab 4 13 th October (½ week) Exam 20 th October According to this plan, you should have started with Lab 2 now Around 50% have passed Lab 1 Comments on muddy card evaluation Not enough space on user accounts sizeof(pintos) < 15 MB. Quota for this course: 20 MB. If you have run out of space it is probably because you have old files still on your account. Tip 1: check your quota quota v Tip 2: list your folders by size (kilobytes): cd ~ du sk ^..* sort n 15% are done with Lab The environment (1) Some of you are still struggling with the Unix environment It's difficult to learn, not difficult to use! Default prompt is not very helpful: mina10 <123> Make the prompt tell you which directory you are in: echo set prompt='%b%n@%m%b %c > ' >> ~/.cshrc.private Now every new prompt will something look like this: mater@mina10 ~/pintos/src/userprog > The environment (2) TAGS (emacs): finding the definition of a variable or an identifier cd pintos/src gmake TAGS emacs & M-. (hold down meta and press dot) Type malloc Emacs will jump to the declaration of malloc M-0 M-. will jump to the next tag matching malloc M-* jump back to the place where M-. was last Alternative: use a more modern shell, like bash 5 invoked (stack) 6

2 The environment (3) Subversion tips cd pintos/src svn up #get latest version from repository svn st #show which files are modified and new svn diff #show difference between working copy and most recently checked in revision svn -r 17 diff #same, but compared to revision 17 svn -r 17 -x -wp diff #ignore changes in whitespace and include context information svn log #show commit log The environment (4) (advanced subversion usage) Ignore this if you only do labs on a single IDA account Starting a subversion daemon cd svnrepository/conf Add users and passwords in authz and passwd Edit svnserve.conf to not allow anonymous access, and uncomment some lines... Login on to astmatix (astmatix reboots every day) svnserve d r $HOME/svnrepository listen port Pick a port number no one else is using (>1024) Now, on another computer: svn -v log #also list which files are modified svn co svn://astmatix.ida.liu.se:12345/pintos pintos 7 8 Lab 2 System calls program (only one at a time) Lab 1 programs run in the kernel System calls Console (keyboard, monitor) Program Drivers Hardware Syscall Handler (Scheduler) 10 Lab 2 Systems calls at a high level System calls: communication between user program and the kernel functions, called from the user program and performed by the kernel computers use interrupts to accomplish that switch from user code to system code Program Drivers Hardware Syscall Handler (Scheduler) 11 Pintos Lab 2: Pintos kernel overview Synchronization primitives File system Timer programs System call Interrupt handlers syscall, timer, kbd, etc Drivers Hardware Page table Scheduler Threads Lab 2 What to implement? You will need to implement these system calls: create - creates a file. open - opens a file. close - closes a file. read - reads from a file or the console (the keyboard). write - writes to a file or the console (the monitor). halt - halts the processor. exit - Terminates a program and deallocates resources allocated to it. (will be extended in Lab 3, don't worry about saving exit status yet.) LAB2

3 System calls Number of system calls Pintos Lab1: 0 Lab2: 7 Lab3: 9 Lab4: 13 Linux 319 system calls Windows More than 1000 system calls Lab 2 Files You have to have a look at: pintos/src/lib/user/syscall[.h.c] the wrapper userprog/syscall[.h.c] Your implementation of system calls! threads/interrupt.[h c] important structures! lib/syscall-nr.h system call constants filesys/filesys.[h c] Pintos file system implementation examples/lab2test.c the official test program for this lab filesys/file.[h c] useful functions for operations with files. Things which you don t find in filesys, you find here userprog/process.c 14 Lab 2: How Do I Start? (1) STEP 1: Understand what a user program is Look into the examples directory and a couple of the user programs, especially halt.c Look into Makefile in this directory to understand how the user programs are compiled Compile the user programs by issuing gmake 15 Lab 2: How Do I Start? (2) STEP 2: Dealing with Pintos disk Pintos uses a simulated hard-drive Can hold up to 16 files Filenames are up to 14 chars long To create it, do in userprog/build: pintos-mkdisk fs.dsk 2 This will create a file fs.dsk with a 2MB simulated disk in the directory. Format the simulated disk: pintos --qemu -- -f -q 16 Lab 2: How Do I Start? (3) Put a file on the disk: pintos --qemu -p../../examples/halt -a halt -- -q To examine the disk: pintos --qemu -- ls pintos --qemu -- cat filename pintos --qemu -- rm file1 rm file2 ls Example: to run halt: pintos -qemu -p../../examples/halt -a halt -- -q pintos --qemu -- run halt Tip: alias pintos pintos qemu 17 Lab 2: How Do I Start? (4) STEP 3: into userprog/process.c, find setup_stack() *esp = PHYS_BASE; change to *esp = PHYS_BASE 12; STEP 4: For now, make process_wait() an infinite loop or the kernel might power off too quickly Because the kernel calls process_wait on the first program. Hint: implement the Write system call (console part) early so that you can use printf() in user programs. 18

4 Lab 2: Shutdown Example The simplest user program: halt.c: #include <syscall.h> int main (void){ halt (); //shuts down the system This one is simple, do it first. Lab 2 Systems calls at a high level user_program.c bool flag; Calling flag = create( file.txt, 1000); corresponding function in the wrapper System call wrapper pintos/src/lib/user/syscall.c bool create (const char *file, unsigned initial_size) { return syscall2 (SYS_CREATE, file, initial_size); Calling corresponding exception in Pintos kernel the kernel syscall_handler(...){ // Your code here! Return result back to the user program pintos/src/userprog/syscall.c Return result back to the wrapper Lab 2: Create Example bool create (const char *file, unsigned size) Example: create( file.txt, 1000); How to retun a value? How to get them? Answer: f->eax Answer: they are passed on the stack: f->esp Hint: note that, in order to get a string, you will need get a pointer from esp. This pointer will point to the first element of the string The syscall2 macro explained /*return syscall2 (SYS_CREATE, file, initial_size);*/ #define syscall2(number, ARG0, ARG1) \ ({ \ int retval; \ asm volatile //Tell compiler to be careful \ //Put syscall arguments on the stack (long = 4 bytes) \ ("pushl %[arg1]; pushl %[arg0]; " \ "pushl %[number]; //Make interrupt nr 0x30 (syscall) int $0x30; //The interrupt handler will do some work now... //When it is done, forget the syscall arguments addl $12, %%esp" //3 words=12 bytes //Bind the symbols to our arguments \ : "=a" (retval) //output \ : [number] "i" (NUMBER), //input \ [arg0] "g" (ARG0), \ [arg1] "g" (ARG1) \ : "memory"); //clobbered \ retval; //retval will be the value of the expression\ 20 Lab 2: Create Example To implement the system calls, which operate with files, look at filesys/filesys.[h c] and filesys/file.[h c]. Everything is already done there for you! Just call the functions But what if the user program passes a bad pointer? e.g. create(null,0); the kernel would crash when reading the filename! you will fix this in Lab 3. The syscall handler Currently the syscall handler kills the thread You must: Get the syscall number Get the correct number of arguments Pointers and arrays are very related in C ptr + i == &(ptr[i]) static void syscall_handler (struct intr_frame *f UNUSED) { printf ("system call!\n"); thread_exit (); ) 24 21

5 Lab 2: File descriptors int open (const char *file) Open returns a file descriptor (fd) that is a handle to the open file Your task is to map the fd's to open files Need a list in the kernel To make it simple you are allowed to limit the number of files that can be open Pintos has a bitmap data structure that can help 0 and 1 must be always reserved for the The Pintos bitmap by example #include lib/kernel/bitmap.h #define FOOSIZE 128 static struct bitmap * foomap;... { /* Allocate memory for our bitmap */ foomap = bitmap_create (FOOSIZE); if (foomap == NULL) PANIC ("Map is too big"); /* Set 0th bit to 1 */ bitmap_mark (foomap, 0); /* Find a bit which is 0 and mark it */ int fnd = bitmap_scan_and_flip (foomap, 0, 1, 0); /* Free the bitmap (otherwise a leak) */ console! 25 bitmap_release (foomap); 26 Lab 2: Read System Call int read (int fd, void *buffer, unsigned size); Reads size bytes from the file open as fd into buffer. Returns the number of bytes actually read, or -1 if the file could not be read. Fd 0 == STDIN_FILENO Should read from the keyboard using input_getc() (see: devices/input.h). 27 Lab 2: Write System Call int write (int fd, const void *buffer, unsigned size); Writes size bytes from buffer to the open file fd Returns the number of bytes actually written or -1 if the file could not be written The expected behavior is to write as many bytes as possible up to end-of-file Fd 1==STDOUT_FILENO use to putbuf() (check lib/kernel/stdio.h and lib/kernel/console.c) 28 Lab 2: Exit System Call void exit (int status); Terminates the current user program, returning status to the kernel. Conventionally, a status of 0 indicates success and nonzero values indicate errors. This system call will be extended in Lab 3, for now you won't use status Important: Free resource in process_exit() function located in process.c CLOSE ALL OPEN FILES 29 Test program lab2test.c A small test program is provided, it will: Create files Open files Test console Try to use fake fd's Gotchas Remove all files before running it again: pintos qemu rm test0 rm test1 rm test2 Your Lab 2 implementation will be checked more thoroughly once Lab 3 is finished (gmake check) won't work now 30

6 Lab 3: General Description Lab 3: Execution, termination and synchronization of user programs Handling program arguments Execution of several user programs Termination of ill-behaving user programs Synchronization of shared data structures Part A Multiple user programs Lab 3: overview New system calls: exec & wait Extended system call from Lab 2: exit Careful synchronization in the kernel Lab 3 is tricky! Program1 Program2 Drivers Syscall Handler Program3 (Scheduler) Hardware 33 Part B Lab 3: overview program arguments cp foo bar : two arguments Part C Make the kernel robust to ill-behaving user programs Evil user program: create( (char *) 0, 1); Or worse, writing to kernel space: write( STDIN_FILENO, 0xc , 666 ); 34 Part A Lab 3: Exec pid_t exec (const char *cmd_line); Runs the executable whose name is given in cmd_line, passing any given arguments, returns the new process s program id (pid) Must return pid -1, if the program cannot load or run for any reason 35 Lab 3: Exit void exit (int status); Terminates the current user program, returning the exit code status to the kernel. By convention, status 0 indicates success and nonzero values often indicate errors. Remember to free all the resources that will be not needed anymore. files dynamically allocated memory 36

7 Lab 3: Wait int wait (pid_t pid); Provides synchronization between user programs. "Parent" process waits until its pid-"child" process dies (if the child is still running) and receives the "child" exit code. If the child has been finished, wait() should return child's exit value without waiting. 37 pid = process ID (user space) tid = thread ID (kernel space) Syscall handler syscall.c { f-eax = process_execute The parent Lab 3: Exec process_execute() { tid = thread_create() FO generate pid from tid; wait until start_process(); return pid or -1 The child start_process() { load(binary); manipulate stack (Part B) signal to process_execute Problem: The loading of the binary is done in the child, hence the parent does not know at fork-time if loading will work Need synchronization! 38 syscall.c { get exit code from user; save the exit code if needed; thread_exit() Lab 3: Exit thread_exit() { process_exit() process_exit() { clean up program s resources; At exit, do: printf("%s: exit(%d)\n", thread name, thread exit value) This is needed for testing purposes (gmake check). 39 Lab 3: Situations with Wait Child exits before the parent and: parent calls wait() afterwards Lab 3: Situations with Wait Parent exits without calling wait() while the child is still running Lab 3: Situations with Wait Child exits before the parent and: parent will exit without calling wait(). Child keep the exit value Destroy the exit value! Child Child keep the exit value Destroy the exit value! Parent exec(child) wait(child) Parent exec(child) Parent exec(child) wait() returns child s exit value without waiting You should keep child s exit value until the parent exits (since the child doesn t know if the parent calls wait() later on)

8 Lab 3: Situations with Wait Parent calls wait() before the child exits. Parent Child exec(child) wait for the child wait(child) the parent waits for its child Destroy the exit value! 43 Reference counting poor man's garbage collector Parent Child needs a new data structure Who will free this memory depends on who exits last Many ways to implement this. Suggestion: you can use reference counting: struct child_status{ int exit_status; /*... */ int ref_cnt; ; child /* Initialize it */ struct child_status * cs = malloc... ; cs >ref_cnt = 2; /* both parent and child live */ parent /* When parent or child is done with cs: */ cs >ref_cnt ; /* Needs to be protected by a lock */ if ( cs >ref_cnt == 0 ){ free( cs ); cs 44 Lab 3: To Hit The Target! Parts of the functions accessing shared resources must be thread safe, e.g. employ synchronization techniques such as locks and semaphores. Particularly, access to global objects and data must be synchronized. Only one thread can have access to the console at a time. Other threads must wait until completion of reading/writing. 45 Lab 3: Part B Sometimes we want to pass arguments to new programs ls l cp foo bar In C programs: argc and argv argv[0] : name of the program argv[1] : first argument... argc : number of arguments argv[argc] = NULL : Required by the C standard Your task: put arguments on the stack when a program is loaded. Ex.echo: while( argc) printf( %s,*++argv); 46 Lab 3: Part B Arguments to the syscall are passed via user program stack Program arguments are passed in a similar fashion The top of the user stack will contain these arguments for the duration of the program 47 Lab 3: PB: Preparatory Steps (1) Lab 2::STEP 3 into userprog/process.c, find setup_stack() *esp = PHYS_BASE; CHANGE IT BACK! change to *esp = PHYS_BASE - 12; Before continuing, explain why you have "KERNEL PANIC" after you have removed "-12". What is wrong and why did the program work before? Hint the C lib wraps main: /* src/lib/user/entry.c */ void _start (int argc, char *argv[]) { exit (main (argc, argv)); 48

9 Lab 3::PB::Preparatory Steps (2) The user program with arguments should be called with single quotes ('...') from the Pintos command line: pintos --qemu -- run nasty_program arg1 arg2 arg3 program with arguments via syscall exec("nasty_program arg1 arg2 arg3"); Lab 3: PB: Implementation (1) STEP 1: Break the string in to parts Use strtok_r() lib/string.[ch] STEP 2: Put it on the stack Necessary details about setting up the stack for this task you can find in Program Startup Details section of Pintos documentation. Lab 3 : Pointer paranoia Memory validation (simple method) VM Physical Memory PHYS_BASE A valid pointer into a syscall: 0 is below PHYS_BASE in VM Page directory is associated with a page of physical memory: use pagedir_getpage() Kill the user program if there is an error! Lab 3 : Pointer paranoia Memory validation (tricky method) VM Physical Memory PHYS_BASE 0 Page directory Another way to validate memory is mentioned in the manual Check that the pointer is below PHYS_BASE Dereference the pointer (*ptr) and then take care of page faults This is how it is done in real operating systems, it is faster Suggestion: Only do it this way if you want the extra challenge Address 0xbffffffb 0xbffffff6 0xbffffff1 0xbfffffe3 0xbfffffe0 0xbfffffdc 0xbfffffd8 0xbfffffd4 0xbfffffd0 0xbfffffcc 0xbfffffc8 0xbfffffc4 0xbfffffc0 Name argv[3][...] argv[2][...] argv[1][...] argv[0][...] word-align argv[4] argv[3] argv[2] argv[1] argv[0] argv argc return address Data arg3\0 arg2\0 arg1\0 nasty_program\ > xbffffffc 0xbffffff8 0xbffffff4 0xbfffffe6 0xbfffffd Type char[5] char[5] char[5] char[14] char * char * char * char * char * char ** int void (*) () 53 exit code -1 Lab3 : Testing When Part B is finished you can run gmake check The following tests Lab 2: halt, exit, create-normal, create-empty, create-null, create-long, create-exists, createbound, open-normal, open-missing, open-boundary, open-empty, open-twice, close-normal, close-stdin, close-stdout, close-bad-fd, read-boundary, read-zero, read-stdout, read-bad-fd, write-normal, write-boundary, write-zero, write-stdin, write-bad-fd Most of the exec-* and wait-* tests (and lab2) should pass when you have finished Part A&B Run a single test (from userprog/build): gmake tests/userprog/halt.result The rest when Part C is finished (around 60 tests in total) 51 54

10 Lab 3 : Part C Lab 3 : Pointer paranoia A trickier user program Part C: Nothing a user-program does should cause the kernel to crash, panic or fail an assertion! Make the kernel robust to ill-behaving user programs! Example evil user program passes NULL for filename: create( (char *) 0, 1); Or worse, writing to kernel space: write( STDIN_FILENO, 0xc , 666 ); Be paranoid All pointers from user programs into the kernel must be checked. PHYS_BASE VM Physical Memory stack pointer, string, buffer 55 exit code A valid pointer into a syscall: 0 is below PHYS_BASE in VM Page directory is associated with a page of physical memory: use pagedir_getpage() Kill the user program if there is an error! #define PGS 4096 /* page size */ #define PMASK 0xfffff000 static char inbss; int main (int argc, char ** argv) { char * bss_page = (char*) ((int)(&inbss) & PMASK); In this program, the beginning of the string is valid Just checking the pointer is not enough memset ( bss_page, 'a', PGS ); create (bss_page+pgs 5, 1024); ; 57 TDDB68 Extra slides test programs Lab 3::Test (1) Now you are ready for a complete check! Run gmake check from userprog/build The following tests should pass: 1) Argument passing when executing: args-none, args-single, args-multiple, args-many, args-dbl-space 2) Different exec-tests: exec-once, exec-arg, exec-multiple, exec-missing, exec-bad-ptr 3) Wait-tests: wait-simple, wait-twice, wait-killed, Lab 3::Test (2) Many of the other checks in gmake check belong to the second lab. If any of them fails, then it means that something is wrong with your implementation of Lab 2: sc-bad-sp, sc-bad-arg, sc-boundary, sc-boundary-2, halt, exit, create-normal, create-empty, create-null, create-bad-ptr, create-long, create-exists, create-bound, open-normal, open-missing, open-boundary, open-empty, open-null, open-bad-ptr, open-twice, close-normal, close-stdin, close-stdout, close-bad-fd, read-bad-ptr, read-boundary, read-zero, read-stdout, read-bad-fd, write-normal, write-bad-ptr, write-boundary, write-zero, write-stdin, wait-bad-pid write-bad-fd 59 60

11 To debug: /* grandfather.c */ #include <syscall.h> #include <stdio.h> int main (void){ int pid1, pid2, pid3; pid1 = exec("parent1"); wait(pid1); pid2 = exec("parent2"); wait(pid2); pid3 = exec("parent3"); wait(pid3); ; Lab 3::Test (2) /* parentx.c */ #include <syscall.h> #include <stdio.h> int main (void){ int i; int pid[10]; for(i = 0; i < 10; i++) { pid[i] = exec("childx"); for(i = 0; i < 10; i++) { wait(pid[i]); ; 61 Lab 3::Test (3) /* childx.c */ #include <syscall.h> #include <stdio.h> int main (void){ int i; for(i = 0; i < 20000; i++) { int a = (i * i) + (i * i); int b = i; i = a; a = b; i = b; write(1, PASS Lab X ON Time.\n,20); ; Create 1 grandfather, 3 parents and 3 children in examples directory: grandfather.c parent1.c parent2.c parent3.c child1.c child2.c child3.c Replace X in all the places in the code for parents and children with 1, 2, and 3, respectively. 62 Lab 3::Test (4) Modify Makefile in examples, such that all the programs are compiled: PROGS = parent1 parent2 parent3 child1 \ child2 child3 grandfather parent1_src = parent1.c child1_src = child1.c parent2_src = parent2.c child2_src = child2.c parent3_src = parent3.c child3_src = child3.c grandfather_src = grandfather.c Compile the programs with gmake. 63 Lab 3::Test (5) Copy them all on the Pintos disk in userprog (don t forget to create the disk and format it). Start them as pintos --qemu -- run grandfather As the output programs should print: Firstly, all 10 PASS Lab 1 ON Time. Secondly, all 10 PASS Lab 2 ON Time. Thirdly, all 10 PASS Lab 3 ON Time. Then, try to comment some wait() in grandfather and parents to see how it behaves. The order should be destroyed (Don t forget to recompile the programs and copy on the disk again. You may need to format the disk before.) 64