Design and Implementation of the MTX Operating System

Transcription

1 Design and Implementation of the MTX Operating System

2 K. C. Wang Design and Implementation of the MTX Operating System 1 3

3 K. C. Wang School of Electrical Engineering and Computer Science Washington State University Pullman Washington USA ISBN DOI / ISBN (ebook) Library of Congress Control Number: Springer Cham Heidelberg New York Dordrecht London Springer International Publishing Switzerland 2015 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms, or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made. Printed on acid-free paper Springer International Publishing AG Switzerland is part of Springer Science+Business Media (

4 Preface The purpose of this book is to provide a suitable platform for teaching and learning the theory and practice of operating systems. It covers the basic concepts and principles of operating systems, and it shows how to apply them to design and implement a complete operating system in incremental steps. Throughout the book, it uses the development of the MTX operating system to convey the real intent of this book, which is to demonstrate the design principles and implementation techniques of operating systems in general. Due to its technical contents, this book is not intended for entry-level OS courses that teach only the basic concepts and principles of operating systems without programming practice. It is intended for technically oriented OS courses that emphasize on both theory and practice. The book s evolutional style, coupled with detailed source code and complete working sample systems, make it especially suitable for self-study. This book contains many new and original materials, especially on the design and use of parallel algorithms in SMP. It is also suitable for advanced study of operating systems. Undertaking this book project has proved to be quite a challenge. While preparing the book for publication, I have been blessed with the encouragements and helps from many people. I would like to take this opportunity to thank all of them. I am also very grateful to Springer International Publishing AG for allowing me to disclose the source code of this book to the public for free. Hopefully, this would help further developments of the MTX system and enhancements to this book in the future. Special thanks go to Cindy Wang for her continuing support and inspirations, which have made this book possible. Finally, I would like to thank my family for bearing with me with endless excuses of being too busy for so long. Source code included in the book can be downloaded from the author s website at: December, K. C. Wang v

5 About the Author K. C. Wang received the BSEE degree from National Taiwan University in 1960 and the Ph.D. degree in Electrical Engineering from Northwestern University, Evanston, Ill in He is currently a Professor in the School of Electrical Engineering and Computer Science at Washington State University. His academic interests are in Operating Systems, Distributed Systems and Parallel Computing. vii

6 Contents 1 Introduction About This Book Objective and Intended Audience Unique Features of This Book Book Contents Use This Book as OS Textbook MTX Systems for Testing and Evaluation... 8 References Foundations and Background Computer System Computer Hardware System CPU Operations System Mode and User Mode Memory and Memory Models Virtual Address and Physical Addresses I/O Devices and I/O Operations Operating Systems Major Functions of Operating Systems Program Development Program Development Steps Static vs. Dynamic Linking Executable File Format Contents of a.out File Program Execution Program Termination Function Call in C Run-Time Stack Usage Stack Frames Return From Function Call Long Jump ix

7 x Contents 2.5 Link C Program with Assembly Code Link Library Mailman s Algorithm Applications of Mailman s Algorithm EXT2 File System EXT2 File System Data Structures Traverse EXT2 File System Tree The BCC Cross-Compiling Package Running MTX From Login to Command Execution References Booting Operating Systems Booting Bootable Devices Booting from Various Devices Floppy Disk Booting Hard Disk Booting CD/DVD_ROM Booting USB Drive Booting Boot Linux with Initial Ramdisk Image Network Booting Develop Booter Programs Requirements of Booter Programs Online and Offline Booters Boot MTX from FD sectors Boot Linux zimage from FD Sectors Fast FD Loading Schemes Boot MTX Image from File System Boot Linux zimage from File System Hard Disk Booter I/O and Memory Access Functions Read Hard Disk LBA Sectors Boot Linux bzimage with Initial Ramdisk Image Hard Disk Partitions Find and Load Linux Kernel and Initrd Image Files Linux and MTX Hard Disk Booter Boot EXT4 Partitions Install HD Booter CD/DVD-ROM Booter Emulation CDROM Booting Emulation-FD Booting Emulation-HD Booting No-Emulation CDROM Booting No-Emulation Booting of MTX... 83

8 Contents xi No-Emulation Linux Booter Comparison with isolinux CDROM Booter USB Drive Booter MTX on USB Drive Linux on USB Drive Listing of Booter Programs References A Simple Operating System Kernel Multitasking The Process Concept Development of the MTX Kernel A Simple Multitasking Program Context Switching A Simple Multitasking System Dynamic Process Creation and Process Scheduling MTX4.2: Demonstration of Process Creation and Process Switch Stop and Continue a Process MTX4.3: Demonstration of Stop/Continue Operations Sleep and Wakeup Operations Sleep and Wakeup Usage Implementation of Sleep and Wakeup MTX4.4: Demonstration of Sleep/Wakeup Operations Process Termination Algorithm of kexit Process Family Tree Wait for Child Process Termination Change Process Scheduling Priority MTX4.5: Demonstration of Process Management MTX Development Environment References User Mode and System Calls Process Execution Image Kernel and User Modes Transition Between User and Kernel Modes System Calls System Call Interrupt Handler Enter and Exit Kernel Mode System Call Handler Function in C Return to User Mode User Mode Programs Develop User Program Programs Program Loader

9 xii Contents 5.5 Create Process with User Mode Image Initialize Process User Mode Stack Execution of User Mode Image MTX5.0: Demonstration of Simple System Calls Validation of System Call Parameters fork-exec in Unix/Linux Implementation of fork in MTX Implementation of exec in MTX MTX5.1: Demonstration of fork-exec in MTX Command Line Parameters Simple sh for Command Execution vfork Implementation of vfork in MTX MTX5.3: Demonstration of vfork in MTX Threads Principle of Threads Advantages of Threads Threads Operations Implementation of Threads in MTX Thread PROC Structure Threads Management Functions Thread Creation Thread Termination Thread Join Operation Threads Synchronization Threads Scheduling Concurrent Program Examples Using Threads MTX5.4: Demonstration of Threads in MTX References Process Synchronization Concurrent Processes Process Synchronization Problem Critical Region Implementation of Critical Region Implementation of Process Synchronization Mechanisms Sleep/Wakeup Semaphore Semaphore Operations for Multiprocessor Systems Applications of Semaphores Semaphore Lock Mutex Lock Resource Management Wait for Interrupts and Messages

10 Contents xiii Process Cooperation Producer-Consumer Problem Reader-Writer Problem Other Synchronization Mechanisms High-level Synchronization Constructs Condition Variables in Pthreads Monitors Deadlock Deadlock Definition Necessary Conditions for Deadlock Deadlock Prevention Deadlock Avoidance Deadlock Detection and Recovery Deadlock Prevention in Concurrent Programs Livelock Starvation Process Synchronization in MTX Pipes Pipe Programming in Unix/Linux Pipe and the Producer-Consumer Problem Implementation of Pipes in MTX MTX6.pipe: Demonstration of Pipes in MTX Message Passing Asynchronous Message Passing Synchronous Message Passing MTX6.message1: Demonstration of Synchronous Message Passing Server-Client Message Passing Multi-level Server-Client Message Passing MTX6.message2: Demonstration of Server-Client Message Passing References Memory Management Goals of Memory Management Virtual Address and Physical Address Variable Sized Partitions Overlay Swapping Segmentation Paging One-Level Paging Multi-Level Paging Other Paging Schemes

11 xiv Contents 7.8 Demand-Paging and Virtual Memory Demand-Paging Page Replacement Rules Virtual Memory Based File Mapping Distributed Shared Memory System General Demand-Paging Model Memory Management in Real-Mode MTX Free Memory List Memory Allocation Memory Deallocation Memory Compaction Swapping MTX7.1. Demonstration of Memory Management in MTX Memory Management in User Space Expand Heap Size by sbrk() Reduce Heap Size by rbrk() Memory Protection in MTX MTX7.2. Demonstration of Memory Management in User Space References Interrupt Processing and Process Scheduling Interrupts Interrupts Hardware Interrupt Processing TIMER and Timer Service PC s Timer Operation Implementation of Timer in MTX Process Switch by Time Slice Implementation of Timer Service in MTX Timer Service in MTX Process Scheduling Process Scheduling Terminology Goals, Policy and Algorithms of Process Scheduling Process Scheduling in Time-Sharing Operating Systems Process Scheduling in Unix Process Scheduling in Linux Process Scheduling in MTX MTX8.sch1: Time Slice MTX8.sch2: Fixed Process Priority and Time Slice MTX3.sch3: Dynamic Process Priority and Time Slice Comparison of MTX Scheduling Algorithms References

12 Contents xv 9 Signals and Signal Processing Signals and Interrupts Examples of Unix/Linux Signals Signal Processing in Unix (Bovet and Cesati 2005) Implementation of Signals in MTX Signal Types in MTX Signals in PROC Resource Signal Origins in MTX Deliver Signal to Process Change Signal Handler in Kernel Signal Processing in MTX Dispatch Signal Catcher for Execution in User Mode MTX9.Signal: Demonstration of Signal Processing in MTX Divide Trap Catcher Other Traps Interval Timer and Alarm Signal Catcher The Kill Command References Device Drivers Device Drivers Console Display Driver MTX10.tv: Demonstration of Timer and Display Driver Keyboard Driver The Keyboard Hardware Interrupt-Driven Driver Design A Simple Keyboard Driver Using Sleep/Wakeup A Simple Keyboard Driver Using Semaphore Interrupt Handlers Should Never Sleep or Block Raw Mode and Cooked Mode Foreground and Background Processes Improved KBD Driver MTX10.vtk: Demonstration of Keyboard Driver Parallel Printer Driver Parallel Port Interface A Simple Printer Driver MTX10.pr: Demonstration of Printer Driver Serial Port Driver RS232C Serial Protocol Serial Data Format PC Serial Ports Serial Terminals

13 xvi Contents Serial Port Driver Design The STTY Structure Serial Port Initialization Lower-Half Serial Port Driver Upper-Half Serial Port Driver MTX10.serial: Demonstration of Serial Port Driver Floppy Disk Driver FDC Controller FDC R/W Operation Sequence FD Driver Design Implications on I/O Buffers in MTX Demonstration of FD Driver IDE Hard Disk Driver IDE Interface IDE R/W Operation Sequence A Simple HD Driver Improved HD Driver HD Driver with I/O Queue MTX10.hd: Demonstration of IDE Driver ATAPI Driver ATAPI Protocol [ATAPI 2006; AT Attachment 1998] ATAPI Operation Sequence ATAPI Driver Code MTX10.hd.cd: Demonstration of ATAPI Driver References File System File Operation Levels File I/O Operations EXT2 File System Specification A Simple mkfs Program Implementation of EXT2 File System File System Organization Files in the MTX/FS directory Implementation of Level-1 FS Mkdir-creat-mknod Chdir-getcwd-stat Rmdir Link-unlink Symlink-readlink Other Level-1 Functions Implementation of Level-2 FS Open-close-lseek Read Regular Files Write Regular Files Read-Write Special Files Opendir-readdir

14 Contents xvii Implementation of Level-3 FS Mount-umount Implications of mount File Protection Real and Effective uid File Locking Extensions of MTX File System References Block Device I/O and Buffer Management Block Device I/O Buffers Unix I/O Buffer Management Algorithm New I/O Buffer Management Algorithms Simple PV-Algorithm Optimal PV-Algorithm Comparisons of I/O Buffer Management Algorithms References User Interface User Interface in MTX User Mode Programs User Mode Programs for MTX Startup The INIT Program The login Program The sh Program Other User Mode Programs Summary of RMTX RMTX Source File Tree Process Management in RMTX Device Drivers File System Install RMTX RMTX Startup Sequence Recompile the RMTX Kernel Future Development MTX in 32-bit Protected Mode Introduction to 32-bit Protected Mode Memory Management in Protected Mode Segmentation Segment Descriptors Segmentation Models Paging Page Directory and Page Tables Address Translation in Paging Translation Lookaside Buffer (TLB)

15 xviii Contents Full Paging Demand-Paging Interrupt and Exception Processing Exceptions in Protected Mode Interrupt Descriptor Table (IDT) Task State Segment (TSS) TSS and Process Interrupt Stack Process Kernel Stack in Protected Mode GCC in 32-bit Mode Inline Assembly Constant, Volatile and Packed Attributes GCC Linker and Executable File Format MTX Kernel in 32-bit Protected Mode Protected Mode MTX Kernel Startup Sequence Interrupt and Exception Processing Remap IRQ Vectors IDT and Exception Handlers The Ultimate Exception Handler in C Interrupt Vectors and Handlers MTX32.1: 32-bit Protected Mode MTX Using Segmentation MTX32.1 Kernel Startup Sequence GDT for Segmentation Process TSS and LDT Switch TSS and LDT during Task Switch Changes in MTX32.1 Kernel Summary on Segmentation Demonstration System of MTX MTX32.2: 32-bit Protected Mode MTX using Static Paging MTX32.2 Virtual Address Spaces MTX32.2 Kernel Page Directory and Page Tables MTX32.2 User Mode Page Directory and Page Tables MTX32.2 Kernel Startup Sequence Process Page Directory and Page Table Process Interrupt and Resume Stack Frames Switch TSS and pgdir during Task Switch Changes in MTX32.2 Kernel Demonstration System of MTX PMTX: 32-bit Protected Mode MTX Using Dynamic Paging PMTX Virtual Address Spaces PMTX Kernel Startup Sequence PMTX Kernel Virtual Address Mapping PMTX Kernel Page Directory and Page Tables Manage Page Frames

16 Contents xix Changes in PMTX Kernel Page Faults and Demand-Paging Page Replacement Paging and I/O Buffer Management PMTX Demonstration System Extensions of 32-bit Protected Mode MTX bit Operating Systems x86-64 CPUs Function Call Convention in 64-bit Mode Summary References Symmetric Multiprocessing MTX Multiprocessor Systems SMP-Compliant Systems SMP System Startup Sequence SMP Data Structures Configure IOAPIC Configure Local APICs Send IPIs to Activate APs Trampoline Code of APs From UP to SMP Giant Lock Stage Coarse Grain Locking Stage Fine Grain Locking Stage SMP_PMTX in 32-bit Protected Mode Running PROCs and CPU Structures Spinlocks and Semaphores Use Sleep/Wakeup in SMP Protect Kernel Data Structures Adapt UP Algorithms for SMP Adapt UP Process Scheduling Algorithm for SMP Adapt UP Pipe Algorithm for SMP Adapt UP I/O Buffer Management Algorithm for SMP Device Driver and Interrupt Handlers for SMP SMP_PMTX Kernel Organization Bootable SMP_PMTX Image SMP_PMTX Kernel Startup Sequence Enter 32-bit Protected Mode Set up Initial Paging and GDTs Find Number of CPUs in SMP Initialize CPU Structures

17 xx Contents CPU Kernel Mode Page Table Load GDT and Enable Paging Initialize SMP Kernel Configure IOAPIC to Route Interrupts Configure APIC and APIC timer Start up APs APs Execute Startup Code APstart(cpuid) SMP_PMTX Demonstration System SMP_MTX using Parallel Algorithms Parallel Algorithms for Process Scheduling Parallel Algorithms for Resource Management Manage PROC Structures by Parallel Algorithm Manage In-memory Inodes by Parallel Algorithm Manage Page Frames by Parallel Algorithm Manage Other Resources by Parallel Algorithms Parallel Pipe Algorithm for SMP Parallel I/O Buffer Management Algorithms Unix MP Buffer Management Algorithm New Parallel I/O Buffer Management Algorithm for SMP I/O Buffer Management in SMP_MTX Parallel fork-exec Algorithms Parallel Algorithms for File System Functions Parallel mkdir-rmdir Algorithms Performance of Parallel Algorithms SMP_MTX Demonstration System SMP and Real-time Operating Systems SMP_MTX for Real-time Processing SMP in 16-bit Real Mode Real-mode SMP_RMTX startup sequence SMP_RMTX Kernel SMP_RMTX Demonstration System Limitations and Future Work Summary of PMTX and SMP_MTX PMTX and SMP_MTX Source File Tree PMTX and SMP_MTX Kernel Files Process Management in PMTX and SMP_MTX Process Scheduling in PMTX and SMP Device Drivers File System

18 Contents xxi PMTX and SMP Startup and login PMTX Startup Sequence SMP_MTX Startup Sequence User Interface and System Call Functions Recompile PMTX and SMP_MTX References Hybrid Operating Systems Monolithic Kernel Microkernel Comparison of Monolithic and Microkernel Based OS Hybrid Operating Systems References Appendix Index