Teaching Operating Systems: The Windows Case

Transcription

1 Teaching Operating Systems: The Windows Case Andreas Polze Operating Systems and Middleware Hasso-Plattner-Institute at University Potsdam Potsdam, Germany Dave Probert Windows Kernel Architecture Microsoft Corporation One Microsoft Way, Redmond, Washington, USA ABSTRACT An operating system (OS) is a program that manages computer hardware. And although today s commercial-off-the-shelf desktop operating systems appear to be an integral part of PCs and workstation to many users, a fundamental understanding of the algorithms, principles, heuristics, and optimizations used is crucial for creating efficient application software. Furthermore, many of the principles in OS courses are relevant to large system applications like databases and web servers. Within this paper, we present our approach towards teaching OS concepts based on the Windows family of operating systems. In contrast to many stable Unix-based curricula, a Windows-based OS curriculum has to take into account the OS as a moving target. And although Windows source code has been made available to academic institutions, managing complexity is among the biggest challenges when teaching OS concepts based on Windows. Teaching experiences reported within this paper have lead to development of the Curriculum Resource Kit (CRK), an entire Windows-based OS curriculum that is freely available for download Categories and Subject Descriptors D.4.0 [Operating Systems]: OS principles, Concurrency, Scheduling and Dispatch, Memory Management, Device Management Experimental Evaluation, Source code analysis. General Terms: Algorithms, Documentation, Experimentation. Keywords: Windows, OS curriculum. 1 INTRODUCTION The roots of Windows 1 reach back to the late 1980s. Back then, many interesting things were happening in the operating system design space including SVR4, the Mach microkernel, a flurry of innovations in networking and windowing systems, and many research projects on OS fundamentals. The desire to gain in-depth knowledge of these exciting developments motivated many computer science students to study operating systems. Things have changed since then. Today most users do not even distinguish the computer and its operating system. And from the high level, GUI point of view, there are few apparent differences among the capabilities of today s off-the-shelf Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SIGCSE'06, March 1-5, 2006, Houston, Texas, USA. Copyright 2006 ACM /06/ $5.00. operating systems. Correspondingly, students often do not see the need to understand the internals of the OS they use. This does not bode well for OS courses, and we are working on changing this trend. Within this paper, we advocate a hands-on approach towards teaching (and learning) OS concepts. We present our experiences from teaching Windows-based OS courses during the last five years. We suggest a three-phase scheme of lectures, seminars, and source code labs to motivate students to learn specific OS implementation details. We have compiled publicly available lecture materials augmented by labs, quizzes, and assignments. The remainder of the paper is structured as follows: Section 2 reports on recent initiatives taken by Microsoft to support teaching OS classes based on Windows. Section 3 gives an overview of an envisioned OS course and takes a brief tour through the publicly available curriculum resource kit. Taking the topic scheduling and dispatch as an example, Section 4 looks into the detailed structuring of the lecture, discusses one lab scenario, and presents a potential homework assignment. Section 5 presents some student feedback aggregated over the last couple of years. Section 6 presents future work a completely new format of teaching OS classes by having students build their own OS from high-level abstractions such as processor or memory (ProjectOz). Section 7 concludes the paper. 2 TEACHING WINDOWS RECENT INITIATIVES The original Windows was built on extensions of the DOS codebase written for single-users running only a few applications, and lacked many standard OS features. Starting in 1988 Microsoft began development of a new code-base for Windows, called NT. The project was led by Dave Cutler, who was the primary architect of many significant operating systems at DEC, including RSX-11 and VAX/VMS. With the release of Windows XP in 2001 Microsoft moved to NT as its single version of Windows for general purpose computing, and began to deprecate DOS-based Windows. The NT kernel is highly asynchronous and preemptively multithreaded. The architecture is organized around a central facility for managing kernel/user references and access to kernel data structures. The I/O system is modular and extensible. The virtual memory API separates virtual addresses management from memory objects, and provides low-level access to addressspace and physical memory for large applications. CPU 1 Within this paper, Windows refers to Windows NT and its descendants Windows 2000/XP/Server 2003/Vista. 298

2 scheduling is thread-based, and carefully uses per-cpu data and lock-free synchronization for scalability. Starting in 2005 Microsoft has embarked on three complementary initiatives to improve the accessibility of the NT kernel for use by universities in instruction and research in operating systems. Microsoft s goals are to (1) improve the diversity of OS instruction by supporting inclusion of cases based on NT, (2) enable the use of Windows as a platform for teaching OS courses, and (3) help revive interest in operating systems topics among students by increasing the excitement and relevance of OS courses. The three initiatives provide curriculum materials illustrating OS principles with Windows, access to the core kernel source code, and an experimental environment based on the native NT API for building OS projects. 2.1 Curriculum Resource Kit (CRK) The Windows CRK is a collection of instructional material that follows the ACM/IEEE Operating System Body of Knowledge to illustrate OS concepts using Windows XP as a case study. The CRK is based on the book Windows Internals[1] and commercial classes by David A. Solomon and Mark E. Russinovich[2] as adapted for university use by Andreas Polze. The experiments, quizzes, and assignments, which are integrated with the course materials, have been tested over the last five years in an Operating Systems Architecture class described in this paper. The CRK Core units are available to universities for supplementing OS lectures and may be downloaded from Microsoft's Curriculum Repository [3]. 2.2 Windows Research Kernel (WRK) The WRK contains the bulk of the source code for the NT kernel (compatible with Windows XP for x86 and AMD64). These sources include all the core sources for object management, processes, threads, virtual memory, the I/O system, etc. The major pieces of kernel code that are not included are the plugand-play/power-management facilities, the virtual DOS machine, and the kernel debugger engine. The omitted modules are provided as binary objects which can be linked to produce a fully functional NTOS executable which can be booted on Windows XP. The initial release of the WRK in the fall of 2005 included just the sources, for complementary use with the CRK, and use by authors of textbooks on Windows. In the spring of 2006 Microsoft will release the WRK in a kit with a build environment, VirtualPC, sources for sample device and file system drivers (from the Device Driver Kit and File System KIT), as well as better documentation, example source-based projects, and kernel regression tests. The WRK is licensed under a simple, more liberal license than previous Microsoft product sources. The WRK license grants wide non-commercial use of the sources, including sharing of derivatives with other faculty, open use in the classroom and by students, and publication of code snippets in textbooks and research papers. 2.3 ProjectOZ When NT was developed it wasn t clear what OS API it would provide (the original guess was OS/2). To accommodate this uncertainty NT was designed using a subsystem model, like Mach. NT is not a microkernel, but it does use a client/server model to implement OS APIs in user-mode server processes. The NT kernel itself has a native API, but apart from use by device drivers little has been published -- since, unlike the Win32 API, applications based on the NT API are not portable to Windows 9x. Because the native NT API has to support the subsystem model, it tends to be lower-level and give the programmer more direct access to the underlying kernel data structures and mechanisms. ProjectOZ is an experimental environment that is built on top of the native NT API for universities to use as a platform for OS projects. It uses the NT API to provide simple abstractions of a CPU, trap mechanism, and MMU using the same approach taken by the SPACE project at UC Santa Barbara [4]. ProjectOZ comes with a basic OS (BasicOZ) built on top of the SPACE abstractions. Various projects are suggested which allow the student to experiment with OS data structures and algorithms by modifying BasicOZ and measuring the resulting change in system behavior. The principles behind the ProjectOZ approach are use of abstraction to minimize the amount of low-level coding necessary and significant reduction in the amount of code that students must read in order to proceed with their projects. 3 WINDOWS COURSE STRUCTURING Commercial off-the-shelf operating systems are complex software systems of remarkable size. Gaining a thorough understanding of an OS s design rationales, algorithms, heuristics, and implementation strategies from just reading source code is at least difficult. Nevertheless, students should be able to learn about these details in terms of case studies. Our approach to teaching an OS curriculum is three-fold: We start with a two-semester undergraduate lecture series on the various aspects of an OS. Classroom experiments, labs, and homework assignments accompany this series. Also, the first semester of the lecture series is mandatory for B.Sc students at our institution.. A seminar on operating system services and administration is the second offering to students. Within this seminar, students are expected to set up and experimentally evaluate systems. Interoperability in heterogeneous settings is of central concern. The last step in our OS curriculum is a source-code analysis project. Students are asked to explain the exact implementation of certain OS functionality based on the actual source code. 3.1 The Lecture Series We are using Windows as the primary vehicle for explaining OS concepts. What started out as an add-on to a traditional Unixbased OS curriculum has evolved to a self-contained OS course. The effect on the course structuring is interesting: whereas in the first case it was sufficient to cover Windows details only, the latter one requires the incorporation of a fair amount of Unix know how (i.e.; scheduler implementation, file system, interprocess communication, networking) in order to explain to students the rationales for certain Windows design decisions. The general structuring of our OS lecture follows the ACM OS Body of Knowledge (OS BoK): OS1. Overview of operating systems [core] OS2. Operating system principles [core] OS3. Concurrency [core] OS4. Scheduling and dispatch [core] OS5. Memory management [core] OS6. Device management [elective] 299

3 OS7. Security and protection [elective] OS8. File systems [elective] OS9. Real-time and embedded systems [elective] OS10. Fault tolerance [elective] OS11. System performance evaluation [elective] OS12. Scripting [elective] However, we found the incorporation of supplementary units covering networking, interoperability, and a comparison of Windows and Unix implementations helpful and valuable. In addition to the OS BoK structuring with core/elective units, we have separated basic and advanced topics this allows for tailoring a one-semester course without omitting too many of the elective units. 3.2 The Seminar Typically, about 20% of a student population will develop an in-depth interest in OS internals, services, and administration. Students attending our OS services & administration seminar are given access to lab computers with full privileges and are expected to give a talk reporting the results of their experiments including setup of services, recovery, packaging and deployment of software. Seminar topics are closely linked to current research [5][6], a number of Ph.D. students act as supervisors and teaching assistants this influences definition of prospective topics. A few recent topics for talks: Microsoft Installer & Windows packaging Kerberos Windows/Unix interoperability Distributed File Systems the Andrew file system Shells & Scripting Windows Scripting Host, WMI, /bin/sh, the OpenVMS Digital Command Language Root Kits Virtualization Virtual PC, VMware, Solaris zones 3.3 The Source-code Analysis Project Microsoft offers access to Windows source code for qualifying academic institutions. Using this opportunity, we offer really interested students a source code analysis project seminar. Within the seminar, students investigate and report about central mechanisms, such as the interrupt processing, scheduler implementation, memory management, and Deferred Procedure Calls in Windows. Also, students are expected to make changes to the OS and build, install, and run their own OS versions. We have an isolated lab setup for source code experiments. Students are required to sign a non-disclosure agreement. About 5% of a student population takes the challenge of studying sources of a complex commercial OS. And, although it is interesting to study Windows sources, other OSes are more readily available to students (i.e.; Linux, Solaris, Mac OS X). However, access to Windows sources will be further simplified in the future (see Section 2). 4 UNIT OS 4 A CLOSER LOOK In context of our curriculum at HPI, we offer a lecture Operating System Architecture. This lecture constitutes the foundation of our OS-related teaching activities and has lead to creation of the curriculum resource kit (CRK)[3]. Commercial classes given by David A. Solomon and Mark E. Russinovich[2] form the basis for the majority of CRK course materials. Here, we are taking a closer look at Unit OS 4, Scheduling and Dispatch and discuss typical learning styles and presentation techniques. Unit OS 4 is structured as follows: 4.1. The Concept of Processes and Threads (Core) 4.2. Windows Processes and Threads (Core) 4.3. Windows Process and Thread Internals (Core/Advanced) 4.4. Windows Thread Scheduling (Core) 4.5. Advanced Windows Thread Scheduling (Core/Advanced) 4.6. Scheduling and Dispatch labs, quizzes, and assignments In 4.1, we introduce basic concepts, such as the Process Concept, Thread States, Context Switches, Approaches to CPU Scheduling, and Multithreading Models. Section 4.1 is independent of any particular OS, however, it relates concepts to their implementations in Windows and Unix (Linux). Section 4.2 focuses on the notions of processes, threads, jobs, and fibers including their APIs and monitoring instrumentation in Windows, whereas Section 4.3 discusses data structures and kernel representation of the above-mentioned abstractions and is augmented by kernel debugger demos and experiments. Section 4.4 discusses the general uni-processor scheduling algorithms and their implementation, including data structures, in Windows. In contrast, Section 4.5 focuses on multiprocessor scheduling and the various optimizations recently added to Windows in order to enhance performance and scalability. Section 4.6 finally consists of numerous labs, quizzes, and two series of homework assignments one investigating concepts and algorithms, the second consisting of programming assignments. (Solutions to quizzes and assignments are made available as an instructor supplement to the CRK). 4.1 Scheduling Lecture - Excerpt Within our lecture, this Section discusses Windows Scheduling Principles, Priorities (Windows API vs. NT Kernel), Scheduling Data Structures, Scheduling Scenarios, and Priority Boosts and Priority Adjustments. Figure 1 shows the thread state diagram. In contrast to many other operating systems, Windows is thoroughly instrumented. Tools, such as the performance monitor can be used to investigate a multitude of performance counters, thus gathering data about the internal state of the Figure 1: Scheduling state diagram Windows OS. In our lecture, after describing the scheduling algorithm used by Windows and the expected system behavior, we incorporate labs that dig into an actual live system. For example, we use performance monitor to display scheduling state changes in real-time (10ms resolution). CPU utilization, fairness, throughput, and minimal response time are typical optimization criteria for CPU schedulers. Most scheduling algorithms try to achieve these goals by dynamically assigning priorities to processes. There are two general classes of scheduler implementation: schedulers based on priority-elevation (1) and decreasing priority schedulers with aging (2). The VMS and Windows schedulers fall into class 300

4 (1), whereas Mach, Unix, and Linux fall into class (2). Since schedulers are workload-dependent and based on heuristics, there exist many variations within both classes (see Figure 2). Priority Base Priority Figure 2: Priority boosting and decay We discuss priority boosting in context of the Windows scheduler implementation and demonstrate the system behavior in a lab using performance counter data (Figure 3). Select labs are demonstrated during the lecture, however, students are also expected to experiment on their own. In some cases, this requires an elevation of privileges for the students accounts. CpuStres settings: 4.2 Quizzes and Assignments From a student s perspective, quizzes and assignments provide a valuable mechanism to identify facts he is expected to know. We provide two example quizzes and assignments for Unit OS 4 Scheduling and Dispatch : Assignment 4.1.3: two active threads activity level = busy (about 25% wait time) normal process priority class, normal thread priorities Usually only visible in PerfMon if target app owns foreground window (hence longer quantum) These are showing +2 boost (from 8 to 10) for foreground apps after wait completion Boost upon wait complete quantum Run Wait Run Priority decay at quantum end Preempt (before quantum end) Figure 3: Observing the scheduler Consider a uni-processor system that uses a round-robin algorithm with 16 priorities (0-15, 0 = lowest, 15 = highest priority) for thread scheduling. The lengths of the time quantum shall be 20ms. Context switching time is negligible. Running threads will not be interrupted until quantum end (Scheduling decisions are only be made at quantum end). Our system has a workload of three single-threaded processes. These processes (threads) have the following parameters: Thread ID Ready at Execution time Th1 t = 0ms 70ms Th2 t = 15ms 90ms Th3 t = 30ms 80ms Round base p - Draw a Gantt chart illustrating the execution of these threads under the assumption that they all are executed at a static priority of 8. - How does the execution order change if Th3 is executed with priority 9. Th1 shall be executed with priority 7 and get into an I/O wait state after 16ms execution time. Th1 s priority is being boosted by 3 after I/O completion. Th1 leaves its wait state at t=50ms. Th1 s priority then shall be decreased by 1 at Run quantum end until it reaches its base priority of 7. Draw the corresponding Gantt chart. Assignment 4.2.3: Design and implement a version of the UNIX time-command (mytime.exe) using the Windows API. The mytime-command should interpret its arguments as a program that has to be executed within a separate process. Mytime.exe shall call GetSystemTime() before creation and after completion of the new process. Use the Winodws-functions CreateProcess() and WaitForSingleObject() to synchronize mytime.exe with the newly created process. A few examples for quizzes are given below: Thread Quiz: Which information is not associated with a thread s Thread Control Block (TCB): a. Program counter b. CPU registers c. Memory management information d. CPU scheduling information e. Pending I/O information Process Data Structures Quiz: For a Windows process, the OS maintains various data. Which information is associated with the executive process object? a. Access Token b. List of Virtual Address Descriptors (VADs) c. List of Threads d. List of open handles e. List of open windows Solutions to assignments and answers to quizzes are provided separately as instructor supplement to the CRK. 5 STUDENT FEEDBACK We have gained experience teaching Operating System Architecture classes in two different contexts: First, as an extension to a classical, Unix-based CS curriculum (at Humboldt University Berlin, Germany) and secondly as the basic, undergraduate OS lecture (at Hasso-Plattner-Institute at University Potsdam, Germany). Feedback varies considerably. A Windows-based OS class as addition to the Unix-based curriculum generates an audience of 30-40% of a students population. These students tend to be highly motivated. They ask for every single implementation detail and are easily able to compare Windows concepts with those found in other OSes. General student feedback: The class (core+advanced) contains lots of details; experiments are very helpful to get hands-on experience; typical self-study time is one hour per lecture hour. A Windows-based OS class as the only (mandatory) OS class in a curriculum presents different challenges. Historic notes, abstract presentation of concepts, and comparison of approaches taken in other OSes are crucial for most students. General student feedback: Many implementation details (esp. advanced sections); sometimes difficult to connect concepts and implementation; programming assignments are (too) hard; time spent on self-study: two hours per lecture hour or more. Obviously, it would be perfect to have the Windows class as an add-on to a Unix class and draw only enthusiastic students as audience. Most students reported, that they gained a much better understanding of the OS internals and behavior. One of the most rewarding comments we actually received was: This was the by far most challenging lecture during my study 301

5 so far; (sometimes confusingly) many implementation details were presented; however, I assume that the knowledge from this lecture will help immensely with future complex software engineering projects. 6 FUTURE WORK ProjectOz ProjectOZ is an extensible collection of projects for investigating OS principles. It is based on the SPACE project at UC Santa Barbara which investigated building operating systems without monolithic kernels [7]. The basic model of a processing unit in SPACE is as follows: RETI External interrupts implementations according to CPU time, context switches, page faults, disk seeks, etc. workloads workloads workloads ProjectOZ ProjectOZ ProjectOZ BasicOZ BasicOZ BasicOZ SPACE.exe SPACE.exe SPACE.exe NT native API CPU MMU MEMORY TRAP handler Figure 4: Processing model in SPACE The trap vector and MMU define the execution environment of the CPU. SPACE provides abstractions of these elements rather than the more complex abstractions of process/virtual-memory, thread, and RPC presented by most OSes. ProjectOZ builds a SPACE-like set of abstractions on top of the native NT API. CPU uses native NT threads and MMU uses mapped views. TRAPS generalized to cross-domain portals using the NT subsystem exception ports, and physical MEMORY is built using a shared NT memory section The SPACE abstractions are implemented as a subsystem program which manages the abstractions and implements portal traversal. The subsystem also provides basic device models, including facilities for device registers, interrupts, and DMA. ProjectOZ takes the space abstractions and builds a basic OS on top, called BasicOZ, with features such as the following: User and Kernel address spaces, including system call stubs that traverse a portal to the kernel space and a basic handletable for referring to kernel data objects System devices such as timer, clock, console, disk, NIC with simple device drivers using DMA, interrupts, and IRQLs Trivial pre-populated filesystem with one directory, simple read/write, and without operations such as rename Threads, but no preemption and no guard pages on the stack Processes with a single thread and static images Virtual memory has no sharing, simple pagefile, no caching of pages, synchronous writes, and random replacement The algorithms and data structures are as simple as possible, using linked lists everywhere. This is in keeping with two primary goals: minimizing the amount of code to be read and convincing students they can easily improve BasicOZ. ProjectOZ comes with SPACE, BasicOZ, and a set of workload programs which exercise the system. There is a smorgasbord of projects implementing various enhancements to BasicOZ to produce different ProjectOZ systems. Using the networking device model in SPACE, multiple SPACE programs can communicate as if part of a distributed system despite running on the same machine. This provides support for experiments in networking protocols and distributed systems. SPACE.exe contains a lot of instrumentation which can be used to measure different aspects of system performance. Students are able to contrast various algorithms and NT Kernel Figure 5: BasicOZ structure Finally, ProjectOZ is intended to be an open-ended platform that faculty can extend by adding device models, specifying new projects, and even fundamentally improving BasicOZ and SPACE.exe itself. 7 CONCLUSIONS We have presented our approach towards teaching OS concepts based on the Windows family of operating systems. The authors suggest a physician s approach to investigating an OS: besides lecture materials discussing principles and concrete implementation details, lab and programming assignments are given to students, requiring them to experimentally evaluate certain hypothesis about the OS implementation and behavior. Student feedback from our Windows-based Operating Systems Architecture classes has been quite rewarding; understanding the inner-workings of the OS (almost) everybody uses has sparked considerable interest in OS-related projects and theses. With the freely available Curriculum Resource Kit (CRK) materials, it is relatively easy to add some Windows into existing OS courses. Supporting the CRK, ProjectOz, and other research initiatives, Microsoft has embarked on providing access to the core kernel source code for instruction and research projects. 8 REFERENCES [1] Mark E. Russinovich and David A. Solomon, Microsoft Windows Internals, 4th Edition, Microsoft Press, [2] David A. Solomon and Mark Russinovich: Windows OS Internals courses at [3] David A. Solomon, Mark E. Russinovich, Andreas Polze: CRK in Microsoft's Curriculum Repository at [4] Probert, Dave, Bruno, John, and Karaorman, Murat Space: A new approach to operating system abstraction. In Proc. of the Intl. WS on Object Orientation in OSes, Oct [5] Andreas Rasche, Marco Puhlmann and Andreas Polze; Dynamic Reconfiguration of Component-based Real-time Software in Proc. of WS on Object-oriented Dependable Real-time Systems (WORDS 2005), Feb [6] Wolfgang Schult and Andreas Polze; Aspect-Oriented Programming with C# and.net in Proc. of Intl. Symp. on Object-oriented Real-time distr. Comp. (ISORC) May [7] Probert, Dave Efficient Cross-domain Mechanisms for Building Kernel-less Operating Systems. Ph.D. Thesis, University of California, Santa Barbara,