Contents. Origins of Mac OS X. Chapter 1

Size: px

Start display at page:

Download "Contents. Origins of Mac OS X. Chapter 1"

Maximillian Craig
5 years ago
Views:

1 Chapter 1 Origins of Mac OS X 1.1 Apple s Quest for the Operating System Star Trek Raptor NuKernel TalOS Copland Gershwin BeOS Plan A 1.2 The NeXT Chapter NEXTSTEP OPENSTEP 1.3 The Mach Factor Rochester s Intelligent Gateway Accent Mach MkLinux Musical Names 1.4 Strategies Mac OS 8 and Rhapsody Blue Box Yellow Box 1.5 Toward Mac OS X vii

2 viii Contents Mac OS X Server 1.x Mac OS X Developer Previews DP DP DP DP Mac OS X Public Beta Mac OS X 10.x Mac OS X Mac OS X Mac OS X Mac OS X Mac OS X 10.4 Chapter 2 An Overview of Mac OS X 2.1 Firmware 2.2 Bootloader 2.3 Darwin Darwin Packages The Benefits of Darwin Darwin and Mac OS X 2.4 The xnu Kernel Mach BSD The I/O Kit The libkern Library The libsa Library The Platform Expert Kernel Extensions 2.5 A User-Space View of the File System File System Domains The User Domain The Local Domain The Network Domain The System Domain The /System/Library/ Directory

3 ix 2.6 The Runtime Architecture Mach-O Files Fat Binaries Linking Multiple Binding Styles Two-Level Namespaces Weakly Linked Symbols dyld Interposing 2.7 The C Library 2.8 Bundles and Frameworks Bundles Property List Files Frameworks Prebinding 2.9 Core Services 2.10 Application Services Graphics and Multimedia Services Quartz QuickDraw 2D OpenGL Core Image and Core Video QuickTime Core Audio Other Application Services 2.11 Application Environments BSD The X Window System Carbon Support for CFM Binaries Carbon APIs Cocoa Nib Files Core Data WebObjects Java QuickTime Classic Rosetta 2.12 User Interface Visual Effects

4 x Contents Resolution-Independent User Interface Productivity Features Fast User Switching Dashboard Spotlight Universal Access Support 2.13 Programming Xcode Compilers and Libraries Interpreters AppleScript Automator Command-Line Support Tools Debugging and Analysis Tools Computer Hardware Understanding Development Tools Visual Tools 2.14 Security Kernel-Space Security User-Space Security CDSA Plug-ins CSSM API Mac OS X Security APIs Security Server and Security Agent Using Authorization Services Miscellaneous Security-Related Features System Administration Interacting with the Security Framework Interacting with Directory Services Managing System Configuration The Auditing System 2.15 Mac OS X Server Xgrid Xgrid Architecture Xgrid Software Xsan Storage in Xsan Metadata Controllers Client Systems

5 xi Communication Infrastructure 2.16 Networking Chapter 3 Inside an Apple 3.1 The Power Mac G The U3H System Controller The K2 I/O Device Controller PCI-X and PCI Express PCI-X PCI Express HyperTransport Elastic I/O Interconnect 3.2 The G5: Lineage and Roadmap Fundamental Aspects of the G bit Processor Superscalar Speculative Execution Out-of-Order Execution New POWER Generations The PowerPC 970, 970FX, and 970MP The Intel Core Duo 3.3 The PowerPC 970FX At a Glance Caches L1 and L2 Caches Cache Properties Associativity Store Policy MERSI Memory Management Unit (MMU) SLB and TLB Address Translation Caching the Caches: ERATs Large Pages No Support for Block Address Translation Mechanism Miscellaneous Internal Buffers and Queues Prefetching Registers UISA and VEA Registers

6 xii Contents OEA Registers Rename Registers Instruction Set Fixed-Point Instructions Floating-Point Instructions Vector Instructions Control-Flow Instructions Miscellaneous Instructions The 970FX Core Instruction Pipeline 3 14 IFAR, ICA 3 14 D D1, D2, D Accounting for 215 In-Flight Instructions Branch Prediction Summary AltiVec Vector Computing The 970FX AltiVec Implementation AltiVec Instructions Power Management PowerTune Power Mac G5 Thermal and Power Management bit Architecture bit Features The 970FX as a 32-bit Processor Softpatch Facility 3.4 Software Conventions Byte Ordering Register Usage Indirect Calls Direct Calls Stack Usage Stack Usage Examples Printing Stack Frames Function Parameters and Return Values Passing Parameters Returning Values 3.5 Examples A Recursive Factorial Function

7 xiii An Atomic Compare-and-Store Function Function Rerouting Instruction Patching Constructing Branch Instructions Cycle-Accurate Simulation of the 970FX Chapter 4 The Firmware and the Bootloader 4.1 Introduction Varieties of Firmware Preferential Storage 4.2 A Whole New World New Is Good News Modern Boot ROM (PowerPC) 4.3 Power-On Reset 4.4 Open Firmware Interacting with Open Firmware Forth Shell TELNET TFTP Serial Download Open Firmware Emulators 4.5 Forth Cells Stacks Words Dictionary A Sampling of Built-in Words Stacks Memory Operators Console I/O Control Flow Searching the Dictionary Debugging 4.6 The Device Tree Properties Methods Data

8 xiv Contents 4.7 Open Firmware Interfaces The User Interface The Client Interface The Device Interface 4.8 Programming Examples Dumping NVRAM Contents Determining Screen Dimensions Working with Colors Drawing a Color-Filled Rectangle Creating an Animated Solution to the Towers of Hanoi Problem Fabricating and Using a Mouse Pointer Stealing a Font Implementing a Clock Drawing Images Creating Windows 4.9 Firmware Boot Sequence The Script Snag Keys 4.10 BootX File Format Structure Operation 4.11 Alternate Booting Scenarios Booting an Alternate Kernel NVRAM Caveats Booting from a Software RAID Device Booting over a Network 4.12 Firmware Security Managing Firmware Security Recovering the Open Firmware Password 4.13 Launching the Kernel 4.14 The BootCache Optimization 4.15 Boot-Time Kernel Arguments 4.16 The Extensible Firmware Interface Legacy Pains A New Beginning EFI EFI Services Boot Services

9 xv EFI Drivers A Sampling of EFI EFI NVRAM The Boot Manager The EFI Shell The GUID-Based Partitioning Scheme Universal Graphics Adapter EFI Byte Code Binary Format The Benefits of EFI Chapter 5 Kernel and User-Level Startup 5.1 Arranging for the Kernel to Execute Exceptions and Exception Vectors Kernel Symbols Run Kernel Run 5.2 Low-Level Processor Initialization Per-Processor Data Reset Types Processor Types Memory Patching Processor-Specific Initialization Other Early Initialization 5.3 High-Level Processor Initialization Before Virtual Memory Low-Level Virtual Memory Initialization Sizing Memory Pmap Initialization Starting Address-Translation After Virtual Memory Console Initialization Preparing for the Bootstrapping of Kernel Subsystems 5.4 Mach Subsystem Initialization Scheduler Initialization High-Level Virtual Memory Subsystem Initialization IPC Initialization Finishing VM and IPC Initialization Initializing Miscellaneous Subsystems Tasks and Threads

10 xvi Contents Launching the Kernel Bootstrap Thread 5.5 The First Thread 5.6 I/O Kit Initialization 5.7 BSD Initialization Miscellaneous BSD Initialization (Part 1) File System Initialization Miscellaneous BSD Initialization (Part 2) Networking Subsystem Initialization Miscellaneous BSD Initialization (Part 3) Mounting the Root File System Creating Process Shared Memory Regions 5.8 Launching the First User-Space Program 5.9 Slave Processors 5.10 User-Level Startup launchd Daemon Configuration and Management Daemon Creation launchd Operation Multiuser Startup User Login User Logout, System Restart, and System Shutdown Single-User Startup Installation Startup Chapter 6 The xnu Kernel 6.1 xnu Source 6.2 Mach Kernel Fundamentals Tasks and Threads Ports Messages Virtual Memory and Memory Objects Exception Handling 6.3 A Flavor of the Mach APIs Displaying Host Information Accessing the Kernel s Clock Services Using a Clock Service to Ring an Alarm

11 xvii Displaying Host Statistics 6.4 Entering the Kernel Types of Control Transfer External Hardware Interrupts Processor Traps Software Traps System Calls Implementing System Entry Mechanisms Exceptions and Exception Vectors Exception-Handling Registers System Linkage Instructions System Call Return from Interrupt Machine-Dependent Thread State Exception Save Areas 6.5 Exception Processing Hardware Interrupts Miscellaneous Traps System Calls 6.6 System Call Processing 6.7 System Call Categories BSD System Calls Data Structures Argument Munging Kernel Processing of BSD System Calls User Processing of BSD System Calls Mach Traps I/O Kit Traps PowerPC-Only System Calls Ultra-Fast Traps Fast Traps Blue Box Calls The Commpage 6.8 Kernel Support for Debugging, Diagnostics, and Tracing GDB (Network-Based or FireWire-Based Debugging) KDB (Serial-Line-Based Debugging) CHUD Support Task-Related Functions Thread-Related Functions Memory-Related Functions

12 xviii Contents CPU-Related Functions Callback-Related Functions Kernel Profiling (kgmon and gprof) Per-Process Profiling (profil(2)) Mach Task and Thread Sampling Per-Process Kernel Tracing (ktrace(2) and kdump) Auditing Support Fine-Grained Kernel Event Tracing (kdebug) kdebug Tracing kdebug Entropy Collection Low-Level Diagnostics and Debugging Interfaces Firmware Call Interface Diagnostics System Call Interface Low-Level Kernel Tracing Low-Memory Global Data Structures Low Tracing 6.9 Virtual Machine Monitor Features Using the VMM Facility Example: Running Code in a Virtual Machine 6.10 Compiling the Kernel Retrieving Prerequisite Packages Compiling Prerequisite Packages Compiling the xnu Package DarwinBuild Chapter 7 Processes 7.1 Processes: From Early UNIX to Mac OS X Mac OS X Process Limits Mac OS X Execution Flavors 7.2 Mach Abstractions, Data Structures, and APIs Summary of Relationships Processor Sets Representation The Processor Set API Processors Interconnections The Processor API Displaying Processor Information

13 xix Stopping and Starting a Processor in a Multiprocessor System Tasks and the Task API Threads The Thread API Kernel Threads Thread-Related Abstractions Remote Procedure Call Activation and Shuttle Thread Migration Continuations 7.3 Many Threads of a New System Mach Tasks and Threads Creating a Mach Task Creating a Mach Thread in an Existing Task Displaying Task and Thread Details BSD Processes The fork() System Call: User-Space Implementation The fork() System Call: Kernel Implementation The vfork() System Call POSIX Threads (Pthreads) The Pthreads API Implementation of Pthread Creation Java Threads The NSTask Cocoa Class The NSThread Cocoa Class The Carbon Process Manager Carbon Multiprocessing Services The Carbon Thread Manager 7.4 Scheduling Scheduling Infrastructure Initialization Timeslicing Quantum Timing and Clocks Converting between Absolute- and Clock-Time Intervals Starting the Scheduler Retrieving the Value of the Scheduler Tick Some Periodic Kernel Activities Scheduler Operation Priority Ranges

14 xx Contents Run Queues Scheduling Information in Tasks and Threads Processor Usage Accounting Scheduling Policies THREAD_STANDARD_POLICY THREAD_EXTENDED_POLICY THREAD_PRECEDENCE_POLICY THREAD_TIME_CONSTRAINT_POLICY Priority Recomputation on Policy Change Task Roles 7.5 The execve() System Call Mach-O Binaries Preparations for the Execution of a Mach-O File Loading the Mach-O File Handling Setuid and Setgid Execution Notification Configuring the User Stack Finishing Up Fat (Universal) Binaries Interpreter Scripts 7.6 Launching Applications Mapping Entities to Handlers Uniform Type Identifiers Chapter 8 Memory 8.1 Looking Back Virtual Memory and UNIX Virtual Memory and Personal Computing Roots of the Mac OS X Virtual Memory Subsystem 8.2 An Overview of Mac OS X Memory Management Reading Kernel Memory from User Space dd and /dev/kmem The kvm(3) Interface Querying Physical Memory Size 8.3 Mach VM Overview Task Address Spaces VM Maps

15 xxi VM Map Entries VM Objects Contents of a VM Object Backing Stores Pagers External Pagers A Pager s Port The Mach Pager Interface Copy-on-Write The Physical Map (Pmap) The Pmap Interface 8.4 Resident Memory The vm_page Structure Searching for Resident Pages Resident Page Queues Page Replacement Physical Memory Bookkeeping Page Faults 8.5 Virtual Memory Initialization during Bootstrap 8.6 The Mach VM User-Space Interface mach_vm_map() mach_vm_remap() mach_vm_allocate() mach_vm_deallocate() mach_vm_protect() mach_vm_inherit() mach_vm_read() mach_vm_write() mach_vm_copy() mach_vm_wire() mach_vm_behavior_set() mach_vm_msync() Statistics 8.7 Using the Mach VM Interfaces Controlling Memory Inheritance Debugging the Mach VM Subsystem Protecting Memory Accessing Another Task s Memory Naming and Sharing Memory 8.8 Kernel and User Address Space Layouts

16 xxii Contents 8.9 Universal Page Lists (UPLs) 8.10 Unified Buffer Cache (UBC) The UBC Interface The NFS Buffer Cache 8.11 The Dynamic Pager Program 8.12 The Update Daemon 8.13 System Shared Memory Applications of Shared Memory Implementation of the Shared Memory Server Subsystem shared_region_make_private_np() shared_region_map_file_np() load_shared_file() reset_shared_file() new_system_shared_regions() The Loading of Shared Object Files by the Dynamic Linker The Use of shared_region_map_file_np() by a System Application A Note on Prebinding 8.14 Task Working Set Detection and Maintenance The TWS Mechanism TWS Implementation 8.15 Memory Allocation in User Space A Historical Break Memory Allocator Internals Tiny Allocations Small Allocations Large Allocations Huge Allocations The malloc() Routine The Largest Single Allocation (32-bit) The Largest Single Allocation (64-bit) Enumerating All Pointers Displaying Scalable Zone Statistics Logging Malloc Operations Intercepting the Malloc Layer 8.16 Memory Allocation in the Kernel Page-Level Allocation kmem_alloc The Mach Zone Allocator

17 xxiii The Kalloc Family The OSMalloc Family Memory Allocation in the I/O Kit Memory Allocation in the Kernel s BSD Portion Memory Allocation in libkern s C++ Environment 8.17 Memory-Mapped Files bit Computing Reasons for 64-bit Computing bit Execution on 64-bit PowerPC Need for Address Space Large-File Support Mac OS X 10.4: 64-bit User Address Spaces Data Model Implementation Usage and Caveats Why Not to Use 64-bit Executables The 64-bit Scene Chapter 9 Interprocess Communication 9.1 Introduction The Evolution of IPC IPC in Mac OS X 9.2 Mach IPC: An Overview Mach Ports Ports for Communication Port Rights Ports as Objects Mach Port Allocation Mach IPC Messages Message Header Message Body Message Trailer 9.3 Mach IPC: The Mac OS X Implementation IPC Spaces IPC Entry Table IPC Entry Splay Tree The Anatomy of a Mach Port What s in a Port s Name?

18 xxiv Contents Validity of a Port Name Tasks and IPC Threads and IPC Port Allocation Messaging Implementation IPC Subsystem Initialization 9.4 Name and Bootstrap Servers The Network Message Server The Bootstrap Server The Bootstrap Port The Bootstrap Context Debugging the Bootstrap Server The Bootstrap Server API Displaying Information about All Known Services Creating a Crash-Resistant Server 9.5 Using Mach IPC A Simple Client-Server Example Dead Names Port Sets Interposition Transferring Out-of-Line Memory and Port Rights 9.6 MIG MIG Specification Files Using MIG to Create a Client-Server System MIG in the Kernel Interfaces to Kernel Objects MIG Initialization in the Kernel 9.7 Mach Exceptions Programmer-Visible Aspects of Mach s Exception- Handling Facility The Mach Exception-Handling Chain Delivering Exceptions Unresolved Kernel Traps Example: A Mach Exception Handler 9.8 Signals Reliability The Number of Signals Application-Defined Signals Signal-Based Notification of Asynchronous I/O Signals and Multithreading

19 xxv Signal Actions Signal Generation and Delivery Mach Exceptions and Unix Signals Living Together Exceptions, Signals, and Debugging The ptrace() System Call 9.9 Pipes 9.10 Named Pipes (Fifos) 9.11 File Descriptor Passing 9.12 XSI IPC 9.13 POSIX IPC POSIX Semaphores POSIX Shared Memory 9.14 Distributed Objects 9.15 Apple Events Tiling Application Windows Using Apple Events in AppleScript Building and Sending an Apple Event in a C Program Causing the System to Sleep by Sending an Apple Event 9.16 Notifications Foundation Notifications The notify(3) API Kernel Event Notification Mechanism (kqueue(2)) Core Foundation Notifications Fsevents Kauth 9.17 Core Foundation IPC Notifications The Run Loop Run-Loop Observers Run-Loop Sources CFMachPort CFMessagePort 9 66 CFSocket 9 67 CFRunLoopTimer 9.18 Synchronization Interfaces for Atomic Operations Low-Level Locking Spinlocks Mutexes Read/Write Locks

20 xxvi Contents Lock Groups and Attributes BSD Condition Variables Mach Lock Sets Mach Semaphores Pthreads Synchronization Interfaces Locking in the I/O Kit Funnels History Funnels in Mac OS X SPLs Advisory-Mode File Locking Chapter 10 Extending the Kernel 10.1 A Driver down the Memory Lane Driver Programming Considered Difficult Good Inheritance Everything Is a File There Is More to Extending the Kernel Than Driving Devices 10.2 The I/O Kit Embedded C I/O Kit Class Hierarchy I/O Kit Families I/O Kit Drivers Nubs General I/O Kit Classes Classes for Memory-Related Operations Classes for Synchronization and Serialization of Access Miscellaneous Classes The Work Loop The I/O Registry The I/O Catalog I/O Kit Initialization Driver Matching in the I/O Kit 10.3 DART 10.4 Dynamically Extending the Kernel The Structure of a Kernel Extension Creation of Kernel Extensions

21 xxvii Management of Kernel Extensions Automatic Loading of Kernel Extensions 10.5 Communicating with the Kernel 10.6 Creating Kernel Extensions A Generic Kernel Extension Implementing Sysctl Variables Using a Generic Kext I/O Kit Device Driver Kext 10.7 A Programming Tour of the I/O Kit s Functionality Rotating a Framebuffer Accessing Framebuffer Memory Retrieving the List of Firmware Variables Retrieving Information about Loaded Kernel Extensions Retrieving Accelerometer Data from the Sudden Motion Sensor Listing PCI Devices Retrieving the Computer s Serial Number and Model Information Retrieving Temperature Sensor Readings Retrieving MAC Addresses of Ethernet Interfaces Implementing an Encrypted Disk Filter Scheme 10.8 Debugging Kernel Panics Remote Core Dumps Logging Debugging by Using GDB Debugging by Using KDB Miscellaneous Debugging Tools Stabs Chapter 11 File Systems 11.1 Disks and Partitions The Apple Partitioning Scheme PC-Style Partitioning GUID-Based Partitioning 11.2 Disk Arbitration Retrieving a Disk s Description Participating in Disk Mounting Decisions Receiving Media Notifications from the I/O Kit

22 xxviii Contents 11.3 The Implementation of Disk Devices 11.4 Disk Images Using the hdiutil Program RAM Disks The BSD Vnode Disk Driver Creating a Virtual Disk from Scratch 11.5 Files and File Descriptors 11.6 The VFS Layer 11.7 File System Types HFS Plus and HFS ISO MS-DOS NTFS UDF UFS AFP FTP NFS SMB/CIFS WebDAV cddafs deadfs devfs fdesc specfs and fifofs synthfs union volfs 11.8 Spotlight Spotlight s Architecture The Fsevents Mechanism Importing Metadata Querying Spotlight Spotlight Command-Line Tools Overcoming Granularity Limitations 11.9 Access Control Lists The Kauth Authorization Subsystem Kauth Concepts Scopes and Actions Listeners and Authorization Decisions

23 xxix Implementation A Vnode-Level File System Activity Monitor Chapter 12 The HFS Plus File System 12.1 Analysis Tools HFSDebug Interface for Retrieving File System Attributes Mac OS X Command-Line Tools HFS+ Source and Technical Note TN Fundamental Concepts Volumes Allocation Blocks Extents File Forks Clumps B-Trees B+ Trees in HFS Nodes Records Searching 12.3 The Structure of an HFS+ Volume 12.4 Reserved Areas 12.5 The Volume Header Viewing the Volume Header Viewing a Volume Control Block 12.6 The HFS Wrapper 12.7 Special Files The Allocation File Viewing the Contents of the Allocation File The Roving Next-Allocation Pointer The Catalog File Catalog Node IDs Examining the Catalog B-Tree The Extents Overflow File Examining Fragmentation Examining the Extents Overflow B-Tree The Attributes File Working with Extended Attributes

24 xxx Contents Examining the Attributes B-Tree The Startup File 12.8 Examining HFS+ Features Case Sensitivity Filename Encodings Permissions Manipulating Volume-Level Ownership Rights Repairing Permissions Journaling Enabling or Disabling Journaling on a Volume Observing the Journal s Operation Quotas Hard Links Unlinking Open Files Symbolic Links Aliases Resource Forks 12.9 Optimizations On-the-Fly Defragmentation The Metadata Zone Hot File Clustering Hot File Clustering Stages The Hot Files B-Tree The Working of Hot File Clustering Miscellaneous Features Special System Calls Freezing and Thawing a Volume Extending and Shrinking a Volume Volume Notifications Support for Sparse Devices Comparing Mac OS X File Systems Comparing HFS+ and NTFS Appendix A Mac OS X on x86-based Macintosh Computers A.1 Hardware Differences A.2 Firmware and Booting A.3 Partitioning A.4 Universal Binaries

25 xxxi A.5 Rosetta A.6 Byte Ordering A.7 Miscellaneous Changes A.7.1 No Dual-Mapped Kernel Address Space A.7.2 Nonexecutable Stack A.7.3 Thread Creation A.7.4 System Calls A.7.5 No /dev/mem or /dev/kmem A.7.6 A New I/O Kit Plane

FIGURE 3 4 Caches and buffers in the 970FX FIGURE 3 5 Retrieving processor cache information using the sysctl command FIGURE 3 6 Address translation

FIGURE 3 4 Caches and buffers in the 970FX FIGURE 3 5 Retrieving processor cache information using the sysctl command FIGURE 3 6 Address translation Figures FIGURE 1 1 Microsoft Windows NT 3.1 FIGURE 1 2 Copland architecture FIGURE 1 3 BeOS FIGURE 1 4 NEXTSTEP FIGURE 1 5 The timeline of NeXT s operating systems FIGURE 1 6 OPENSTEP FIGURE 1 7 Mac OS