Threads, System Calls, and Thread Switching

Size: px

Start display at page:

Download "Threads, System Calls, and Thread Switching"

Clement Underwood
6 years ago
Views:

1 µ-kernel Construction (2) Threads, System Calls, and Thread Switching (updated on )

2 Review from Last Lecture

3 The 100-µs Disaster 25 MHz MHz MHz Pentium 133 MHz Alpha 3

4 C Costs (486, 50 MHz) [µs] Mach [µs] 400 Mach L msg len L4 0 raw copy msg len 4

5 Analysis Mach uses asynchronous C Data buffered in kernel Cache destruction, infinite resources Memory oybus traffic Workplace OS switched to synchronous C Performance still horrible Why? Microkernel construction is difficult! 5

6 Size Comparison Mach 4 x86: 90,000 Xen 2.0 hypervisor: 45,000 L4Ka::Pistachio x86: 45,000 6

7 Threads and How to Switch Them

8 Fundamental Abstractions Thread Address space What is a thread? How to implement it? What conclusions can we draw from our analysis with rect to µ-kernel construction? 8

9 Fundamental Abstractions A thread is an independent flow of control inside an address space. Threads are identified by unique identifiers and communicate via C. Threads are characterized by a set of registers, including at least an instructioni pointer, a stack pointer and state information. A thread s state also includes the address space in which the thread currently executes. 9

10 Thread Properties Regs Memory Internal properties External properties Register set General purpose registers,, and Stack Status Flags, privilege, etc. OS-specific state Priority, time, etc. Address space Unique ID Communication status 10

11 Construction Conclusion Thread state must be saved/restored on thread switch We need a Thread Control Block (TCB) per thread TCBs must be kernel objects TCBs implement threads We often need to find Any thread s TCB using its global ID The currently executing thread s TCB (per processor) At least partially. We have found some good reasons to implement parts of the TCB in user memory ( local C). 11

12 Thread Switch A B Processor tcb A user mode A Memory tcb B 12

13 Thread Switch A B Processor tcb A user mode A kernel Memory tcb B 13

14 Thread Switch A B Processor tcb A user mode A kernel Memory tcb B 14

15 Thread Switch A B Processor tcb A user mode A kernel Memory tcb B 15

16 Thread Switch A B Processor tcb A user mode A kernel user mode B Memory tcb B 16

17 Thread Switch A B Thread A is running in user mode Thread A experiences its end of time slice or is preempted by a (device) interrupt We enter kernel mode The microkernel saves the status of thread A on A s TCB The microkernel loads the status of thread B from B s TCB We leave kernel mode Thread B is running in user mode 17

18 Thread Switch A kernel B Processor tcb A user mode A Memory 18

19 Thread Switch A kernel B Processor? tcb A user mode A kernel Memory 19

20 Thread Switch A kernel B Processor Kernel code Kernel stack tcb A user mode A kernel Memory 20

21 Thread Switch A kernel B Processor Kernel code Kernel stack tcb A user mode A kernel Memory 21

22 Thread Switch A kernel B Processor Kernel code Kernel stack tcb A user mode A kernel Memory 22

23 Thread Switch A kernel B Processor Kernel code tcb A Kernel stack user mode A kernel Memory 23

24 Thread Switch A kernel B Processor Kernel code tcb A Kernel stack user mode A kernel tcb B Memory Kernel stack 24

25 Thread Switch A kernel B Processor Kernel code tcb A Kernel stack user mode A kernel tcb B Memory Kernel stack 25

26 Construction Conclusion From the view of the designer there are two alternatives: Single Kernel Stack Only one stack is used in kernel mode all the time Per-Thread Kernel Stack Each thread has its own stack in kernel mode 26

27 Single Kernel Stack Per processor, event model Either continuations Complex to program Or stateless kernel No kernel threads, kernel not interruptible, difficult to program Structurally inefficient system calls + Kernel can be exchanged on-the-fly E.g. the fluke kernel from Utah Low cache footprint The same stack is always used! 27

28 Per-Thread Kernel Stack Thread model Simple, elegant, flexible Conclusion: We have to look for a solution that minimizes the kernel stack size Kernel can always use threads, no special methods required for keeping state while interrupted/blocked No conceptual difference between kernel mode and user mode Larger cache footprint Difficult to exchange kernel on-the-fly Conclusion: Either no persistent TCBs or TCBs must hold symbolic addresses 28

29 Kernel Entry and Exit on IA-32

30 Kernel Entry (IA-32) CPU eflags eax ebx ecx edx ebp esi edi user mode tcb A kernel code user stack 0 Trap/fault occurs (int n / exception / interrupt) points to the currently running thread s kernel stack 30

31 Kernel Entry (IA-32) CPU eflags eax ebx ecx edx ebp esi edi kernel mode tcb A kernel code ss user stack 0 Trap/fault occurs (int n / exception / interrupt) Push user SS:E onto kernel stack, load kernel SS:E 31

32 Kernel Entry (IA-32) CPU eflags eax ebx ecx edx ebp esi edi kernel mode tcb A kernel code flg ss user stack 0 Trap/fault occurs (int n / exception / interrupt) Push user SS:E onto kernel stack, load kernel SS:E Push user E, reset flags (I := 0, CPL := 0) 32

33 Kernel Entry (IA-32) CPU eflags eax ebx ecx edx ebp esi edi kernel mode tcb A kernel code cs flg ss user stack 0 Trap/fault occurs (int n / exception / interrupt) Push user SS:E onto kernel stack, load kernel SS:E Push user E, reset flags (I := 0, CPL := 0) Push user CS:E, load kernel entry CS:E 33

34 Kernel Entry (IA-32) CPU eflags eax ebx ecx edx ebp esi edi kernel mode tcb A X kernel code cs flg ss user stack 0 Trap/fault occurs (int n / exception / interrupt) Push user SS:E onto kernel stack, load kernel SS:E Push user E, reset flags (I := 0, CPL := 0) Push user CS:E, load kernel entry CS:E Push X: error code (hw, at exception) or kernel-call type hardware programmed, single instruction 34

35 Kernel Entry (IA-32) CPU eflags eax ebx ecx edx ebp esi edi kernel mode tcb A edi eax X kernel code cs flg ss user stack 0 Trap/fault occurs (int n / exception / interrupt) Push user SS:E onto kernel stack, load kernel SS:E Push user E, reset flags (I := 0, CPL := 0) Push user CS:E, load kernel entry CS:E Push X: error code (hw, at exception) or kernel-call type Push registers (optional) hardware programmed, single instruction 35

36 System Call (IA-32) int $0x32 push $X pusha Error code, e.g., 3 means page fault Push all the register content to the stack User mode Kernel mode popa addl $4, % iret Pop all registers E := E + 4 Restore the old E, discard error code $X Interrupt return: single instruction that restores user SS:E and CS:E from kernel stack 36

37 Kernel Stack State Uniprocessor: Any kernel stack myself is up-to-date! myself s kernel stack is up-to-date when in kernel mode except for brief moment. One thread running All the others in their kernel-state We can analyze their kernel stacks (e.g., L4 ExchangeRegisters(), kdb) All processes except the one running are in kernel mode tcb edi eax 3 cs flg ss 37

38 Kernel Stack State Uniprocessor: Any kernel stack myself is up-to-date! myself s kernel stack is up-to-date when in kernel mode except for brief moment. X permits to differentiate between stack layouts Interrupt, exception, some system calls C V86 mode tcb edi eax 4 cs flg ss ds es fs gs tcb 5 cs flg ss tcb edi eax 3 cs flg ss 38

39 Kernel Stack State Uniprocessor: Any kernel stack myself is up-to-date! myself s kernel stack is up-to-date when in kernel mode except for brief moment. X permits to differentiate between stack layouts Interrupt, exception, some system calls C V86 mode tcb edi eax 4edi eax cs flg 4 ss ds cs es flg fs gs ss ds es fs gs tcb 5 cs flg ss tcb edi eax 3 cs flg ss 39

40 Thread Switch on IA-32

41 Locating the TCB Remember: We need to find Any thread s TCB using its global ID The currently executing thread s TCB Next lecture Kernel E Align TCBs: movl %, %ebp andl -sizeof_tcb, %ebp tcb 0 41

42 Thread Switch (IA-32) Thread A int $0x32 pushl $X pusha movl %, %ebp andl -sizeof_tcb, %ebp /* edi == address of B s TCB */ movl %, OFF_E(%ebp) movl OFF_E(%edi), % addl sizeof_tcb, %edi movl %edi, 0 Switch current kernel stack pointer Thread B Switch E0 so that next kernel entry uses new kernel stack popa addl iret $4, % int $0x32 42

43 Switching Threads (IA-32) CPU eflags tcb tcb edi eax Kernel code X cs flg ss 0 eax ebx ecx edx ebp esi edi user stack user stack int $0x32 43

44 Switching Threads (IA-32) CPU eflags tcb tcb edi eax Kernel code X cs cs flg flg ss ss 0 eax ebx ecx edx ebp esi edi user stack user stack int $0x32 44

45 Switching Threads (IA-32) CPU eflags tcb tcb edi eax edi eax Kernel code X X cs cs flg flg ss ss 0 eax ebx ecx edx ebp esi edi user stack user stack int $0x32, push registers of blue thread 45

46 Switching Threads (IA-32) CPU eflags tcb tcb edi eax edi eax Kernel code X X cs cs flg flg ss ss 0 eax ebx ecx edx ebp esi edi user stack user stack int $0x32, push registers of blue thread Switch kernel stacks (store and load E) 46

47 Switching Threads (IA-32) CPU eflags tcb tcb edi eax edi eax Kernel code X X cs cs flg flg ss ss 0 eax ebx ecx edx ebp esi edi user stack user stack int $0x32, push registers of blue thread Switch kernel stacks (store and load E) Set E0 to new kernel stack 47

48 Switching Threads (IA-32) CPU eflags tcb tcb edi eax Kernel code X cs cs flg flg ss ss 0 eax ebx ecx edx ebp esi edi user stack user stack int $0x32, push registers of blue thread Switch kernel stacks (store and load E) Set E0 to new kernel stack Pop red registers 48

49 Switching Threads (IA-32) CPU eflags tcb tcb edi eax Kernel code X cs flg ss 0 eax ebx ecx edx ebp esi edi user stack user stack int $0x32, push registers of blue thread Switch kernel stacks (store and load E) Set E0 to new kernel stack Pop red registers, return to red user thread (iret) 49

50 Fast System Calls on IA-32

51 Syscalls via int n are too slow int n touches too much memory (cache, TLB) Save user mode CS:E, SS:E, E Load kernel mode SS0:E0 (from TSS) Load kernel mode CS:E (from IDT) Does too much (microcode) Segmentation not used today Many checks could be avoided 51

52 Sysenter/sysexitsysexit (IA-32) Model Specific Register (MSR) Kernel Kernel Flat 4 GB segments Kernel 4 GB Sysenter E := MSR[Kernel ] E := MSR[Kernel ] EFlags.I := 0, CPL := 0 Interrupts off Sysexit E := ECX E := EDX CPL := 3 Enter kernel mode User 0 GB 52

53 Sysenter/sysexitsysexit (IA-32) Model Specific Register (MSR) Kernel Kernel Flat 4 GB segments Kernel 4 GB Sysenter E := MSR[Kernel ] E := MSR[Kernel ] EFlags.I := 0, CPL := 0 User has to provide current user and By convention (e.g., ECX, EDX) User Flags undefined Sysexit E := ECX E := EDX CPL := 3 Kernel has to re-enable interrupts 0 GB 53

54 Sysenter/sysexitsysexit and int n Emulate effects of int instruction after sysenter (assuming ECX=U, EDX=U) subl $20, % movl %ecx, 16(%) movl %edx, 4(%) movl $5, (%) iret instruction movl 16(%), %ecx movl 4(%), %edx sti sysexit Emulate iret E tcb 5 cs flg ss 54

55 Sysenter/sysexitsysexit and int n Emulate effects of int instruction after sysenter (assuming ECX=U, EDX=U) movl (%), % subl $20, % movl %ecx, 16(%) movl %edx, 4(%) movl $5, (%) iret instruction movl 16(%), %ecx movl 4(%), %edx sti sysexit Emulate iret Trick: MSR points to TSS.0 + Avoids slow updates of the MSR + Update only TSS.0 on address space switch + Keeps MSR (used by sysenter) and TSS.0 (used otherwise) in sync - Requires memory access tcb 5 cs flg ss 55

56 Case study: Itanium (aka IA-64) Thread Switching and Kernel Entry

57 IA-64 User Accessible Registers General Registers Floating-point Registers Predicates Branch Registers Application Registers gr 0 gr 1 gr fr 0 fr 1 fr pr 0 pr 1 pr 2 pr 63 1 br 0 br 1 br 2 br 7 63 Instruction Pointer 63 Current Frame Marker 63 5 CFM User Mask ar 0 ar 7 ar 16 ar 17 ar 18 ar 19 ar 21 ar 24 ar 25 ar 26 ar 27 ar 28 ar 29 ar 30 ar 32 ar 36 ar 40 ar 44 ar 64 ar 65 ar KR0 KR7 RSC B BSTORE RNAT FCR EFLAG CSD SSD CFLG FSR FIR FDR CCV UNAT FPSR ITC PFS LC EC 0 gr 127 fr 127 ar

58 Thread Switching Overhead General Registers Floating-point Registers Predicates Branch Registers Application Registers All registers must be saved on context ar 0 KR gr 0 gr 1 gr fr switches 0 fr 1 fr 2 pr 0 pr 1 pr 2 More than 3.2 kb br 0 br 1 br 2 ar 7 KR7 RSC B BSTORE RNAT FCR EFLAG CSD SSD CFLG FSR FIR FDR CCV UNAT FPSR ITC PFS LC EC ar 16 ar 3.2 kb of register contents 17 ar 18 ar 19 br 7 ar 21 ar 24 ar 25 Instruction Pointer ar ar 27 ar 28 ar 29 pr 63 Current Frame Marker ar ar 32 CFM ar 36 ar 40 User Mask ar ar 64 ar 65 ar 66 Certain optimizations made possible by hardware gr 127 fr 127 ar

59 Thread Switching Overhead gr 0 gr 1 gr 2 General Registers fr 0 fr 1 fr 2 Floating-point Registers Static Registers gr 0 gr 31 Stacked Registers gr 32 gr 127 Predicates gr 0 fixed Branch Registers to zero Application Registers pr 0 1 br 0 ar 0 KR0 pr 1 On br 1 thread switch pr 2 br ar 2 7 KR7 ar 16 RSC ar 17 B ar 18 BSTORE ar 19 RNAT ar br 21 FCR 7 ar EFLAG Stacked registers 24 ar 25 CSD Instruction Pointer ar 26 SSD 63 0 ar 27 CFLG ar 28 FSR ar 29 FIR pr 63 Current Frame Marker ar 30 FDR 63 0 ar 32 CCV CFM ar 36 UNAT ar 40 FPSR Only User 2.5 Mask kb ar 44 ITC 5 0 ar 64 PFS contents left ar 65 LC EC 0 Static registers must be saved explicitly handled by register stack engine (RSE) 2.5 kb of register ar 66 0 gr 127 fr 127 ar

60 Thread Switching Overhead gr 0 gr 1 gr 2 General Registers fr 0 fr 1 fr 2 Floating-point Registers Predicates fr 0 and Branch Registers fr 1 fixedapplication Registers pr 0 1 br 0 ar 0 KR0 pr 1 Remaining br 1 floating- pr 2 br ar 2 7 KR7 ar 16 RSC ar 17 B ar 18 BSTORE handled lazily ar 19 RNAT ar br FCR Only 7 28 (of 128) 21 ar 24 EFLAG ARs ar 25 CSD implemented Instruction Pointer ar 26 SSD 63 0 ar 27 CFLG 28 FSR Only ~0.5 kbar 29 offir pr 63 Current Frame Marker ar 30 FDR register 63 contents 0 ar 32 left CCV CFM ar 36 UNAT ar 40 FPSR User Mask ar 44 ITC 5 0 ar 64 PFS ar 65 LC EC point registers can be ar 66 gr 127 fr 127 ar

61 Thread Switch Example [pistachio/kernel/include/glue/v4-ia64/tcb.h] asm volatile ( "r_bsp = r14 \n" "r_ip = r15 \n" "r_rp = r16 \n" "r_cfm = r17 \n" "r_pfs = r18 \n" "r_rnat = r19 \n" "r_unat = r20 \n" "r_pr = r21 \n" "r_psr = r22 \n" "new_stack = r31 \n" "sp1 = r10 \n" "sp2 = r11 \n" " \n" " // Move context into general registers \n" " mov r_pfs = ar.pfs \n" " mov r_rp = rp \n" " movl r_ip = 2f \n" " \n" " // Make stacked reg is not used \n" sure that " mov new_stack = %[dest_stack] \n" " \n" " // Allocate frame \n" for switch " add sp \n" = -%[sizeof_ctx],sp ;; " ;; \n" " mov ar.rsc = 0 \n" " add sp1 = %[offset_pfs], sp \n" " add sp2 = %[offset_pfs]+8,sp \n" " st8 [%[this_stack_ptr]] stack = sp \n" " \n" " // Set for new thread \n" thread ids and ksp " mov "MKSTR(r_GLOBAL_ID)" = %[dest_gid] \n" " mov "MKSTR(r_LOCAL_ID) " = %[dest_lid] \n" " mov "MKSTR(r_KERNEL_)" = %[dest_stack_top] \n" " mov "MKSTR(r_PHYS_TCB_ADDR)" = %[dest_tcb_phys] \n" " \n" " // Set region id \n" " mov rr[r0] = %[dest_rid] \n" " \n" " // Make a call so can get CFM \n" that we " br.call.sptk.many rp = 1f \n" " \n" " // Store context into switch frame \n" "1: alloc r_cfm = \n" ar.pfs, 0, 0, 0, 0 " ;; \n" " mov r_bsp = ar.bsp \n" " flushrs \n" " ;; \n" " \n" " st8 [sp1] = r_pfs, 16 \n" " st8 [sp2] = r_cfm, 16 \n" " mov r_unat = ar.unat \n" " mov r_rnat = ar.rnat \n" " mov r_pr = pr \n" " mov r_psr = psr \n" " ;; \n" " st8 [sp1] = r_ip, 16 \n" " st8 [sp2] = r_bsp, 16 \n" ;; \n " st8 [sp1] = r_rnat, 16 \n" " st8 [sp2] = r_unat, 16 \n" " ;; \n" " st8 [sp1] = r_pr \n" " st8 [sp2] = r_psr, 16 \n" " invala \n" " loadrs \n" " ;; \n" " st8 [sp2] = r_rp \n" " \n " // Get new switch frame \n" " add sp1 = %[offset_pfs], new_stack \n" " add sp2 = %[offset_pfs]+8,new_stack \n" " ;; \n" " \n" " // Load context from new frame \n" " ld8 r_pfs = \n" [sp1], 16 " ld8 r_cfm = [sp2], 16 \n" " ;; \n" " ld8 r_ip = [sp1], 16 \n" " ld8 r_bsp = [sp2], 16 \n" " mov ar.pfs = r_cfm \n" " ;; \n" " ld8 r_rnat = [sp1], 16 \n" " ld8 r_unat = [sp2], 16 \n" " mov ar.bspstore = r_bsp \n" " ;; \n" " ld8 r_pr = [sp1] \n" " ld8 r_psr = [sp2], 16 \n" " mov ar.rnat = r_rnat \n" " mov ar.unat = r_unat \n" " ;; \n" " ld8 r_rp = [sp2] \n" " mov rp = r_ip \n" " add sp = %[sizeof_ctx],new_stack \n" " mov pr = r_pr, 0x1ffff \n" " mov psr.l = r_psr \n" " ;; \n" " srlz.d \n" " ;; \n" " br.ret.sptk.many rp \n" " \n" "2: // Restore non-clobberable registers \n" " mov ar.pfs = r_pfs \n" " mov rp = r_rp \n" " mov ar.rsc = 3 \n" " ;; \n" : : [this_stack_ptr] "r" (&this->stack), [dest_stack] "r" (dest->stack), [sizeof_ctx] "i" (sizeof (ia64_switch_context_t)), [offset_pfs] "i" (offsetof (ia64_switch_context_t, pfs)), [dest_gid] "r" (dest->get_global_id ().get_raw ()), [dest_lid] "r" (dest->get_local_id ().get_raw ()), [dest_stack_top] "r" (dest->get_stack_top ()), [dest_tcb_phys] "r" (dest->arch.phys_addr), ((dest->space->get_region_id () << 8) + (12 << [dest_rid] "r" 2)) : CALLER_SAVED_REGS, CALLEE_SAVED_REGS, "memory"); 61

62 Exception Handling gr 0 gr 1 gr 2 General Registers fr 0 fr 1 fr 2 Floating-point Registers Banked Registers gr 16 gr 31 Predicates Bank Branch 1 Registers used normally Application Registers pr 0 1 br 0 ar 0 KR0 pr 1 Automatic br 1 switch to bank 0 pr 2 br ar 2 7 KR7 on exceptions ar 16 RSC ar 17 B ar 18 BSTORE Frees up registers ar 19 RNAT for ar br FCR storing 7 context 21 ar 24 EFLAG t ar 25 CSD Instruction Pointer ar 26 SSD Can switch manually 63 0 ar 27 CFLG ar 28 FSR ar 29 FIR pr 63 Current Frame Marker ar 30 FDR 63 0 ar 32 CCV CFM ar 36 UNAT ar 40 FPSR User Mask ar 44 ITC 5 0 ar 64 PFS ar 65 LC EC ar 66 gr 127 fr 127 ar

63 Exception Handling General Registers Backing Store Run on bank 1 gr 0 gr 1 gr Exception Switches to bank 0 Store other registers gr

64 Exception Handling gr 0 gr 1 gr 2 General Registers Backing Store Run on bank Exception Switches to bank 0 Store other registers Switch to bank 1 Store remaining registers Must not receive interrupts or raise exceptions while storing exception context gr

65 Kernel Entry Kernel entry by exception is slow Must flush instruction pipeline IA-64 provides an epc instruction Raises privileges to kernel mode Continues execution on next instruction Can only be executed in special regions of virtual memory 65

66 System Call Trampoline User call ipc ipc: epc call ipc return Kernel ipc: 66

67 System Call Trampoline for IA-32 Is it useful? Yes! Kernel dictates trampoline code Kernel can offer best kernel entry method (e.g., sysenter/sysexit if supported by CPU) Some system calls can execute entirely in user-mode (e.g., reading current time) Used in L4Ka::Pistachio Other systems are starting to use system call trampolines (e.g., Linux 2.6) 67

µ-kernel Construction

µ-kernel Construction Fundamental Abstractions Thread Address Space What is a thread? How to implement? What conclusions can we draw from our analysis with respect to µk construction? Fundamental Abstractions