Comprehensive Kernel Instrumentation via Dynamic Binary Translation

Transcription

1 Comprehensive Kernel Instrumentation via Dynamic Binary Translation Peter Feiner Angela Demke Brown Ashvin Goel University of Toronto 011

2 Complexity of Operating Systems 012

3 Complexity of Operating Systems Growth in code size Palix, ASPLOS 2011 Many new drivers! 012

4 Complexity of Operating Systems Growth in code size Palix, ASPLOS 2011 Many new drivers! More swearing Vidar Holen, 2012 Swear Count $%!^%* penguin

5 Complexity of Operating Systems Growth in code size Palix, ASPLOS 2011 Many new drivers! More swearing Vidar Holen, 2012 Bugs are inevitable! Swear Count Need coping strategies 2 01 $^&@!%&%! *&#@ $%!^%* penguin

6 Tools would be nice Awesome tools for user code Memcheck Program Shepherding Use Dynamic Binary Translation (DBT) Rewrite binaries as they execute No need for source s make building DBT tools easy DynamoRIO, Valgrind, Pin No framework for OS code 013

7 Our 014

8 Our Ported DynamRIO to Linux kernel Runs on bare metal 014

9 Our Ported DynamRIO to Linux kernel Runs on bare metal Port took 18 Months 014

10 Our Ported DynamRIO to Linux kernel Runs on bare metal Port took 18 Months Built OS debugging tools in 5 days Heap debugging Use after free Heap corruption Stack overflow monitor 014

11 Our Ported DynamRIO to Linux kernel Runs on bare metal Port took 18 Months Built OS debugging tools in 5 days Heap debugging Use after free Heap corruption Stack overflow monitor Practical One author ran system on her desktop for 1 month 014

12 What About Hypervisors? 015

13 What About Hypervisors? VMWare can use DBT on guests No instrumentation API 015

14 What About Hypervisors? VMWare can use DBT on guests No instrumentation API PinOS has instrumentation API PinOS = Pin + Xen Guest needs emulated devices Useless for most driver code 015

15 What About Hypervisors? VMWare can use DBT on guests No instrumentation API Drivers Sound 6% Net 6% FS 9% PinOS has instrumentation API PinOS = Pin + Xen Guest needs emulated devices Useless for most driver code 47% Other 6% Arch 25% Palix, ASPLOS

16 What About Hypervisors? VMWare can use DBT on guests No instrumentation API Drivers Sound 6% Net 6% FS 9% PinOS has instrumentation API PinOS = Pin + Xen Guest needs emulated devices Useless for most driver code 47% Other 6% Arch 25% Palix, ASPLOS 2011 So add DBT to a hypervisor with pass-through devices? Then you d have the problems we show you how to solve... problems with interrupts! 015

17 Overview 1. Boot normally User Code 2. Take over OS Code 3. JIT the OS x86 instrumented x86 Interrupts Exceptions 016

18 Overview 1. Boot normally User Code 2. Take over OS Code 3. JIT the OS x86 instrumented x86 Interrupts Exceptions 016

19 Overview 1. Boot normally User Code 2. Take over OS Code Instrumented OS Code 3. JIT the OS x86 instrumented x86 Interrupts Exceptions 016

20 Dynamic Instrumentation User Code OS Code Instrumented OS Code Interrupts Exceptions 017

21 Dynamic Instrumentation User Code Code Cache OS Code Interrupts Exceptions 017

22 OS Code Dynamic Instrumentation User Code Code Cache bb2 bb3 Interrupts Exceptions 017

23 OS Code Dynamic Instrumentation User Code Code Cache bb2 bb3 017

24 OS Code Dynamic Instrumentation User Code Code Cache bb2 bb3 System call example is entry point 017

25 OS Code Dynamic Instrumentation User Code Code Cache bb2 bb3 System call example is entry point 017

26 OS Code Dynamic Instrumentation User Code Code Cache bb2 bb3 System call example is entry point 017

27 OS Code Dynamic Instrumentation User Code Code Cache bb2 bb3 System call example is entry point 017

28 OS Code Dynamic Instrumentation User Code Code Cache bb2 bb3 System call example is entry point 017

29 OS Code Dynamic Instrumentation User Code Code Cache bb2 bb3 System call example is entry point 017

30 OS Code Dynamic Instrumentation User Code Code Cache bb2 bb3 System call example is entry point 017

31 OS Code Dynamic Instrumentation User Code Code Cache bb2 bb2 bb3 System call example is entry point 017

32 OS Code Dynamic Instrumentation User Code Code Cache bb2 bb2 bb3 System call example is entry point 017

33 OS Code Dynamic Instrumentation User Code Code Cache bb2 bb2 bb3 System call example is entry point 017

34 OS Code Dynamic Instrumentation User Code Code Cache bb2 bb2 bb3 System call example is entry point 017

35 OS Code Dynamic Instrumentation User Code Code Cache bb2 bb2 bb3 bb3 System call example is entry point 017

36 OS Code Dynamic Instrumentation User Code Code Cache bb2 bb2 bb3 bb3 System call example is entry point 017

37 OS Code Dynamic Instrumentation User Code Code Cache bb2 bb2 bb3 bb3 System call example is entry point 017

38 OS Code Dynamic Instrumentation User Code Code Cache bb2 bb2 bb3 bb3 System call example is entry point 017

39 OS Code Complications User Code Code Cache bb2 bb2 bb3 bb3 018

40 OS Code Complications User Code Code Cache bb2 bb2 bb3 bb3 Reentrance How do you do I/O? Can t use OS 018

41 OS Code Complications User Code Code Cache bb2 bb2 bb3 bb3 Reentrance How do you do I/O? Can t use OS 018 Concurrency Multiple CPUs using and building cache

42 OS Code Complications User Code Code Cache bb2 bb2 bb3 bb3 Reentrance How do you do I/O? Can t use OS Interrupts 018 Concurrency Multiple CPUs using and building cache

43 OS Code Handling Interrupts User Code Code Cache IH * iret Interrupts 019

44 OS Code Handling Interrupts User Code Code Cache IH * iret Interrupts 019

45 OS Code Handling Interrupts User Code arrival Code Cache IH * iret Interrupts 019

46 OS Code Handling Interrupts User Code arrival Code Cache IH * iret Interrupts 019

47 OS Code Handling Interrupts User Code arrival Code Cache IH * iret Interrupts What should framework do with interrupt? 019

48 OS Code Handling Interrupts User Code arrival Code Cache IH * iret Interrupts What should framework do with interrupt? Can it run interrupt handler IH immediately? 019

49 OS Code Handling Interrupts User Code arrival Code Cache IH * iret Interrupts IH What should framework do with interrupt? Can it run interrupt handler IH immediately? 019

50 OS Code Handling Interrupts User Code arrival Code Cache IH * iret Interrupts IH What should framework do with interrupt? Can it run interrupt handler IH immediately? 019

51 OS Code Handling Interrupts User Code arrival Code Cache IH * iret Interrupts IH What should framework do with interrupt? Can it run interrupt handler IH immediately? 019

52 OS Code Handling Interrupts User Code arrival Code Cache IH * iret Interrupts IH tail What should framework do with interrupt? Can it run interrupt handler IH immediately? 019

53 OS Code Handling Interrupts User Code arrival Code Cache IH * iret Interrupts IH tail What should framework do with interrupt? Can it run interrupt handler IH immediately? Problem if instrumentation isn t reentrant 019

54 OS Code Handling Interrupts User Code arrival Code Cache lock(l) IH * iret Interrupts IH tail What should framework do with interrupt? Can it run interrupt handler IH immediately? Problem if instrumentation isn t reentrant 019

55 OS Code Handling Interrupts User Code arrival Code Cache lock(l) IH * iret Interrupts lock(l) IH tail What should framework do with interrupt? Can it run interrupt handler IH immediately? Problem if instrumentation isn t reentrant 019

56 OS Code Handling Interrupts User Code arrival Code Cache lock(l) IH * iret lock(l) IH Interrupts What should framework do with interrupt? Can it run interrupt handler IH immediately? Problem if instrumentation isn t reentrant deadlock tail 019

57 OS Code Handling Interrupts User Code arrival Code Cache lock(l) IH * iret lock(l) IH What should framework do with interrupt? Can it run interrupt handler IH immediately? Problem if instrumentation isn t reentrant Need to delay Interrupts 019 deadlock tail

58 Delaying Interrupts arrival 01 10

59 Delaying Interrupts Where do we delay it until? arrival 01 10

60 Delaying Interrupts Where do we delay it until? Delay until end of? delivery arrival 01 10

61 Delaying Interrupts Where do we delay it until? Delay until end of? Avoids tail delivery arrival 01 10

62 Delaying Interrupts Where do we delay it until? Delay until end of? Avoids tail pending interrupt delivery arrival 01 10

63 Delaying Interrupts Where do we delay it until? Delay until end of? Avoids tail pending interrupt delivery arrival 01 10

64 Delaying Interrupts Where do we delay it until? Delay until end of? Avoids tail pending interrupt delivery arrival 01 10

65 Delaying Interrupts Where do we delay it until? Delay until end of? Avoids tail pending interrupt delivery arrival 01 10

66 Delaying Interrupts Where do we delay it until? Delay until end of? Avoids tail delivery arrival 01 10

67 Delaying Interrupts Where do we delay it until? Delay until end of? Avoids tail Problem if disables interrupt disabled delivery arrival 01 10

68 Delaying Interrupts Where do we delay it until? Delay until end of? Avoids tail Problem if disables interrupt Could be any MMIO cannot detect if enabled arrival disabled delivery 01 10

69 Delaying Interrupts Where do we delay it until? Delay until end of? Avoids tail Problem if disables interrupt Could be any MMIO cannot detect if enabled arrival disabled delivery 01 10

70 Delaying Interrupts Where do we delay it until? Delay until end of? Avoids tail Problem if disables interrupt Could be any MMIO cannot detect if enabled arrival disabled delivery delivery Must deliver before next OS instruction 01 10

71 Delaying Interrupts Where do we delay it until? Delay until end of? Avoids tail Problem if disables interrupt Could be any MMIO cannot detect if enabled arrival disabled delivery delivery Must deliver before next OS instruction Delay until end of instrumentation 01 10

72 Delaying Interrupts Where do we delay it until? Delay until end of? Avoids tail Problem if disables interrupt Could be any MMIO cannot detect if enabled arrival disabled delivery delivery Must deliver before next OS instruction Delay until end of instrumentation Still duplicates tail 01 10

73 How to Delay Interrupts 01 11

74 How to Delay Interrupts Could disable them on the CPU 01 11

75 How to Delay Interrupts Could disable them on the CPU push, disable 01 11

76 How to Delay Interrupts Could disable them on the CPU push, disable pop 01 11

77 How to Delay Interrupts Could disable them on the CPU push, disable pop push, disable 01 11

78 How to Delay Interrupts Could disable them on the CPU push, disable pop push, disable pop 01 11

79 How to Delay Interrupts Could disable them on the CPU push, disable pop push, disable But performance would be bad pop 01 11

80 How to Delay Interrupts Could disable them on the CPU push, disable pop push, disable But performance would be bad pop Instead, have framework handle it Extra overhead for interrupt, cheaper instrumentation 01 11

81 How to Delay Interrupts Could disable them on the CPU push, disable pop push, disable But performance would be bad pop Instead, have framework handle it Extra overhead for interrupt, cheaper instrumentation Instrumentation more frequent than interrupts Gigabit NIC sends interrupt every 100µs 100K instr

82 Delaying with Patches Example: interrupt 239 arrival 01 12

83 Delaying with Patches Example: interrupt 239 arrival Interrupt Stack Frame Interrupts enabled... yes 01 12

84 Delaying with Patches Example: interrupt 239 arrival Interrupt Stack Frame Interrupts enabled... yes 01 12

85 Delaying with Patches Example: interrupt 239 arrival The framework 1. Patches next native instruction int 239 Interrupt Stack Frame Interrupts enabled... yes 01 12

86 Delaying with Patches Example: interrupt 239 arrival The framework 1. Patches next native instruction 2. Disables interrupts on iret Interrupts enabled... int 239 Interrupt Stack Frame no yes 01 12

87 Delaying with Patches Example: interrupt 239 arrival The framework 1. Patches next native instruction 2. Disables interrupts on iret 3. iret Interrupts enabled... int 239 Interrupt Stack Frame no yes 01 12

88 Delaying with Patches Example: interrupt 239 arrival The framework 1. Patches next native instruction 2. Disables interrupts on iret 3. iret Interrupts enabled... int 239 Interrupt Stack Frame no yes 01 12

89 Delaying with Patches Example: interrupt 239 arrival The framework 1. Patches next native instruction 2. Disables interrupts on iret 3. iret Interrupts enabled... int 239 Interrupt Stack Frame no yes 01 12

90 Delaying with Patches Example: interrupt 239 arrival The framework 1. Patches next native instruction 2. Disables interrupts on iret 3. iret Interrupts enabled... int 239 Interrupt Stack Frame no yes Patch Interrupt Stack Frame Interrupts enabled... no

91 Delaying with Patches Example: interrupt 239 arrival The framework 1. Patches next native instruction 2. Disables interrupts on iret 3. iret Interrupts enabled... int 239 Interrupt Stack Frame no yes Patch Interrupt Stack Frame Interrupts enabled... no

92 Delaying with Patches Example: interrupt 239 arrival The framework 1. Patches next native instruction 2. Disables interrupts on iret 3. iret 4. Removes patch Interrupts enabled... int 239 Interrupt Stack Frame no yes Patch Interrupt Stack Frame Interrupts enabled... no

93 Delaying with Patches Example: interrupt 239 arrival The framework 1. Patches next native instruction 2. Disables interrupts on iret 3. iret 4. Removes patch 5. Enables interrupts on iret Interrupts enabled... int 239 Interrupt Stack Frame no yes Patch Interrupt Stack Frame Interrupts enabled... yes no

94 Delaying with Patches Example: interrupt 239 arrival The framework 1. Patches next native instruction 2. Disables interrupts on iret 3. iret 4. Removes patch 5. Enables interrupts on iret 6. Run instrumented interrupt handler Interrupts enabled... int 239 Interrupt Stack Frame Interrupts enabled... no yes Patch Interrupt Stack Frame yes no

95 Delaying with Patches Example: interrupt 239 arrival The framework 1. Patches next native instruction 2. Disables interrupts on iret 3. iret 4. Removes patch 5. Enables interrupts on iret 6. Run instrumented interrupt handler IH Interrupts enabled... int 239 Interrupt Stack Frame Interrupts enabled... no yes Patch Interrupt Stack Frame yes no

96 Delaying with Patches Example: interrupt 239 arrival The framework 1. Patches next native instruction 2. Disables interrupts on iret 3. iret 4. Removes patch 5. Enables interrupts on iret 6. Run instrumented interrupt handler Okay, what s the performance? IH Interrupts enabled... int 239 Interrupt Stack Frame Interrupts enabled... no yes Patch Interrupt Stack Frame yes no

97 Performance Ran framework with instruction counting tool Intel Quad Core i7 2.8Ghz, 8GB, 64-bit Ubuntu Low application overhead JavaScript, Mozilla Kraken: 3% overhead 18% user time increase 143% system time increase Parallel Linux kernel compile: 30% overhead Overhead commensurate with OS activity How bad can this get? 01 13

98 Stress Test Setup Apachebench and Filebench Configured benchmarks to stress CPUs and kernel Large buffer cache - no disk I/O Many threads - lots of context switching 100% utilization - shows interrupt processing overhead nthreads data size fileserver GB webserver MB webproxy MB varmail MB Table 1. Filebench parameters concurrency level 200 Apachebench Parameters 01 14

99 Stress Test Results Apachebench Filebench Native Instrumented Throughput (ops/s) apachebench fileserver webserver webproxy varmail Less than 5x - Reasonable overhead for debugging tools 01 15

100 Stress Test Results Apachebench Filebench Native Instrumented Throughput (ops/s) apachebench fileserver webserver webproxy varmail CPU-bound IO-bound Less than 5x - Reasonable overhead for debugging tools 01 15

101 Summary Enables dynamic binary instrumentation of OS Makes it easy to write complex instrumentation Built useful memory checking tools Works with arbitrary devices & drivers 01 16

102 Questions? 01 17