Enhancing Memory Error Detection for Large-Scale Applications and Fuzz testing

Transcription

1 Enhancing Memory Error Detection for Large-Scale Applications and Fuzz testing Wookhyun Han, Byunggil Joe, Byoungyoung Lee *, Chengyu Song, Insik Shin KAIST, * Purdue, UCR 1

2 Memory error Heartbleed Shellshock glibc: getaddrinfo stack-based buffer overflow Information leakage Heartbleed Privilege escalation Shellshock Remote code execution Shellshock, glibc, Conficker 2

3 Memory error detection Pointer-based [SoftBound+CETS, Intel MPX] Hardware support (cannot detect temporal memory errors) Challenges to support complex applications Redzone-based [AddressSanitizer (ASan)] Compatible to complex applications Most popular in practice Google Chrome, Mozilla Firefox, Linux Kernel American Fuzzy Lop (AFL), ClusterFuzz, OSS-Fuzz 3

4 Redzone-based memory error detection Buffer overflow (spatial memory errors) ptrx objx Check before access Shadow memory: a bitmap to validate all addresses Shadow memory Accessible 4

5 Redzone-based memory error detection Buffer overflow (spatial memory errors) ptrx objx Shadow memory Check before access Accessible Inaccessible (redzone) Shadow memory: a bitmap to validate all addresses Redzone: inaccessible region between objects 4

6 Redzone-based memory error detection Buffer overflow (spatial memory errors) ptrx objx Shadow memory: a bitmap to validate all addresses Shadow memory Error! Accessible Inaccessible (redzone) Redzone: inaccessible region between objects 4

7 Redzone-based memory error detection Use-after-free (temporal memory errors) ptrx objx Accessible Inaccessible Shadow memory 5

8 Redzone-based memory error detection Use-after-free (temporal memory errors) ptrx ptrx objx free(ptrx) Quarantined Region is invalidated and quarantined, but not actually deallocated Accessible Inaccessible Shadow memory 5

9 Redzone-based memory error detection Use-after-free (temporal memory errors) ptrx ptrx objx free(ptrx) Quarantined Hold the region until quarantine zone is full (FIFO) Accessible Inaccessible Shadow memory 5

10 Redzone-based memory error detection Use-after-free (temporal memory errors) ptrx ptrx ptry objx free(ptrx) Quarantined The region is actually deallocated, and can be allocated to a new object ptry = malloc() objy Accessible Inaccessible Shadow memory 5

11 Limitations of redzone-based approach 1. What if a pointer accesses beyond redzone? ptrx objx objy 6

12 Limitations of redzone-based approach 1. What if a pointer accesses beyond redzone? objx ptrx objy Spatial memory error 6

13 Limitations of redzone-based approach 1. What if a pointer accesses beyond redzone? 2. What if a dangling pointer accesses after another object is allocated in the region? objx ptrx ptrx objy objx Spatial memory error 6

14 Limitations of redzone-based approach 1. What if a pointer accesses beyond redzone? 2. What if a dangling pointer accesses after another object is allocated in the region? objx ptrx ptrx ptrx objy objx objz Spatial memory error Temporal memory error 6

15 Limitations of redzone-based approach 1. What if a pointer accesses beyond redzone? 2. What if a dangling pointer accesses after another object is allocated in the region? objx ptrx ptrx ptrx objy objx Cannot detect! objz Spatial memory error Temporal memory error 6

16 Motivation To enhance detectability of redzonebased memory error detection P1. Large gap to detect spatial memory errors P2. Large quarantine zone to detect temporal memory errors P1 obj1 P1 obj1 P1 7

17 Motivation To enhance detectability of redzonebased memory error detection P1. Large gap to detect spatial memory errors P2. Large quarantine zone to detect temporal memory errors P2 P2 P1 obj1 P1 obj2 obj1 P1 7

18 Motivation To enhance detectability of redzonebased memory error detection P1. Large gap to detect spatial memory errors P2. Large quarantine zone to detect temporal memory errors Huge physical memory required P2 P2 P1 obj1 P1 obj2 obj1 P1 7

19 MEDS overview Enhances detectability of redzone-based memory error detection Idea: Fully utilize 64-bit virtual address space to support P1. Large gap to detect spatial error P2. Large quarantine zone to detect temporal error Approach: minimize physical memory use Page aliasing allocator and page protection Hierarchical memory error detection 8

20 Page aliasing (P1) Maps multiple virtual pages to single physical page Virtual obj1 obj2 A memory page Allocated Redzone Page aliasing obj4 9

21 Page aliasing (P1) Maps multiple virtual pages to single physical page Virtual obj1 obj2 Physical obj1 obj2 obj3 obj4 A memory page Allocated Redzone Page aliasing obj4 9

22 Page aliasing (P1) Maps multiple virtual pages to single physical page Virtual obj1 obj2 Physical obj1 obj2 obj3 obj4 Redzone itself does not occupy physical memory A memory page Allocated Redzone Page aliasing obj4 9

23 Page protection (P1) Redzone only pages are unmapped Virtual obj1 obj2 Physical obj1 obj2 obj3 obj4 A memory page Unmapped page Allocated Redzone Page aliasing 10

24 Page protection (P1) Redzone only pages are unmapped Virtual obj1 Physical Do not occupy shadow memory and physical memory obj2 obj1 obj2 obj3 obj4 A memory page Unmapped page Allocated Redzone Page aliasing 10

25 Page aliasing & Page protection (P2) Virtual obj1 obj4 Physical obj1 obj2 obj3 obj4 A memory page Unmapped page Allocated Redzone Page aliasing 11

26 Page aliasing & Page protection (P2) Virtual obj1 Virtual Quarantined Physical Physical obj4 obj1 obj2 obj3 obj4 obj4 obj2 obj3 obj4 A memory page Unmapped page Allocated Redzone Page aliasing 11

27 Page aliasing & Page protection (P2) Virtual obj1 Virtual Quarantined Physical Physical obj4 obj1 obj2 obj3 obj4 obj4 objx obj2 obj3 obj4 A memory page Unmapped page objx Allocated Redzone Page aliasing 11

28 Page aliasing & Page protection (P2) Virtual obj1 obj4 Physical obj1 obj2 obj3 obj4 Virtual Quarantined obj4 Physical objx obj2 obj3 obj4 Reuse physical memory immediately, while not reusing virtual addresses A memory page Unmapped page objx Allocated Redzone Page aliasing 11

29 Hierarchical memory error detection Many different ways to represent redzones Further optimizing physical memory uses ptr 12

30 Hierarchical memory error detection Many different ways to represent redzones Further optimizing physical memory uses ptr #1. Shadow memory is invalid 12

31 Hierarchical memory error detection Many different ways to represent redzones Further optimizing physical memory uses ptr #1. Shadow memory is invalid #2. Virtual page is unmapped 12

32 Hierarchical memory error detection Many different ways to represent redzones Further optimizing physical memory uses ptr #1. Shadow memory is invalid #2. Virtual page is unmapped #3. Shadow memory is unmapped 12

33 Evaluation Configuration ASan MEDS Improv. Redzone bytes 4MB 16,384x Quarantine 128MB 80TB 65,536x ASan cannot use configuration for MEDS (lack of memory) Compatibility Performance: 2 times slowdown Detection (fuzz testing): 68% more detection 13

34 Compatibility Unit tests from real-world applications Test cases in Chrome, Firefox, Nginx All Passed Memory error unit tests ASan unit tests All Passed NIST Juliet test suites All Passed except random access tests ASan: 35% vs. MEDS: 98% 14

35 Micro-scale performance overhead TLB misses 5 times more than ASan (more virtual pages with page aliasing) Number of system calls mmap(), munmap(), and mremap() 32 times more than ASan (page aliasing and page protection) Memory footprint 218% more than baseline 68% more than ASan (much larger redzone and quarantine) 15

36 End-to-end performance overhead 108% compared to baseline, 86% to ASan Baseline 0 Chrome Firefox Apache Nginx ASan MEDS 16

37 End-to-end performance overhead 108% compared to baseline, 86% to ASan % to baseline 22% to ASan 1 Baseline 0 Chrome Firefox Apache Nginx ASan MEDS 16

38 End-to-end performance overhead 108% compared to baseline, 86% to ASan % to baseline 22% to ASan Large number of small objects on stack 243% to baseline 211% to ASan 1 Baseline 0 Chrome Firefox Apache Nginx ASan MEDS 16

39 Detection (fuzz testing) Run AFL (8 cores, 6 hours) Despite the performance overhead, explore 68.3% more unique crashes than ASan ASan 17

40 Detection (fuzz testing) Run AFL (8 cores, 6 hours) Despite the performance overhead, explore 68.3% more unique crashes than ASan MEDS finds more unique crashes in initial phase, but saturated in the end ASan 17

41 Detection (fuzz testing) Number of unique crashes with time spent (metacam) Found crashes Saturated Time spent (hrs) ASan MEDS 18

42 How MEDS explores more crashes? More input sets can be detected Higher probability to detect Bugs can be found earlier than ASan Fuzzer can focus on the other paths MEDS can detect the cases that ASan cannot detect Always bypass redzone e.g., Miscalculation of structure array size Size of the structure is larger than redzone size Access to certain element cannot be detected. int a[10]; a[x] = x; struct A { int num[10]; }; struct A *a = malloc(sizeof(struct A));... (a+i)->num[8] = i; 19

43 How MEDS explores more crashes? More input sets can be detected Higher probability to detect Bugs can be found earlier than ASan Fuzzer can focus on the other paths MEDS can detect the cases that ASan cannot detect Always bypass redzone e.g., Miscalculation of structure array size Size of the structure is larger than redzone size Access to certain element cannot be detected. int a[10]; a[x] = x; struct A { int num[10]; }; struct A *a = malloc(sizeof(struct A));... (a+i)->num[8] = i; 19

44 How MEDS explores more crashes? More input sets can be detected Higher probability to detect Bugs can be found earlier than ASan Fuzzer can focus on the other paths MEDS can detect the cases that ASan cannot detect Always bypass redzone e.g., Miscalculation of structure array size Size of the structure is larger than redzone size Access to certain element cannot be detected. int a[10]; a[x] = x; struct A { int num[10]; }; struct A *a = malloc(sizeof(struct A));... (a+i)->num[8] = i; 19

45 How MEDS explores more crashes? More input sets can be detected Higher probability to detect Bugs can be found earlier than ASan Fuzzer can focus on the other paths MEDS can detect the cases that ASan cannot detect Always bypass redzone e.g., Miscalculation of structure array size Size of the structure is larger than redzone size Access to certain element cannot be detected. int a[10]; a[x] = x; struct A { int num[10]; }; struct A *a = malloc(sizeof(struct A));... (a+i)->num[8] = i; 19

46 Conclusion Idea Support large gap and large quarantine zone Approach Page aliasing and page protection Hierarchical memory error detection Despite overhead (108%), MEDS finds more crashes during fuzz testing (68.3%) Open source will be available soon Please use to detect bugs 20

47 Thank you for listening! 21