Causes of Software Failures

Transcription

1 Causes of Software Failures Hardware Faults Permanent faults, e.g., wear-and-tear component Transient faults, e.g., bit flips due to radiation Software Faults (Bugs) (40% failures) Nondeterministic bugs, e.g., data races, atomic violations, due to timing-dependent issues such as process scheduling and signals Deterministic bugs, e.g., buffer overflow, memory leaks

2 An Example of Non-deterministic Bug /* From Apache httpd mod_log_config.c */ ap_buffered_log_writter() {.. s = &buffer[buf->outcnt]; memcpy(s, str, len); temp = buf->outcnt + len; buf->outcnt = temp; } /4/12

3 Failure Recovery Methods Reboot-Based Methods General Checkpointing and Rollback Methods Studied already Application-Specific Methods Redundancy-Based Methods Failure-Oblivious Computing Randomization-Based Methods Execution Environment Based Methods

4 Rebooting Simply reset the computer or restart the failed program Pros: A simple (and common) way to recover failures on a single system or program Cons: Losing work that have not been saved (Auto-save alleviates this problem) May cause inconsistent state Rebooting a large complex systems (e.g., data center) is time consuming

5 Software Rejuvenation Periodically restart a system when load is low Commonly used for maintaining servers Likely not cause big disruption to service E.g., cannot log on bank web sites on Saturday night Expect some failures would happen in near future Based on observed MTTF (Mean-Time-To-Failure) E.g., memory leaks, resource leaks

6 Micro-rebooting[1]* Full system rebooting is costly for large-scale Internet systems[1] Unexpected recovery time Unexpected data loss Micro-rebooting: individually rebooting of fine-grained application components. [*] Microreboot. A technique for cheap recovery. G. Candea, S. Kawamoto, Y. Fujiki, G. Friedman, and A. Fox. In Proceedings of the 6th Symposium on Operating System Design and Implementation, Dec 2004.

7 Requirements for Microrebootable Applications (I) Fine-grained components Enabling fast recovery State segregation prevent corruption in application persistent state Using mechanisms such as transactional databases and session state managers

8 Requirements for Microrebootable Applications (II) Decoupling: Components must be loosely coupled. Retryable requests Smooth reintegration of microrebooted components.

9 Trade-Offs of Micro-rebooting Pros: Low runtime overhead with normal execution Fast recovery No loss data Cons: Reconstruction of software with several requirements

10 Application-Specific Methods Exception Handling Java Programs Process/Thread-Pool Based Methods Apache Application-Specific Checkpointing HPC applications

11 Trade-offs of Previous Approaches Nicely deal with hardware transient errors and non-deterministic bugs Cannot handle deterministic bugs (except for some application-specific methods) We will discuss approaches that can deal with deterministic bugs on failure recovery

12 Redundancy-Based Methods Multiple instances of the program, modules, functions are used for failure recovery Examples: N-version Programming Recovery blocks

13 N-version Programming* Different teams to generate independent N>=2 software modules Detecting/Masking Failures using results from N versions of modules. [*] The N-version approach to fault-tolerant software. Algirdas Avizienis. IEEE Transactions on Software Engineering, 11(12), 1985.

14 Recovery Blocks* Program is structured into blocks (e.g., modules, functions) A recovery block consists of a conventional block which is provided with a means of error detection and stand-by spares with the same functionality. Once an error is detected, a spare block will be executed [*] System structure for software fault tolerance. Brian Randell. IEEE Transactions on Software Engineering, 1(2): , 1975

15 Pros: Trade-offs of Redundancy-Base Methods Can handle both deterministic and nondeterministic software bugs Cons: Expensive Implicit assumption is different team may have different bugs for implementing the same functionalities

16 Other Approaches Failure-Oblivious Computing Randomization-based Method Execution Environment based Methods

17 Failure-Oblivious Computing (FOC) Memory errors are common sources of program failures E.g., out of bound array accesses, invalid pointer accesses FOC enables servers to execute through memory errors without memory corruption [*] Enhancing Server Availability and Security Through Failure-Oblivious Computing. Martin Rinard et. al. OSDI 04

18 How to be Oblivious to Failures? The C Compiler inserts checks for dynamically detecting invalid memory accesses What to do when an invalid memory access is detected? Not terminating or throwing an exception Discarding invalid writes Manufacturing values to return for invalid reads Hopefully the server will continue its normal path

19 A Simple Example char buf[80]; for (int i = 0; i <= 80; I ++) { a = buf[i]; a ++; buf[i] = a; } i = 80, overflowed read i = 80, overflowed write

20 Rationales of FOC Memory errors may occur in irrelevant computations => FOC enables the server to continue the relevant computation Even when memory errors occur in relevant computations => FOC converts unanticipated requests into anticipated invalid inputs that the error-handling logic can handle

21 C Safety Checking A Checker developed by Jones and Kelly, enhanced by Ruwase and Lam Maintaining a table that maps locations and length info to data units (e.g., struct, array, and variable) For each memory access, use this table to distinguish in-bounds and out-of-bounds pointers

22 Randomization-Based Method Randomizing memory allocation with multiple replicas or running multiple times for surviving memory errors Handled errors: Double free, Invalid free, Uninitialized read, Dangling pointer, Buffer overflow [*] DieHard: Probabilistic Memory Safety for Unsafe Programming Languages. Emery Berger and Ben Zorn. PLDI 06

23 DieHard Architecture Replica 1 input Replica 2 output Replica 3 Each replica randomize the heap differently Majority voting at the output point Assumption: different replica fail differently (likely valid in this randomization case)

24 DieHard Randomize Heap Bitmap-based, segregated size classes: One bit for the status (free or allocated) of one object Objects with the same size are grouped (2^3, 2^4, ) Malloc: Get the closet size class Randomly assign one available space for the object Free: Check the object properly aligned, bit status, and reset bits Prevent invalid frees, double frees

25 An Example From the Paper

26 Rx Execution Environment Based Methods Change program execution environment Survive software failures caused by both nondeterministic and deterministic bugs [*] Rx: Treating Bugs as Allergies A Safe Method to Survive Software Failures. Feng Qin et. al. SOSP 05

27 Rx Idea: Treat Bugs As Allergies Allergies: Hard to cure Can be avoided by changing the living environments! Avoidance Therapy for Allergies 2018/4/12 27

28 Rx Avoids Software Bugs Avoidance Therapy: Introduce non-determinism during program recovery by changing the execution environments Goals: Deterministic bugs Nondeterministic bugs Non-deterministic bugs Lower probability to occur Software execution Execution environment /4/12

29 Examples of Environmental Changes Category Potentially- Avoided Bugs Environmental Changes Cost Double free, dangling pointer Delaying recycling of freed buffer Relatively Cheap Memory Management Dynamic buffer overflow Memory corruption Padding allocated memory blocks Allocating memory in an isolated location Relatively Cheap Expensive Un-initialized reads Zero-filling newly allocated memory buffers Expensive Timing- Related Concurrency Bugs Scheduling Signal delivery Cheap Cheap Message Reordering Cheap User- Related Bugs related to user requests Selectively Dropping user requests Really Expensive /4/12

30 The Rx Process Change execution environment on-demand upon software failures Fail App Software Failure App Rollback App Change Env App Re-execute App Succeed App checkpoint Env Env Env Env Env Env Log Time Out programmers Other Approaches 2018/4/12 30

31 Rx Architecture 1. Hide server failures from clients 2. Help server to recover Server Application Detect software failures or faults in servers Proxy Clients Sensors Environment Wrappers Implement those Checkpoint the program environmental changes periodically and rollback it whenever failure happens Checkpoint 1. Devise a Rx recovery System scheme & Rollback 2. Log diagnostic information Control Unit report errors programmers /4/12