Debugging distributed systems. CS514: Intermediate Course in Operating Systems. Today s topic. Waterfall Model. V Model.

Transcription

1 CS514: Intermediate Course in Operating Systems Professor Ken Birman Vivek Vishnumurthy: TA Debugging distributed systems The proliferation of component reuse has made debugging really hard! Many applications are just glue. The underlying components come from elsewhere Multiple languages, multiple paradigms Issues arise at every stage: development, execution, upgrades to components, performance problems Today s topic Waterfall Model Review some examples of software engineering approaches and tools Discuss their capabilities and limits Then shift attention to technologies for debugging, monitoring, performance evaluation Finally offer some thoughts about how models (like virtual synchrony or Paxos) can simplify the task ANALYSIS PROGRAM CODING UNIT & INTE- GRATION TESTING TESTING ACCEPTANCE TESTING OPERATION & MAINTENANCE V Model Prototyping Model ANALYSIS Validate requirements OPERATION & MAINTENANCE Verify design ACCEPTANCE TESTING TESTING Listen to Customer Build/Revise Mock-Up PROGRAM UNIT & INTE- GRATION TESTING Customer Test-drives Mock-up CODING [Pressman 97] 1

2 Prototyping Model Operational Specification Model LIST OF REVISIONS LIST OF REVISIONS LIST OF REVISIONS Execute and Revise revise prototype user/ customer review PROTOTYPE PROTOTYPE PROTOTYPE TEST OPERATIONAL SPECIFICATION (problem-oriented) TRANSFORMED SPECIFICATION (implementationoriented) TEST (sometimes informal or incomplete) DELIVERED (sometimes informal or incomplete) DELIVERED Phased Development Model Boehm s Spiral Model DEVELOPERS USERS Build Release 1 Development systems Build Release 2 Build Release 3 Use Release 1 Use Release 2 Production systems Time Use Release 3 DETERMINE GOALS, ALTERNATIVES, CONSTRAINTS Budget 4 PLAN Alternatives 4 Budget 3 Integration and test plan Alternatives 3 Constraints 4 Alternatives 2 Constraints 3 Constraints 2 Constraints 1 Alternatives 1 Budget 2 Budget 1 start Requirements, life-cycle plan Development plan Implementation plan Risk analysis 3 Risk analysis 2 Risk analysis 1 Risk analysis 4 EVALUATE ALTERNATIVES ANDRISKS Proto - Proto - Proto - Prototype 1 type 2 type 3 type 4 Concept of operation Validated requirements Acceptance test Software requirements Validated, verified design System test Software design Unit test Code Detailed design DEVELOP AND TEST Focus on debugging Clearly, this is just one part of the picture A professional team really needs to cover the whole picture And all members need to be comfortable with entire process! Debugging 101 Start by discussing issues and options in a single non-distributed program Two broad cases Programs written in unsafe languages, like C or C++ Programs written in safe languages like Java and C# The former are probably more common 2

3 First, know what correct means Many programs lack clear specifications. Start by writing down what it does in the simplest, highest level form you can Not how it does it, but just what inputs it has, what it computes, where it puts results Next, separately, document how it does it The high-level algorithm. If relevant, could include a correctness argument Now you code it Ideally, use a type-checked language Java, C# prevent many kinds of errors But can get some of the same guarantees using type-checking mode in compilers for languages like C, C++ and Fortran Many argue that object-oriented design and coding is inherently less bug prone I m not really convinced of this Concurrency Concurrency is a major source of bugs Take a class like CS414, or read a book Try and code in what we call monitor style. This is similar to [synchronized] class in C# or Java Forces a rather stylized handling of variables shared between threads and associated synchronization easier to reason about Completion: Testing (Q/A) Requires skill and hard work to design a comprehensive test suite Many people start by testing object classes in isolation (not in the context of their program) Idea is to build a separate test harness that exercises your object class in every possible way This also facilitates performance tuning Completion: Testing (Q/A) Must obtain maximum level of code coverage Try to ensure that your test harness will exercise every code path There are coverage tools available to help; they tell you whether there are untested paths in the application Even with total coverage bugs often remain, but less of them Completion: Testing (Q/A) Now test your program as an integrated whole, with same approach Goal is to expose it to the widest possible range of realistic scenarios that you can Keep in mind that the logic combining your correct components could have flaws. So the sum of a set of correct objects might be a very incorrect program! 3

4 Use of debuggers Reasonable development platforms have high-quality debuggers At a minimum: Means of setting and controlling breakpoints Looking at objects, tracing changes to them Ways to see the set of active threads Some go even further and offer ways to write chunks of debug code that will run when a breakpoint is reached, e.g. to check a data structure for correctness Performance tuning Normally, done after the program is initially debugged Must Compile with optimization Then begin to measure key properties, perhaps by doing something lots of times Example: if putting a node in a tree is too fast to really instrument, put 10,000 nodes in, and measure the costs of doing that Performance tuning Tools available to us The system clock is quite useful, but often has 1/60 th second resolution Fine-grained execution profilers help a great deal Visual Studio: profiling support planned soon Today, can obtain the ANTS profiler On Linux, use gprof tool for C, C++ Now we finally have a solid single-machine solution Begin to repeat the process in a distributed manner Again, start by documenting intended behavior So: Specification, algorithm, correctness argument Test harness can be a challenge but remains worthwhile Often obtained by hacking your actual program, e.g. to issue a legitimate request 10,000 times Important that a scenario be repeatable, so that after fixing a bug you can test the repair! An issue: Distributed debuggers There aren t any! Some debuggers do set breakpoints across component boundaries But this isn t often what developers need It s hard to debug distributed code!!! A distributed data mining task. Astrolabe was built for this Distributed logging Suppose an application After a test run the has several components Database One option is to developer that support run remotely can collect the a debug mode logs, in which merge them, and remote servers (hopefully) keep logs identify issues Web Service CORBA Application 4

5 Distributed logging: Merge Put timestamps, component name on each line in each log [10:22: ] [DB] Received WS UPDATE req Then combine files, run sort. Ideally, you see events as they occurred across the system But recall: clocks can t be perfectly synchronized So do some sanity checking look at a few obvious entries to make sure that requests arrive after they are sent, that replies arrive after request was processed, etc Might need to build a merge sort program of your own that can adjust timestamps if needed Insight from Group Comm. Why did we discuss virtual synchrony, transactions, Paxos? Because these are cool tools But also because these tools simplify the task of the application developer If a programming environment automates handling of tricky events, applications are far more likely to be robust Simplifies in what sense? Transactions: ACID properties We code as if transaction runs against an idle database and doesn t fail System cleans up and also hides concurrency so these coding assumptions seem to be valid at runtime In contrast, for an older generation of systems, runtime was ruintime! Virtual synchrony Similar argument Programmers have trouble implementing replication, tracking membership So platform solves this once and offers the user a simplified model That developer is far less likely to make mistakes because he/she has far fewer cases and events to worry about! But virtual synchrony was complex! Yes and no Implementing virtual synchrony is tricky But ideally, it lives in a pre-built library Using virtual synchrony gives simplified code Often performance is not an issue and we just use abcast for everything Fancy issues and primitives are for hackers who want to put the pedal to the metal But correctness isn t the whole story Is our system correct? But is it also secure? Can performance be badly disrupted by unexpected events What about rare events that become common as a deployment scales up? Few developers think much beyond the initial moment of getting things working 5

6 Performance evaluation is hard Examples of things commonly measured: Requests per second that can be handled From a single source and from many Delay when a request is done Within the server From the client s perspective, with network in the middle Load on the network, and breakdown of messages between workload and overhead Load on individual services Scalability Here we typically look at the impact of Increased numbers of clients Increased numbers of servers Making the network itself larger, while holding other things constant Impacts latencies, frequency of network disruptions visible to programs But how can we get lots of machines? Value of testbeds One option is to build a simulation This can be invaluable but it isn t reality Best simulators run actual code in a simulated environment. Like JIST/SWANS Much better: run real code in a real environment For this we need testbeds Examples of testbeds Many groups have access to some form of cluster At Cornell, Ken s group has a 250 node rack-style cluster We can use it directly to emulate a data center Or we can run Emulab on it (Utah) Emulab and Modelnet These creates an instrumented runtime environment (mostly Linux) Special device drivers can Slow down messages Discard messages Play with message spacing/dups/ordering Permits us to emulate a wide area network configuration! How they differ Emulab routes packets via a special node that injects the desired dynamics Also, they hope to support Windows soon Modelnet creates a kind of virtual PC and is more focused on running LOTS of images on each physical node Seems to be exclusively a Linux technology 6

7 Emulab example Suppose that at noon on Wednesday, communication from Cornell to Berkeley had delays of 53ms +/- 10ms, bandwidth of 180MB/sec, etc Then if we label some emulab nodes as Cornell and some as Berkeley It can intercept packets en route Throw dice to compute the actual delay (or loss) for this particular packet Make it sit in the driver for that long Then deliver it The nodes run in the Cornell system lab as if half were on the west coast and half in Ithaca! Planetlab This is a real wide-area network setup Various participants contribute a few machines each machines offer virtual private partitions Users run experiments distributed over hundreds of real nodes on the real Internet But share the system, interference a worry Injected events With sophisticated applications we need to understand more than the best case behavior! Often the real issue is what happens during failures and recoveries? Studies will often set up a steady-state load and then inject events at known times (e.g. at time t=30, t=60.) Then look at impact of load on overheads, throughputs, etc Other words of wisdom Remember that the peak stress isn t always caused by the peak load Sometimes problems only occur when a system is idle and timeouts can go off So don t make the mistake of only testing under extremely high loads must consider a wide range of operating conditions and scenarios Red-teams For applications designed to be robust, designer shouldn t do Q/A alone Hence work with red teams They study application using public info only Then attack it to break in crash it degrade it. subvert it You use this as feedback for improvement Metrics: The holy grail Your boss will want metrics What exactly is our measure for success? How can we evaluate how fault-tolerant the system is, or how secure? But this is tough Easy to talk about TPC transactions/second Hard to say A is more fault-tolerant than B or A is more secured 7

8 Presenting the story At the end of the day you ll have a tremendous pile of data Although instrumentation and collection of data is (yet another) black art! But now face yet another challenge: how to show people what you ve got This takes a great deal of thought A fantastic book Edward Tufte: The Visual Display of Quantitative Information ( Urges that we escape flatlands by introducing dimensionality to graphs and illustrations Key idea is to create intuitive but rich presentations that tell us a great deal in a single graph, and yet that make sense without a lot of words to explain what we are seeing This is an art and separates the very best systems people from the pack Summary? Building a program is just one step in a process And even those working alone need to think about that entire process You ll never debug a program unless you decide how it should work first! Then can stress test against your theory of how it works and why it works Payoff is that code will work better even under stress 8