How do we troubleshoot this? How does Esmeralda know how to fix this?

Transcription

1

2 How do we troubleshoot this? How does Esmeralda know how to fix this? 2

3 Goal Find bugs in networked applications Large complex unknown applications!!! Large complex unknown networks!!! Understandable output / fix 3

4 Motivation pache Server Chrome Client 4

5 Motivation Chrome Client probing ping Different traffic (ICMP) Often different result pache Server 6

6 Motivation Chrome Client Requires detailed protocol / app knowledge packet capture pache Server 9

7 Motivation Need a model per application pache Server Chrome Client Model Model apps Magpie, Xtrace, Pip... Model 12

8 Motivation Chrome Client Header Space nalysis, etc. Network Config nalysis pache Server Model & Config Model & Config Model & Config Model & Config 14

9 Motivation Chrome Client Need detailed network knowledge HW + config Network Config nalysis pache Server Model & Config Model & Config Model & Config Model & Config 15

10 Motivation pache Server Chrome Client? 16

11 NetCheck Chrome Client programmer programmer pache Server 17

12 NetCheck Chrome Client programmer programmer pache Server 18

13 NetCheck Chrome Client programmer programmer pache Server Model Programmer s Understanding Deutsch s Fallacies 19

14 Outline Motivation NetCheck Overview Trace Ordering Network Model Fault Classification Results / Conclusion 20

15 NetCheck overview pplication Fail Traces NetCheck Likely Faults 21

16 NetCheck overview pplication Fail Traces ktrace strace NetCheck Likely Faults 22

17 NetCheck overview pplication Traces Host Traces Ordering lgorithm Diagnoses Engine Diagnosis NetCheck Input syscall simulation result Network Model simulation state errors Output NetCheck Likely Faults 23

18 NetCheck overview pplication Traces Network Configuration Issues NetCheck Likely Faults Traffic Statistics Problem Detected 24

19 Outline Motivation NetCheck Overview Trace Ordering Network Model Fault Classification Results / Conclusion Traces (a) Trace Ordering 25

20 Traces Series of locally ordered system calls Don t want to modify apps or use a global clock Gathered by strace, ktrace, systrace, truss, etc. Call arguments and return values! socket() = 3 bind(3, ) Call arguments = 0 listen(3, 1) = 0 accept(3, ) = 4 recv(4, "HTTP", ) = 4 Return values close(4) = 0 Return buffer 26

21 What we see is this:! Node Node B 1. socket() = 3 1. socket() = 3 2. bind(3,...) = 0 2. connect(3,...) = 0 3. listen(3, 1) = 0 3. send(3, "Hello",.) = 5 4. accept(3,...) = 4 4. close(3) = 0 5. recv(4,"hello",..) = 5 6. close(4) = 0 - one trace per host - local order but no global order Q: how do we reconstruct what really happened? 27

22 What we want is this 1. socket() = 3 B1. socket() = 3 2. bind(3,...) = 0 3. listen(3, 1) = 0 B2. connect(3,...) = 0 4. accept(3,...) = 4 B3. send(3, "Hello",...) = 5 5. recv(4, "Hello",...) = 5 B4. close(3) = 0 6. close(4) = 0 The ground truth B 28

23 What we want is this 1. socket() = 3 B1. socket() = 3 2. bind(3,...) = 0 3. listen(3, 1) = 0 B2. connect(3,...) = 0 4. accept(3,...) = 4 B3. send(3, "Hello",...) = 5 5. recv(4, "Hello",...) = 5 B4. close(3) = 0 6. close(4) = 0 The ground truth!!!!!!! Goal: find an equivalent interleaving B 29

24 Observation 1: Order Equivalence! Node Node B 1. socket() = 3 1. socket() = 3 2. bind(3,...) = 0 2. connect(3,...) = 0 3. listen(3, 1) = 0 3. send(3, "Hello",.) = 5 4. accept(3,...) = 4 4. close(3) = 0 5. recv(4,"hello",..) = 5 6. close(4) = 0 - one trace per host - local order but no global order Q: how do we reconstruct what really happened? The socket() calls are not visible to the other side Some orders are equivalent! 30

25 Observation 2: Return Values Guide Ordering! Node Node B 1. socket() = 3 1. socket() = 3 2. bind(3,...) = 0 2. connect(3,...) = 0 3. listen(3, 1) = 0 3. send(3, "Hello",.) = 5 4. accept(3,...) = 4 4. close(3) = 0 5. recv(4,"hello",..) = 5 6. close(4) = 0 - one trace per host - local order but no global order Q: how do we reconstruct what really happened? 31

26 Return values guide ordering! 2. bind(3,...) = 0! 3. listen(3, 1) = 0 B2. connect(3,...) = 0! One valid ordering: all syscalls returned successfully. 2. bind(3,...) = 0 B2. connect(3,...) = -1, ECONNREFUSED second valid ordering: connect failed with ECONNREFUSED. 3. listen(3, 1) = 0! call s return value may-depend-on a remote call s action Result indicates order of calls 32

27 Deciding call order bind socket getsockname getsockopt, setsockopt accept getpeername poll, select connect recv, recvfrom, recvmsg, read listen close, shutdown send, sendto, sendmsg, write, writev, sendfile full set of may-depend-on relations 33

28 Ordering lgorithm Input traces B socket socket lgorithm process bind listen connect send Output Ordering accept recv 34

29 Ordering lgorithm Input traces B socket socket lgorithm process Try socket on host : accepted bind listen accept connect send Output Ordering socket recv 35

30 Ordering lgorithm Input traces B lgorithm process listen connect Try connect on host B: rejected accept send Output Ordering recv B socket socket bind 36

31 Ordering lgorithm Input traces B lgorithm process listen connect Try listen on host : accepted accept send Output Ordering recv B socket socket bind listen 37

32 Ordering lgorithm recv Input traces B send lgorithm process Try recv on host : rejected TCP BUFFER: Output Ordering B B socket socket bind listen connect accept Hola! 38

33 Ordering lgorithm Input traces B recv None send lgorithm process Try send on host B: accepted TCP BUFFER: Output Ordering B B socket socket bind listen connect accept B send Hola! 39

34 Ordering lgorithm Input traces B recv None lgorithm process Try send on host B: accepted TCP BUFFER: Hello Output Ordering B B socket socket bind listen connect accept B send Hola! 40

35 Ordering lgorithm Input traces B lgorithm process recv TCP BUFFER: Hello None Try recv on host : Fatal Error Output Ordering B B socket socket bind listen connect accept B send Hola! 41

36 Outline Motivation NetCheck Overview Trace Ordering Network Model Fault Classification Results / Conclusion Model Fatal Error ccept Reject 42

37 Network Model Simulates invocation of a syscall datagrams sent/lost reordering / duplication is notable track pending connections buffer lengths and contents send -> put data into buffer recv -> pop data from buffer! Simulation outcome ccept can process (correct buffer) Reject wrong order (incomplete buffer) Model Fatal Error Permanent reject abnormal behavior (incorrect buffer) ccept Reject 43

38 Network Model Simulates invocation of a syscall Capture programmer assumptions ssumes a simplified network view ssume transitive connectivity Little, random loss No middle boxes ssume uniform platform Flag OS differences 44

39 How Model Return Values Impact Trace Ordering Blackbox Tracing mechanism Host Traces Ordering lgorithm Diagnoses Engine Diagnosis Input syscall simulation result simulation state errors Output Network Model NetCheck Trace Ordering: linear running time (total trace length) * number of traces 45

40 Outline Motivation NetCheck Overview Trace Ordering Network Model Fault Classification Results / Conclusion (c) Fault Classifier Output 46 46

41 Fault Classifier Goal: Decide what to output Problem: Show relevant information Fault classifier: global (rather than local) view uncovers high-level patterns by extracting low-level features Examples: middleboxes, non-transitive connectivity, MTU, mobility, network disconnection ll look like loss, but have different patterns in the context of other flows 47

42 Fault Classifier Options to show different levels of detail Network admins / developers detailed info Network Configuration Issues End users Classification Recommendations Traffic Statistics Problem Detected 48

43 Outline Motivation NetCheck Overview Trace Ordering Network Model Fault Classification Results / Conclusion 49

44 Evaluation: Production pplication Bugs Reproduce reported bugs from bug trackers (Python, pache, Ruby, Firefox, etc.) total of 71 bugs Grouped into 23 categories Virtualization incurred/portability bugs SO_REUSEDDR behaves differently across OSes accept inherit O_NONBLOCK Correct analysis of >95% bugs 50

45 Evaluation: Observed Network Faults Twenty faults observed in practice on a live network MTU bug Intermediary device Port forward Traffic sent to non-relevant addresses Provide supplemental info packet loss buffers being closed with data in 90% of cases correctly detected 51

46 General Findings in Practice Middle boxes Multiple unaccepted connections client behind NT in FTP TCP/UDP non-transitive connectivity in VLC Complex failures o VirtualBox send data larger than buffer size o Pidgin returned IP different from bind o Skype NT + close socket from a different thread Used on Seattle Testbed seattle.poly.edu 52

47 NetCheck Performance Overhead VLC Skype Telnet Firefox SSH 53

48 Conclusion Built and evaluated NetCheck, a tool to diagnose network failures in complex apps! Key insights: model the programmer s misconceptions relation between calls reconstruct order NetCheck is effective Everyday applications & networks Real network / application bugs No per-network knowledge No per-application knowledge Try it here: 54