Sherlock Diagnosing Problems in the Enterprise

Size: px

Start display at page:

Download "Sherlock Diagnosing Problems in the Enterprise"

Duane York
5 years ago
Views:

1 Sherlock Diagnosing Problems in the Enterprise Srikanth Kandula Victor Bahl, Ranveer Chandra, Albert Greenberg, David Maltz, Ming Zhang Enterprise Management: Between a Rock and a Hard Place Manageability Stick with tried software, never change infrastructure Cheap Upgrades are hard, forget about innovation! Usability Keep pace with technology Expensive IT staff in 1000s 72% of MS IT budget is staff Reliability Issues Cost of down-time 1

2 Well-Managed Enterprises Still Unreliable Fraction Of Requests Response time of a Web server (ms) 85% Normal 10% Troubled % Down 10% responses take up to 10x longer than normal How do we manage evolving enterprise networks? Well-Managed Enterprises Still Unreliable Fraction Of Requests Response time of a Web server (ms) 85% Normal ~10% 10% Troubled 0.7% Down How 10% do responses we manage take evolving up to 10x enterprise longer than networks? normal 2

3 Current Tools Miss the Forest for the Trees Authentication Server DNS Web Server SQL Backend Monitor Individual Boxes or Protocols Flood admin with alerts Don t convey the end-to-end picture But, the primary goal of of enterprise management is is to to diagnose user-perceived problems! Client Sherlock Instead of looking at the nitty-gritty of individual components, use an end-to-end approach that focuses on user problems 3

4 Challenges for the End-to-End Approach Don t know what user s performance depends on Challenges for the End-to-End Approach E.g., Web Connection DNS Auth. Server Web Server SQL Backend Don t know what user s performance depends on Dependencies are distributed Dependencies are non-deterministic Don t know which dependency is causing the problem Server CPU 70%, link dropped 10 packets, but which affected user? Client 4

5 Sherlock s Contributions Passively infers dependencies from logs Builds a unified dependency graph incorporating network, server and application dependencies Diagnoses user problems in the enterprise Deployed in a part of the Microsoft Enterprise Sherlock s Architecture 5

6 Sherlock s Architecture Network Dependency Graph Servers + User Observations Clients Inference Engine Web2 30ms Web1 File1 1000ms Timeout = List Troubled Components Sherlock works for various client-server applications Sherlock s Architecture Network Dependency Graph Servers + User Observations Clients Inference Engine = List of Suspicious Components Sherlock works for various client-server applications 6

7 DNS Video Server Data Store How do you automatically learn such distributed dependencies? Strawman: Instrument all applications and libraries Not Practical Sherlock exploits timing info My Client talks to B Dt My Client talks to C Time If talks to B, whenever talks to C Dependent Connections 7

8 Strawman: Instrument all applications and libraries Not Practical Sherlock exploits timing info B B B B C BB B Dt False Dependence Time If talks to B, whenever talks to C Dependent Connections Strawman: Instrument all applications and libraries Not Practical Sherlock exploits timing info B C Dt Inter-access time Dependent iff Dt << Inter-access time If talks to B, whenever talks to C B Time Dependent Connections As long as this occurs with probability higher than chance 8

9 Sherlock s Algorithm to Infer Dependencies Infer dependent connections from timing Video Store DNS Dependency Graph Sherlock s Algorithm to Infer Dependencies Infer dependent connections from timing Infer topology from Traceroutes & configurations Video Store DNS Bill s Client Video Store DNS Dependency Graph Billfi DNS Bill Video Video fistore Bill Watches Video Works with legacy applications Adapts to changing conditions 9

10 But hard dependencies are not enough But hard dependencies are not enough DNS Bill s Client Video Store Video fistore Billfi DNS Bill Video p3 p1=10% p2=100% Bill watches Video If Bill caches server s IP DNS down but Bill gets video Need Probabilities Sherlock uses the frequency with which a dependence occurs in logs as its edge probability 10

11 How do we use the dependency graph to diagnose user problems? Diagnosing User Problems DNS Bill s Client Video Store Billfi DNS Bill Video Video fistore Bill Watches Video Which components caused the problem? Need to disambiguate!! 11

12 Diagnosing User Problems Sales DNS Bill s Client Video Store Video2 Billfi Sales Billfi DNS Video fistore Video2 fistore Bill Video Paul Video2 Bill Sees Sales Bill Watches Video Paul Watches Video2 Which Disambiguate components by caused correlating the problem? Use Across correlation logs to from disambiguate!! same client Across clients Prefer simpler explanations Will Correlation Scale? 12

13 Will Correlation Scale? Corporate Core Building Network Microsoft Internal Network O(100,000) client desktops O(10,000) servers O(10,000) apps/services O(10,000) network devices Campus Core Data Center Dependency Graph is Huge Will Correlation Scale? Can we evaluate all combinations of component failures? The number of fault combinations is exponential! Impossible to compute! 13

14 Scalable Algorithm to Correlate Only a few faults happen concurrently But how many is few? Evaluate enough to cover 99.9% of faults For MS network, at most 2 concurrent faults 99.9% accurate Exponential Polynomial Scalable Algorithm to Correlate Only a few faults happen concurrently But how many is few? Only few nodes change state Evaluate enough to cover 99.9% of faults For MS network, at most 2 concurrent faults 99.9% accurate Exponential Polynomial 14

15 Scalable Algorithm to Correlate Only a few faults happen concurrently But how many is few? Only few nodes change state Evaluate enough to cover 99.9% of faults For MS network, at most 2 concurrent faults 99.9% accurate Re-evaluate only if an ancestor changes state Exponential Polynomial Reduces the cost of of evaluating a case by by 30x-70x Results 15

16 Experimental Setup Evaluated on the Microsoft enterprise network Monitored 23 clients, 40 production servers for 3 weeks Clients are at MSR Redmond Extra host on server s Ethernet logs packets Busy, operational network Main Intranet Web site and software distribution file server Load-balancing front-ends Many paths to the data-center What Do Web Dependencies in the MS Enterprise Look Like? 16

17 What Do Web Dependencies in the MS Enterprise Look Like? Auth. Server Client Accesses Portal What Do Web Dependencies in the MS Enterprise Look Like? Auth. Server Client Accesses Portal 17

18 What Do Web Dependencies in the MS Enterprise Look Like? Auth. Server Client Accesses Portal Client Accesses Sales Sherlock discovers complex dependencies of of real apps. What Do File-Server Dependencies Look Like? File Server 100% Auth. Server WINS DNS Proxy 10% 6% 5% 2% Backend Server 1 8% 5% 1%.3% Backend Server 2 Backend Server 3 Backend Server 4 Client Accesses Software Distribution Server Sherlock works for many client-server applications 18

19 Sherlock Identifies Causes of Poor Performance Dependency Graph: 2565 nodes; 358 components that can fail Component Index Time (days) 87% of of problems localized to to components Sherlock Identifies Causes of Poor Performance Inference Graph: 2565 nodes; 358 components that can fail Component Index Time (days) Corroborated the three significant faults 19

20 Sherlock Goes Beyond Traditional Tools SNMP-reported utilization on a link flagged by Sherlock Problems coincide with spikes Sherlock identifies the troubled link but SNMP cannot! Comparing with Alternatives Dataset of known (fault, observations) pairs Accuracy = 1 (Prob. False Positives + Prob. False Negatives) SCORE (non-probabilistic) 53% 20

21 Comparing with Alternatives Dataset of known (fault, observations) pairs Accuracy = 1 (Prob. False Positives + Prob. False Negatives) Shrink (probabilistic) SCORE (non-probabilistic) 53% 59% Comparing with Alternatives Dataset of known (fault, observations) pairs Accuracy = 1 (Prob. False Positives + Prob. False Negatives) Sherlock 91% Shrink Shrink (probabilistic) SCORE (non-probabilistic) 53% 59% Sherlock outperforms existing tools! 21

22 Time to Localize Faults Experiments From Inference Graph, create scaled versions Time to to Localize faults is is roughly linear! Sensitivity to Errors in the Inference Graph Perturb Inference Graph in Several ways Robust up to to 10% error in in dependencies Can run off auto-generated graphs 22

23 Conclusions Sherlock passively infers network-wide dependencies from logs and traceroutes It diagnoses faults by correlating user observations It works at scale! Experiments in Microsoft s Network show Finds faults missed by existing tools like SNMP Is more accurate than prior techniques Steps towards a Microsoft product Sherlock System A Authentication Servers A Web Servers A Exchange Inference Engine Cluster A A A Client Machines 23

24 Ongoing Work Helping MS product groups that maintain business apps like payroll Integrating automated dependency inference into MS product MOM Sherlock System MOM Scripts SMS A Active Directory A MSWeb A Exchange Inference Engine Cluster SMARTS SNMP NetFlow A A A Client Machines 24

25 Challenge 1: At what level of abstraction should dependencies be modeled so as to be feasibly automated client B DNS Server SQL Server client A web front 1 client A Root causes DNS Server A DNS web front 1 SQL Server WF1 SQL A WF1 User Experience A WF1 25

26 The Inference Graph DNS Web1 SQL Root Causes Computer (IP address) Service (IP address, port pair) Routers, Links Root causes Client A DNS Web1 SQL Meta-nodes A DNS A Web1 Web1 SQL Client A User Experience A Web1 Challenge 2: Non-deterministic dependencies Model allows significant re-use of nodes DNS Web1 SQL Client A Client B DNS Web1 SQL Web1 SQL Client A Client B A DNS B DNS A Web1 A Web1 B Web1 B Web1 26

27 The Inference Graph DNS Web1 SQL Root Causes Computer (IP address) Service (IP address, port pair) Routers, Links Root causes Client A DNS Web1 SQL Meta-nodes A DNS A Web1 Web1 SQL Client A User Experience A Web1 Challenge 4: How to find dependent connections without deep packet inspection? DNS Web1 SQL Client request depends on DNS Server response depends on SQL backend Host A Packet from/to B Δt Packet from/to C time Idea If host A talks to host B, whenever it talks to host C, then the two connections are dependent Client A Algorithm Given a pair A C, find all other servers that are accessed by A 27

28 Experimental Setup Sherlock Agents on Clients Local Servers Monitored 40 production servers for 3 weeks Intranet Portal and Software Dist. File-server Load-balancing and ECMP 28

10 BEST PRACTICES TO STREAMLINE NETWORK MONITORING. By: Vinod Mohan

10 BEST PRACTICES TO STREAMLINE NETWORK MONITORING By: Vinod Mohan 10 Best Practices to Streamline Network Monitoring Introduction As a network admin, you are tasked with keeping your organization s network