Exposing server performance to network managers through passive network measurements

Size: px

Start display at page:

Download "Exposing server performance to network managers through passive network measurements"

Denis Watkins
5 years ago
Views:

1 Exposing server performance to network managers through passive network measurements Jeff Terrell Dept. of Computer Science University of North Carolina at Chapel Hill October 19,

2 databases web secure web Aggregation Point Server Farm Internet storage portal mail Problem: Monitor server performance 1. passively (no instrumenting or probes) 2. pervasively (all servers) 3. in real-time 2

3 TCP/IP headers Monitoring Traffic Router/Firewall Server Farm packets packets Clients Span port (passive) Aggregation point (pervasive) Analysis Gigabit Streaming (real-time) Output 3

4 Monitoring Traffic Local Net Border link Internet link tap (passive) TCP/IP headers Analysis Output Aggregation point (pervasive) Output includes: real-time display alarms/notifications forensic analysis 4

5 databases web secure web Aggregation Point Server Farm Internet storage portal mail SSL traffic (i.e. no accessible payload) 5

6 TCP Connection Vectors A connection vector is a representation of the application-level dialog in a TCP connection. For example: Client: A A time Server: T B T T B Response Times Client-side think-times 6

7 Constructing connection vectors client monitor server client monitor server SYN SYN/ACK ACK SEQ= think-times relative to monitor time ACK=372 SEQ=1460 ACK=1460 SEQ=2920 SEQ=3030 time 3030 ACK=3030 SEQ= ACK=712 SEQ=3730 ACK=3730 FIN 700 (Connection closes...) 7

8 Needed Measurements Application-level measurements from TCP/IP headers: server response time count of application-level requests/responses per server (i.e. server load) per connection (i.e. dialog length) size of application-level requests/responses connection duration 8

9 Viability of Netflow What can Netflow provide? server response time - No count of application-level requests/responses - No per server (i.e. server load) - sort of per connection (i.e. dialog length) - No size of application-level requests/responses - No connection duration - sort of 9

10 Previous approach Previous work by Felix Hernandez-Campos on building connection vectors. Internet Capture Offline Analysis Local Net Packet header traces (on disk) Connection vectors (on disk) 10

11 Our approach Our innovation: build connection vectors online, with a single pass. Internet 1 Gbps fiber link 1.4 GHz Xeon 1.5 GB RAM No packet loss Now, no intermediate files Capability for continuous measurement Capture/Analysis Elements of connection vectors available immediately Local Net Connection vectors Capability for online understanding of server performance 11

adudump The tool we wrote to do this is called adudump. Here s the output of adudump for an example connection: TYPE TIMESTAMP LOCAL_HOST DIR REMOTE_HOST OTHER_INFO SYN: 1202706002.650917 190.40.1.180.

12 adudump The tool we wrote to do this is called adudump. Here s the output of adudump for an example connection: TYPE TIMESTAMP LOCAL_HOST DIR REMOTE_HOST OTHER_INFO SYN: < RTT: > SEQ: < ADU: < SEQ ADU: > SEQ ADU: < SEQ ADU: > SEQ END: < computing all kinds of things in real-time...contextual information as well as ADUs... 12

13 Data For this paper: 66 days 1.54 TB (uncompressed) 16.8 billion requests and responses 1.6 billion connections Overall: 180 days 3.35 TB (uncompressed) 34.8 billion requests and responses 4.0 billion connections 13

14 Case study: the incident databases web secure web Aggregation Point Server Farm Internet storage portal mail Thursday, April 10th, 4:28pm Representative, though manual analysis 14

15 Case study: the issue avg. response time over 15 minutes (s) weeks ago last week this week 0 Fri Sat Sun Mon Tue Wed Thu Date/time (15-minute intervals) 15

16 Case study: the issue cumulative probability comprise ~35% of the distribution. That is, 35% of response times are < 10 ms. response times < 10 ms Server s response time (s) 16

17 Case study: the issue 1 cumulative probability ~95% of response times......are < 1s Server s response time (s) 17

18 Case study: the issue cumulative probability Server s response time (s) 18

19 Case study: the issue cumulative probability Historical (all prior to incident) Historical (Thursday 3:28-4:28 pm) Server s response time (s) So, generally faster responses on Tuesday afternoons (i.e. lower response times). 19

20 Case study: the issue cumulative probability Server s response time (s) So, generally slower responses during the hour of the incident. 20

21 Case study: investigation What could cause this incident? Larger requests (more processing required) Larger responses (implying more processing) More requests per connection (more work) More requests per time unit (more work) 21

22 Case study: investigation cumulative probability Before - all Before - hours Hour of incident Client s request size (bytes) 22

23 Case study: investigation What could cause this incident? Larger requests (more processing required) Larger responses (implying more processing) More requests per connection (more work) More requests per time unit (more work) 23

24 Case study: investigation cumulative probability Before - all Before - hours Hour of incident e+06 Server s response size (bytes) 24

25 Case study: investigation What could cause this incident? Larger requests (more processing required) Larger responses (implying more processing) More requests per connection (more work) More requests per time unit (more work) 25

26 Case study: investigation 1 cumulative probability Before - all Before - hours Hour of incident Epochs per connection 26

27 Case study: investigation What could cause this incident? Larger requests (more processing required) Larger responses (implying more processing) More requests per connection (more work) More requests per time unit (more work) 27

Case study: investigation # requests per hour 35000 30000 25000 20000 15000 10000 5000 point is that this is an app-level

28 Case study: investigation # requests per hour point is that this is an app-level measurement of load...not that we couldn t have figured out high load via other mechanisms. 0 Mar 13 Mar 20 Mar 27 Apr 03 Apr 10 Date 28

29 Case study: investigation time Bin 1 Bin 2 Bin

30 Case study: investigation 14 median w/ Q1 and Q3 - historical median response time (s) i.e. the median of bin 3 error bars are first and third quartile response time ordinal 30

31 Case study: investigation 14 median w/ Q1 and Q3 - historical median w/ Q1 and Q3 - incident 12 median response time (s) response time ordinal 31

32 Case study: investigation median response time (s) median w/ Q1 and Q3 - historical median w/ Q1 and Q3 - incident 94% of connections have < 3 responses (~55% have exactly 3) response time ordinal 32

33 Conclusions Achieved monitoring of server performance: for all servers, of any type in real-time, at gigabit speeds, on older hardware, completely passively. adudump data provides diagnostic insight into performance issues. 33

34 Questions? 34

Tracking the Evolution of Web Traffic:

Tracking the Evolution of Web Traffic: The University of North Carolina at Chapel Hill Department of Computer Science 11 th ACM/IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS)