Entropy-based event detection

Transcription

1 Entropy-based event detection New Zealand The Univer rsity of Auckland Ulrich Speidel Department of Computer Science The University of Auckland Joint work with Raimund Eimann

2 What s the problem? Complex systems such as computer networks - have observables that yield multivariate time series data Chaotic behaviour is actually normal (to an extent) New Zealand The Univer rsity of Auckland So: How does one detect significant system events and how does one distinguish them from normal behaviour? What if we don t know very well in advance what effect the event will have on the observables?

3 Scenario: Computer networks Internal network Upstream router 1 New Zealand Gateway router (DMZ) Upstream router 2 The Univer rsity of Auckland Upstream router 3

4 Scenario: Computer networks Internal network Upstream router 1 New Zealand The Univer rsity of Auckland Gateway router (DMZ) Normal traffic across router: Chaotic: many different packet sizes, protocols, addresses, ports But: patterns exist (e.g., repeated HTTP connections to same web site, polls, other handshakes) Very complex multivariate statistics Upstream router 2 Upstream router 3

5 Scenario: Computer networks Network events Internal network Upstream router 1 New Zealand The Univer rsity of Auckland (D)DoS attacks Port scans SPAM/SPIM Gateway router (DMZ) Link failures / service outages Other things we don t know about? Upstream router 2 Upstream router 3

6 Other examples Industrial process data (e.g., sensor data in smelter processes) Medical data (e.g., ECG/EEG/BP data) Traffic monitoring i Airline data New Zealand rsity of Auckland The Univer Need to detect anomalies in order to find out what causes them

7 Conventional approaches E.g., monitor packet rate/interarrival times - not useful if router hits saturation during normal operation rsity of Auckland New Zealand The Univer

8 Conventional approaches Monitor packet size distribution - complex diagram (histogram), fluctuates significantly with time, does not detect some events (e.g., port scans or link failures may go undetected) rsity of Auckland New Zealand The Univer

9 Conventional approaches Monitor individual protocols, ports, or payload - generally too selective and complex to monitor information is hard to aggregate and events are easily missed (especially new ones) Currently one of the more popular techniques, thoughh rsity of Auckland New Zealand The Univer

10 Conventional approaches Most if not all conventional approaches are complex and rather narrowband Focus is on a single observable, not aggregate New Zealand Patterns do not play a major role rsity of Auckland The Univer

11 Patterns IP network traffic contains patterns E.g., handshakes, request/response packets in protocols such as HTTP etc. E.g., certain ports and IP addresses are seen more often than others, and tend to occur in close temporal proximity Permits a certain degree of predictability (low entropy) New Zealand rsity of Auckland The Univer In other words: certain possible patterns occur much more often than others

12 Entropy monitoring Entropy = information rate (f.t.p.o.t.t.*) Postulate: Entropy of network traffic changes as patterns in the traffic change Network events cause change in patterns and hence change the observed entropy New Zealand The Univer rsity of Auckland Not in itself a new concept: - Kulkarni, Bush, and Evans (2002): approximate entropy by LZ compression - Feinstein, Schnackenberg, Balupari, and Kindred (2003): use Shannon entropy - Wagner and Plattner (2005): also use compression-based monitoring *for the purposes of this talk

13 Entropy measurement Kulkarni, Bush, and Evans (2002): Quality of entropy-based detection depends on having a good entropy measure Fundamental problem: computability We can t measure, but we can estimate New Zealand The Univer rsity of Auckland Classical estimators: Statistical/Shannon (bad), but also Lempel- Ziv algorithms (better, 1976 production complexity, LZ77, LZ78) Fundamental problem: Overestimation or time/space complexity

14 Possible alternative: T-Entropy Entropy measure developed by Mark Titchener in the late 1990 s Based on the duality between finite strings and a family of recursively constructed variable-length code sets called T-codes Can be implemented to run in O(N log N) [Speidel and Yang 2005] New Zealand Seems to be more sensitive for short strings than LZ-based estimators t but correlates well with the latter 5000 [Speidel 2009] 4800 The Univer rsity of Auckland T-complexity LZ production complexity

15 T-entropy: conceptual overview x= T-decomposition duality T-code decoding tree C T = weighted number of steps needed to build x common part of longest codewords x0 x1 New Zealand The Univer rsity of Auckland T-complexity C T T-information I T T-entropy H T C T = log of # of internal nodes in tree linearization wrt. x via inverse logarithmic integral Gradient I T / x

16 Network event detection: The Univer rsity of Auckland New Zealand header payload p a c k e t s Methodology in principle i n g m a p p 0x84 0x75 0xc2 0x88 0x84 v a l u e s 0x9b 0x74 0x77 0x9d 0x9d 0xcd m a p p e d 0xcd 0xcd 0x84 0x95 0x9b 0x78 0x8d 0x84 0x84 0xba H1 Window position 1 entropy H2 Window position 2 entropy H3 Window position 3 entropy Observe packets, e.g., at border gateway Map packet properties (e.g., length, port, interarrival time, protocol, src/dest address) into binary 8-bit symbols (one or several per packet) Aggregate several hundred or more successive mapped packets into a symbol string (sliding window) Measure average T-entropy of that string Repeat for sliding window over time and plot T-entropy values against time sliding window

17 Experimental results Three hour IP datagram traces from U of Auckland s DMZ gateway Typical datagram rate about 8000 datagrams per second Processing time for a three hour trace file: 45 minutes on a normal stateof-the-art PC (2006) New Zealand The Univer rsity of Auckland Various mappings and filters were applied The ones shown here today use the full IPv4 information + 48 bytes of the payload and use a 5000 packet window shifting i by seconds at a time

18 Experimental results 3.5 Our first stab at the data with a 500 packet window The Univer rsity of Auckland New Zealand Average T-e -entropy [b bits/byte] T-entropy of normal traffic complete IPv4 header & first 26 bytes of payload Windows Messaging Service spam SYN flood attack Time [s]

19 Experimental results Observation 1: Data is noisy! Observation 2: Depth of entropy drops depends on size of window the longer the window, the shallower the drops Observation 3: The longer the window, the less noise we get Question: Can we define a kind of SNR (signal-to-noise ratio) and try to optimize the window size? Findings: New Zealand SNR SNR Window size of approx maximizes SNR in this sample The Univer rsity of Auckland trace trace Fragment length (symbols) Window size (symbols) Optimal window size is event duration dependent

20 Before SNR optimization The Univer rsity of Auckland New Zealand Average T-e -entropy [b bits/byte] Non-SNR-optimized i with window size Time [s]

21 After SNR optimization The Univer rsity of Auckland New Zealand Av verage T-e entropy [b bits/byte] SNR-optimized i with window size Time [s]

22 How do we know that the drops are caused by the events? Need to show that the events are both necessary and sufficient to cause entropy drops Can show necessity by removing event-related packets New Zealand Can show sufficiency by artificially inserting synthetic events into the traces (simulation) rsity of Auckland The Univer

23 Entropy - unfiltered The Univer rsity of Auckland New Zealand Av verage T-e entropy [b bits/byte] SNR-optimized with window size Time [s]

24 Entropy - filtered 3.5 The Univer rsity of Auckland New Zealand Av verage T-e entropy [b bits/byte] MS Messenger spam filter applied to packet stream Time [s]

25 Entropy - unfiltered The Univer rsity of Auckland New Zealand Av verage T-e entropy [b bits/byte] SNR-optimized with window size Time [s]

26 Entropy - filtered The Univer rsity of Auckland New Zealand Av verage T-e entropy [b bits/byte] All packets with SYN flag removed (attack and 0.5 non-attack) Time [s]

27 Entropy - filtered 3.5 The Univer rsity of Auckland New Zealand Av verage T-e entropy [b bits/byte] All packets with SYN flag or MS Messenger Spam signature removed (attack and non-attack) Time [s]

28 Some of the other stories The Univer rsity of Auckland New Zealand Av verage T-e entropy [b bits/byte] Local Web download inside DMZ Large number of almost uniform packets from outside (spam delivery?) Strange traffic (only 52 byte packets originating from port 25) Time [s]

29 T-entropy sensitivity The Univer rsity of Auckland New Zealand Average T-e -entropy [b bits/byte] No packet injection 500 packets/s 2000 packets/s 5000 packets/s packets/s packets/s packets/s Time [s]

30 Comparison T-entropy vs. LZ The Univer rsity of Auckland New Zealand Ave verage T-e -entropy [b bits/byte] LZ77

31 Comparison T-entropy vs. Shannon The Univer rsity of Auckland New Zealand T-entropy Shannon entropy

32 Observations T-entropy suffers least from overestimation and has a denser range LZ production complexity does slightly better than T-entropy but practical algorithms are slow New Zealand Combination of different measures may be useful in event classification rsity of Auckland The Univer

33 Observations Monitoring just a subset of data from IP headers can mean high or low normal entropy May monitor for drops OR rises depending on normal entropy Normal entropy seems to be site-dependent rsity of Auckland New Zealand The Univer

34 IP datagrams and entropy Entropy in TCP/IP traffic is contributed by several sources: IP header, usually 20 bytes TCP/UDP header, usually 20 bytes TCP, 4 bytes UDP Payload (packet content), first N bytes captured by tcpdump utility New Zealand The Univer rsity of Auckland TCP packet encapsulated inside IP datagram IP header IP header TCP header UDP header Payload Payload UDP packet encapsulated inside IP datagram May want to use all or just part of the header and packet information Ulrich Speidel Detecting Network Events via T-entropy - ulrich@cs.auckland.ac.nz

35 Entropy sources in IP headers New Zealand rsity of Auckland The Univer Version (4 bits) IHL Type of service (4 bits) (TOS, 8 bits) Identification (16 bits) Time-to-live Protocol (TTL, 8 bits) ( ) Total length of datagram (16 bits) D M Fragment offset (13 bits) (8 bits) Header CRC checksum (16 bits) Source IP address (32 bits) Destination IP address (32 bits) Entropies typically observed at a gateway router in normal traffic: high low Ulrich Speidel Detecting Network Events via T-entropy - ulrich@cs.auckland.ac.nz

36 Entropy sources in TCP/UDP headers Source port Destination port Own sequence number Acknowledgement number HdrLen Flags Advertised window New Zealand Checksum UrgPtr Entropies typically observed at a gateway router in normal traffic: high The Univer rsity of Auckland low TCP only UDP has a 16 bit length field before the checksum & UDP checksum covers header only Ulrich Speidel Detecting Network Events via T-entropy - ulrich@cs.auckland.ac.nz

37 Future work At some observation sites, sampling all packets is not feasible Need to look at flow records and sampled packets rather than full records This means throwing information away that may be useful rsity of Auckland New Zealand The Univer

38 Other applications Technique isn't restricted to networks The Univer rsity of Auckland New Zealand Observation: most time-varying observables of complex systems have a pretty stable entropy as long as the system itself is stable Can use entropy changes as indicators for events

39 Example MIMO communication system The Univer rsity of Auckland New Zealand Multi-user, mobile, etc. - channel conditions fluctuate naturally in complex way

40 Example How do we notice permanent changes that may indicate deterioration in equipment performance? rsity of Auckland New Zealand The Univer

41 Possible answer The Univer rsity of Auckland New Zealand Monitor entropy of channel quality data from feedback If entropy remains near-constant compared to reference sample, it is usually reasonable to assume that all is OK If entropy rises or falls significantly - something is afoot!

42 For the conspiracy theorists... Entropy of pager messages of 9/11... T-entropy [nats/byte] T T-entropy of byte+ chunks of 9/11 pager messages New Zealand The Univer rsity of Auckland :30:00 09:00:00 09:30:00 10:00:00 10:30:00 11:00:00 11:30:00 12:00:0 Time First plane hits Second plane hits All NY airports shut Ulrich Speidel - Detecting Network Sth Tower Events collapses via T-entropy -Nth ulrich@cs.auckland.ac.nz Tower collapses Message source: wikileaks.org

43 Conclusions Duality between T-codes and strings opens up a number of areas of application network event monitoring is one of them Seems to be pretty useful in network event detection! Theoretical results are slowly catching up rsity of Auckland New Zealand The Univer