Stream Mode Algorithms and. Analysis

Transcription

1 Stream Mode Algorithms and Architecture for Line Speed Traffic Analysis Steve Liu Computer Science Department Texas A&M University March 7,

2 Background Network security solutions have broad presence in every network point Antivirus scanner, network intrusion detection systems, spamming filters Most solutions designed to operate at desktops or servers serve the intended purposes very well, but they are not perfect, nothing is perfect A DoD doctrine of defense-in-depth makes sense Use layers of (different) protection tools to make intrusion very inconvenient and very expensive Our interest: enhance network security via a stream mode traffic analysis approach at Network Access Point (NAP) of an enterprise network 2

3 Stream Mode Traffic Analysis Highly concentrated traffic flow at the network access point (NAP) is an ideal location for enterprise traffic analysis Single location to observe ingress & egress flows When the conditions are right, could even slow/stop the intrusion packets before they spread too deep, too broad into the network Commercial systems Deep Packet Inspection (DPI) engines, DAG cards Some virus and spamming filters at the gateway a Firewall is one of the oldest products for such purpose p 3

4 Stream Mode Packet Flow Analysis Packet sensor Promiscuous mode NIC card, Router feed, Libpcap, TCPDUMP. N-gram rules Remote Image src Regular expression src-dest IP pairs URL Feature extractors HW:Bivio, Cloudshield, SW: Flex Feature instances How to identify malicious traffic from the time series of feature instances? Feature: Any string that fits a regular expression rule, e.g., URL link Feature instance: An instance of a feature, e.g., 4

5 Two Key Issues: Algorithms and Resource Management Fast algorithms Efficient data structures Memory efficiency i critical for stateful t detection ti e.g., a 32 bit, y/n hash table 500MB Real time vs. virtual clocks Progressive Classifier (PEC) system architecture 5

6 spamming is no longer just a Some Facts: nuisance Botnet farms can hit any target (over millions of them) bandwidth waste (3:1 or higher) Network resource exploit & information stealing (malware planting) Highly effective hit and run strategy at different protocol levels (BGP, DNS, domain name, credit card fraud) Existing anti-spamming ware Large number of software copies and signatures to maintain Comprehensive detection rules, but slow to respond Signatures management a major bottleneck Acquisition and the deployment of signatures to numerous machines A small variation in the known signatures can easily defeat a signature based filter Spammers can test their designs with anti-spamming ware before starting the (hit and run) campaign 6

7 Spamming Behavior at a Glance Spammers do not have full freedom in launching spamming. Follow the transport protocols to deliver messages Messages must be perceivable and appealing to human users Expensive to compose and personalize spamming messages: interactive (click my URL links) or passive Low yield combined with greed lead to high h spamming volumes Cheap to launch spamming: millions of zombie machines each send a few copies Any hit back, interactive method could cause severe harm to the innocents Summary Very difficult for spammers to achieve financial goals without leaving noticeable signatures, i.e. feature instances A challenge is how to keep up with their speed,, volume,, and diversity 7

8 Our Approach Lossy detection: focused mainly on the major offenders Avoid false positive Timely acquisition of the spamming signatures: features and their instances Position the detector at the Network Access Points (NAP) Regular s are expected to have white noise like distributions of strings that happen to fall into the spamming feature space Mediated delivery of bulk, legitimate The content of a spamming campaign is divided into Invariants and variants An invariant that also appears in regular s cannot be used for filtering For the first cut effort: URL (over 95% spamming have them) 8

9 Competitive Aging-Scoring Scheme (CASS) A spamming invariant (string) is called its feature instance (FI). The essence of our technique: Extract FIs of s and keep track of their occurrences. If exceeding a threshold: an UNBE stream In a naïve approach, it takes O(1) to update the score of an FI, but O(N) to update ages of all other FIs A major computing cost CASS: The time-to-live of an FI is reset each time when its score is increased by one (when a new copy arrives) The time-to-live of all other FIs is reduced by one New complexity: O(1) for both scoring and aging Exceeding a threshold: black; move it to the blacklist No further copies in a time period: white; discard the feature instance 9

10 PEC Architecture Hash table of Known strings Feature instance extraction flow Hash vs string 32bit Sendmail Berkley DB Birth& Death Of strings New string identified Aging and scoring of unknown strings 10

11 Data Structure of Scoreboard Entries for feature instances Scoreboard Hit (SH) Table Exceeds anomaly threshold (ATF)? Scoreboard Miss (SM) Table Exceeds miss threshold (MTF) Entries for feature instances 11

12 An execution snapshot of scoreboard Hash URL : (414738(20-bit)+3724(12-bit)) HashURL : (124489(20-bit)+176(12-bit)) Current feature being processed Entry moved to blacklist MOD queue Placement Active features Arranged in their ages (mod N) history The current time location The current time location newest oldest time ATF =10, MTF =20 Queue size = 20 Next feature instance The entry [ ] is purged 12

13 Testbed Environment Three Modules included: 1. generation 2. PEC (Blacklist and scoreboard): 3. Control and visualization console 13

14 Experimental Configuration generator: Intel P4-3.0 Windows XP Server: Xeon 3.0GHz, two single core CPUs, Linux, Sendmail Within a bin, the sender sends 2000 copies of s (mixed with bulks and regulars). The distribution of bulks and regulars is uniform. Default Score threshold: 50 Miss table length: 2048 The average mail size: 1.5K bytes generator sends one mail per seconds on average. 14

15 Workflow of Generation Density Generation (uniform dist.) SMTP Protocol s (bulk/regular) URL Feature Dictionary Bulk Regular U R U U.. R MIME structures Linux Server (Sendmail) Image Src Bulk Regular Random Text Spamming Keyword selection Message Composer simulation parameters ` Subject Generation Windows Control Console From Generation 15

16 Generation Generate bulk/regular mixed copies by injecting different features, such as URL links or image sources Can adjust density or interval time between bulk copies, placement of variants and invariants. According to the parsed parameters, message composer picks the materials to generate MIME messages (bulks or regulars). extracted from 2005 TREC Public Spam Corpus, Random Text: from Internet Keywords: User defined. The message composer calls an SMTP module to send the generated s. 16

17 Detection Latency of Single UNBE source Fix threshold and age table length under different densities. Test six different UNBE densities (50, 100, 150, 200, 300 UNBE messages/bin) Experimental Value Expected Value Detection Laten ncy Number of messages in a bin 17

18 Interactive Effects Under Multiple l UNBE Sources Observe the change of the detection latency of UNBE A in the tests. Given an UNBE source A, six tests were made where one addition UNBE source is added to the experiment at a time. The density of A is fixed at 100 instances per bin, and the density of every remaining UNBE sources is increased from 50 to 300 instance/bin Line Test2: Detection latency of UNBE A when adding 2 additional UNBE sources. Conclusion: The more UNBEs sources, the detection latency of an UNBE decreases. Detecti on latency test 1 test 2 test 3 test 4 test 5 test 6 other sources Number of messages in a bin for each non-a UNBE 18

19 Throughput of Feature Parser 30 Bodys/sec Thro oughput ( K 3.0K 4.5K 6.0K 7.5K Size of Mial Body (K Bytes) The average size is from 1.5 KB to 7.5 KB, and each has 2 URLs. 19

20 Throughput of Scoreboard and Blacklist Scoreboard: 1.2M transactions Blacklist: 0.9M (avg. 30 B) URLs, without including database access Throughp put ( K URLs/sec URL length (bytes) 20

21 Pointer Table During the detection ti time window, only a limited it number of hashed h values need to be tracked Full table for 32-bit hash system takes too much space Higher order bits used as the index, and the rest, and the rest bits maintained by a linked list (for each entry) If pointer table uses 20 bits for indexing, that means it has 1M entries, and age table length is 20K~70K, the maximum depth of linked list pointed by pointer table is 2. Very effective in reducing the actual space requirements, at minor cost of more search cycles 21

22 Current Work The first generation PEC demonstrates the feasibility of high speed UNBE filtering Not meant to replace existing solutions, but to defeat major offenders (80-20 rule) Next Step Packet level filtering Handle multiple features (bad words, dirty subnets, black lists, etc) Integration with existing tools 22

23 23

24 Screen Shot (4) Aging out an Orphaned Packet \ 24

25 Screenshot (7) Parsing An message has 3 packets. Parser 1 uses DFA 0 to extract a URL link, and uses DFA 1 to extract a domain name in this message. 25

26 System Performance Parameters 26

27 Thank You! 27