Black and White Lists

Transcription

1 Black and White Lists Open Source Sensor Grids and Highly Predictive Black Lists Marcus H. Sachs SANS Internet Storm Center

2 Black Lists A collection of source IP addresses that have been deemed undesirable Typically these addresses have been observed as the source of previous illicit activities The concept of blacklisting is a wellestablished defensive measure Compiling and sharing lists of the worst offenders of unwanted traffic is a blacklisting strategy that has remained virtually unchanged since the early years of the Internet

3 White Lists A collection of source IP addresses that have been deemed desirable Typically these addresses belong to internal resources or well known friendly computers on the Internet The concept of whitelisting has also been a wellestablished defensive measure since the origins of the Internet community Whitelists are the allow portion of a firewall rule set Most security professionals prefer a deny all rule to start with, then allow only known IP addresses inside

4 A Source of B/W Lists: DShield SANS Internet Storm Center uses the DShield distributed incident detection technology The database receives over ten million log lines daily from intrusion detection systems run by volunteers world wide Thousands of system administrators send in additional observations and findings via and the web Volunteer incident handlers analyze detected problems and anomalies, then post a daily diary of analysis Analysis also feeds SANS research efforts Direct feed to courseware, Top20 list, and weekly newsletters Service is free to the Internet community

5 A Short Digression: The SANS Internet Storm Center SANS ISC analysis parallels weather analysis Small sensors in as many places as possible Sensors send raw information to multiple regional data collection points for early analysis and correlation Regional collection feeds a national or global watch center DShield sensors see tcp or udp flows, then the SANS ISC infers developing situations There is no content monitoring capability

6 SANS ISC Statistics: Reports Submitted Past 30 Days: 414,196,491 log lines Past Seven Days: 95,558,962 log lines Past 24 Hours: 11,906,713 log lines All submitted by volunteer collectors!

7 The SANS ISC Process Data Collection Analysis Dissemination DShield Users DShield.org

8 SANS ISC Services Sensor software for dozens of popular devices Private website and report generation capability for participants Fightback program with major ISPs Large incident database with preconfigured queries and search capabilities

9 Sensor Software DShield client software is available for over 60 different devices Most popular hardware and software devices are included, such as Linksys NetGear Cisco Snort ZoneAlarm Checkpoint SonicWall PortSentry BlackIce Norton Personal Firewall Windows Firewall Client software includes an installation program and directions for use If you have a device or software that is not supported we will write the code needed to build a sensor

10 Private Reporting and Analysis Each participant receives a custom view of the DShield database Users can compare own data with information from thousands of other sensors All submitted data is available for later retrieval and analysis Pre generated data visualization templates make viewing past events very easy for most users

11 Sensor Report

12 Fightback Capability Unique to DShield, participants can assist in locating and stopping the worst offenders Participating sites include their own IP address information in the submitted data By aggregating attack data, the SANS ISC can provide strong evidence to ISPs willing to stop abuse coming from their networks Feedback is returned to participants when an attacker is successfully stopped

13 Where Did That Attack Come From?

14 SANS ISC Incident Database Data collected over the past few years are contained in a database that can be queried by any Internet user Some pre configured queries include Top ten offending sites Top ten targeted ports History of reported activity on any port Summary information about any user supplied IP address

15 Survival Time

16 Simple Blacklists Sites such as DShield.org compile global worst offender lists (GWOL) of the most prolific attack sources, and regularly post firewall parsable filters of these lists to help the Internet community fight back Another common practice is for a local network to create its own local worst offender list (LWOL) of those sites that have attacked it the most

17 Top Offending IP Addresses: A Simple GWOL

18 Improving the Blacklist LWOLs have the property of capturing repeat offenders that are indeed more likely to return to the local site in the future But do not react well to new encounters with previously unseen attackers GWOLs have the potential to inform a local network of highly prolific attackers And also have the potential to provide a subscriber with a list of addresses that may not yet have been encountered

19 Highly Predictive Blacklists Highly Predictive Blacklists (HPBs) represent a radically different approach to blacklist formulation HPBs are derived uniquely per DShield contributor, and rank each attacker in the blacklist based on an estimation of the probability that the attacker will visit the contributor s network in the future The HPB algorithm exploits a correlation relationship observed when compiling firewall logs from thousands of Internet contributors

20 Link Analysis HPBs employ a link analysis algorithm similar to Google s PageRank scheme used to find the most relevant web pages given a user s query Instead of web queries, the firewall logs of DShield contributors are cross compared with one another in search of overlaps among the attackers they report The attacker addresses included within an HPB are selected by favoring the inclusion of those attackers who have been encountered by other contributors who share degrees of overlap with the HPB owner

21 How Does It Work? Within the DShield repository, there are groups of networks that share various degrees of common attacker overlap, which can be called correlated victims To build an HPB for a network "A", we treat attack sources that have reportedly made attacks on networks correlated with "A" differently from attack sources that attacked the same number but uncorrelated networks The former are more probable to attack "A" and should be included in A's HPB with high priority Traditional blacklisting approaches such as GWOL treat these two attackers equally, therefore, ignore the characteristics of individual networks shown in the alert history

22 An Attack Table v 5 s 1 X X s 2 s 3 s 4 s 5 s 6 s 7 v 1 v 2 v 3 v 4 X X X X X X X X X X

23 Attack Correlation The rows represent attack sources and the columns represent the targeted networks (attack victims) An ``X'' in the table cell indicates that the corresponding source has reportedly attacked the corresponding network Consider s2 and s7 Although they have attacked the same number of victims, from the viewpoint of v1, one may say that s2 is more likely to attack than s7, because s2 has attacked v2, which shares more common attackers with v1

24 Attack Correlation (cont.) Compare the source s5 to s7 Both sources attacked only one network None of these networks share common attacks with v1 However, for v1, s5 and s7 are not equal Notice that v2 shares common attacks with v1, and v3 shares common attacks with v2 A path v3 v2 v1 connects s5 to v1 One may say that s5 is more likely to attack for v1.

25 HPB Summary The full HPB system implements many other practical considerations for blacklisting The source ranking and the probability estimation by random walk are the heart of the system Experiments show that the HPB exhibits a higher hit count than traditional blacklists for most of the contributors The experiments also show that HPB's performance is consistent over time, and these advantages remain stable across various list lengths and predict windows

26 An Invitation to Participate The SANS ISC s success is based on the active participation of thousands of users All Internet users, information analysis and sharing centers, and others willing to participate in a large distributed data collection and analysis project are invited to join Details about the SANS ISC are online at More information about HBLs are at ta.org/releases/hpb