CINBAD. CERN/HP ProCurve Joint Project on Networking. Post-C5 meeting, 12 June 2009 (hepix, 26 May 2009)

Transcription

1 CINBAD CERN/HP ProCurve Joint Project on Networking Post-C meeting, 12 June 2009 (hepix, 26 May 2009) Ryszard Erazm Jurga - CERN Milosz Marian Hulboj - CERN

2 Outline Introduction to CERN network CINBAD project and its goals Data - sources, collection and analysis Results and conclusions 2

3 Simplified overall CERN campus network topology Computer Center Vault Computer Center Vault Farms 230 x BL3 BL RL BL4 RL4 RL3 BL2 RL2 BL1 RL1 Numbers: PL1 SL1 BL7 RL7 BB16 RB16 ATCN 40x 0x BB1 AB1 RB1 ZB1 BB2 AB2 ZB2 RB2 BB3 AB3 RB3 ZB3 ZB4 RB4 2-S S2 376-R 13-C 874-R BB4 AB4 13-CCR 13-CCR 100x or 874-CCC x active user devices Minor Starpoints 100x CERN sites 1Gb user ports Gb routers Gb ports 887-R switches ZB0 RB0 BB0 140 Gbps WAN connectivity 4.8 Tbps LCG Core 1x 1x TN Control AB0 BB17 RB17 ZB RB BB20 RB20 BB AB 1x ZB6 RB6 BB6 AB6 1x TN PG1 SG1 IB1 PG2 SG2 IB2 Control Monitoring BT1 RT16 Meyrin area BT2 BT3 BT RT3 RT4 RT14 RT13 LHC area 10x x BT12 433x Internet BT4 RT BT RT BT6 RT7 BT7 RT8 BT8 RT9 BT9 RT10 BT10 RT11 BT11 RT12 6 TT1 TT2 TT3 TT4 TT TT6 TT x 227 TN TT8 Control 10x Prevessin area BT1 BT RT17 RT x 287 TN CDR CDR CDR 8x CDR HLT 90x DAQ 2x DAQ 90x DAQ DAQ 3

4 CINBAD codename deciphered CERN Investigation of Network Behaviour and Anomaly Detection Project Goal To understand the behaviour of large computer networks ( nodes) in High Performance Computing or large Campus installations to be able to: Detect traffic anomalies in the system Be able to perform trend analysis Automatically take counter measures Provide post-mortem analysis facilities

5 What is anomaly? (1) Anomalies are a fact in computer networks Anomaly definition is very domain specific: Network faults Malicious attacks Viruses/worms Misconfiguration But there is a common denominator: Anomaly is a deviation of the system from the normal (expected) behaviour (baseline) Normal behaviour (baseline) is not stationary and is not always easy to define Anomalies are not necessarily easy to detect

6 What is anomaly? (2) Just a few examples of anomalies: The network infrastructure misuse unauthorised DHCP/DNS server (either malicious or accidental) network scans worms and viruses Violation of a local network/security policy NAT, TOR usage (not allowed at CERN) 6

7 data sources CINBAD project principle storage analysis collectors

8 sflow main data source Based on packet sampling (RFC 3176) on average 1-out-of-N packet is sampled by an agent and sent to a collector packet header and payload included (max 128 bytes) switching/routing/transport protocol information application protocol data (e.g. http, dns) SNMP counters included low CPU/memory requirements scalable For more details, see our technical report 8

9 sflow data usage Typically: traffic accounting (e.g. for billing, network planning, SLA etc.) Useful for post-mortem network analysis CERN-wide network tcpdump access to the whole CERN sampled network traffic information where and when the traffic has been seen particularly useful in detecting packets that should not be there (policy violation) Rare examples of sflow usage for anomaly detection 9

10 Other data sources Packet sampling data is not enough! Data is partial, cannot provide 100% accuracy Not always easy to identify the anomaly More data to understand flow of data in the network External sources provide useful information and time triggers Correlation between various data sources Example: Central Antivirus Service at CERN 10

11 CINBAD sflow data collection Current collection based on traffic from ~1000 switches ~6000 sampled packets per second ~ 300 snmp counter sets per second

12 Multistage data storage Stage 1 sflow datagram tree-like format unpacked into CINBAD file format to enable fast direct access Minimal space overhead introduced Stage 2 Oracle DB as a long-term storage data aggregation tradeoff between sflow randomness and space, data lifetime and anomaly detection requirements e.g. number of destination IPs for a given source IP 12

13 Data analysis Various approaches are being investigated Statistical analysis methods detect a change from normal network behaviour selection of suitable metrics is needed can detect new, unknown anomalies poor anomaly type identification Signature based we ported SNORT to work with sampled data performs well against know problems tends to have low false positive rate does not work against unknown anomalies 13

14 Synergy from both detection techniques Translation Rules Incoming sflow stream Sample-based SNORT evaluation engine Anomaly Alerts Statistical Analysis engine New baselines New Signatures Traffic Profiles 14

15 Campus and Internet traffic analysis Current results Identified anomalies which went undetected by the central CERN IDS Detected number of misbehaviors both statistical and pattern matching approaches used TOR, DNS abuse, Trojans, worms, network scans, p2p applications, rogue DHCP servers, etc. Findings reported to the CERN security team Security team adapted their policies 1

16 Achievements and next steps It has been demonstrated that pattern matching for anomaly detection is possible with sflow data Payload data is a key advantage of sflow sflow allows distributed detection at the edge of the network Combining statistical analysis with pattern matching is providing encouraging initial results Integration of the two mechanisms holds promise for zero-day detection 16