Traffic Flow Measurements within IP Networks: Requirements, Technologies and Standardization Jürgen Quittek NEC Europe Ltd., Network Laboratories, Heidelberg, Germany Tanya Szeby, Georg Carle, Sebastian Zander FhI FOKUS, Berlin, Germany
Outline Scope and general requirements Applications requiring detailed flow-based traffic measurements Requirements analysis Capabilities of existing technologies Standardization efforts at the IETF Network Laboratories, Heidelberg 2
Scope and General Requirements Goal: Find or develop a basic common IP Traffic Flow measurement technology to be available on (almost) all future routers Fulfilling requirements of many applications Low hardware/software costs Simple and scalable Metering to be integrated in general purpose IP routers and other devices (probes, middleboxes) Data processing to be integrated into various applications Interoperability by openness or standardization Network Laboratories, Heidelberg 3
Applications (1) Requiring Traffic Flow Measurement Usage-based accounting input to charging and billing various business model time-based, volume-based, QoS class-based per application, per user, per user group Traffic engineering optimizing network usage traffic analysis on congested links origin of traffic type of traffic dynamic behavior (bursty, adaptive, ) Traffic profiling Network Laboratories, Heidelberg 4
Applications (2) Requiring Traffic Flow Measurement QoS monitoring (passive) measurement of QoS properties validating Service Level Agreements Attack detection and analysis detecting (high volume) traffic patterns investigation of origin of attacks Intrusion detection detecting unexpected or illegal packets Network Laboratories, Heidelberg 5
Requirements (1) Distinguishing flows by 5-tuple IP addresses, transport type, port numbers Supporting MPLS, DiffServ Flexible aggregation of flows Metering Process Reliability Timestamps, time synchronization Flow timeouts Overload behavior sampling, simplifying, stopping Network Laboratories, Heidelberg 6
Requirements (2) Data Export Information model many header fields and statistics required Data model flexible, extensible anonymization? Data Transfer reliability security push and pull model reporting? regular reporting interval notification on specific events Configuration Network Laboratories, Heidelberg 7
Existing Technologies IETF standards RTFM RMON, RMON2 Proprietary technologies NetFlow (Cisco) sflow (InMon) LFAP (Riverstone) Crane (XACCT) Network Laboratories, Heidelberg 8
Real-Time Flow Measurement (RTFM) Very flexible and powerful meter Application programmable rule sets can serve several readers Manager programmable overload behavior Reader Reader polls meter Realization by SNMP Meter MIB Free software implementation Meter NeTraMet No acceptance at manufacturers Complicated to use (too powerful) Specified by RFCs 2720-2724 Network Laboratories, Heidelberg 9
Remote Network Monitoring MIB Very flexible and powerful Serves more general goals (analysis on layers 2-4) Just a monitoring tool, no measurement architecture defined Suited for very specific analysis tasks High (hardware) performance requirements Too complicated and too expensive for massive usage in routers Specified by RFCs 2021(RMON2), 2613, 2819(RMON), 2895, 2896, 3144 Network Laboratories, Heidelberg 10
NetFlow Proprietary by Cisco, but de-facto standard Fast and efficient, implemented for IOS Configurable measurement per 5-tuple Unreliable (measurement & data transport) Hardware-supported on some models Not well documented re-engineered by Juniper Versions 1-7 fixed data model Version 9 (under development) data model templates optional reliable transport Application Data collector Meter Router Network Laboratories, Heidelberg 11
sflow By InMon Corporation Includes metering and data transmission Probabilistic sampling at meter Packet sampling and counter sampling Timestamping by data collector Configuration by sflow MIB Poorly documented by informational RFC 3176 Not adapted yet by other vendors Application Data collector smon Meter Network Laboratories, Heidelberg 12
LFAP Light-weight Flow Accounting Protocol Application Proprietary by Riverstone (Cabletron) Just data transfer protocol FAS Meter at Connection Control Entity (CCE) communicates to Flow Accounting Server (FAS) Tight and reliable interaction CCE between CCE and FAS Reliable data transport Flexible TLV coding of transferred data Larger overhead than NetFlow More cost-intensive at meter/cce and at data collector/fas See <draft-riverstone-lfap-00.txt> Network Laboratories, Heidelberg 13
CRANE Common Reliable Accounting for Network Element (CRANE) Protocol Proprietary by XACCT Just data transfer protocol Template-based data model Focus on reliability Not yet in extensive commercial use See <draft-kzhang-crane-protocol-02.txt> Network Laboratories, Heidelberg 14
IETF IPFIX Working Group Current standardization effort at IETF: IP Flow Information export (IPFIX) working group Preparations 12/00 and 08/01, active since 10/01 Successor of RTFM Target (official): standardizing current practise Target (unofficial): standardizing NetFlow Planned documents Requirements RFC (almost completed) Architecture RFC (just starting) Data model RFC (not yet started) Protocol development not yet chartered, but protocol evaluation/selection Configuration of meter will not be standardized Network Laboratories, Heidelberg 15
IPFIX Architecture Overview Flow Information Export Application Exporter Probe (meter) Flow Record Collector PAYLOAD HEAD PAYLOAD HEAD PAYLOAD HEAD PAYLOAD HEAD PAYLOAD HEAD PAYLOAD HEAD PAYLOAD HEAD PAYLOAD HEAD Observation Point Network Laboratories, Heidelberg 16
IPFIX Flow Definition A flow is a set of packets passing an observation point in the network during a certain time interval. All packets belonging to a particular flow have a set of common properties derived from the data contained in the packet and from the packet treatment at the observation point. Rather general definition Not closely related to application-level flows Network Laboratories, Heidelberg 17
Many Open IPFIX Issues Support of bi-directional flow model? Reliability vs. costs vs. congestion-friendliness Overload behavior dynamic flow timeouts? dynamic flow measurement rules? dynamic sampling on/off? stop measuring? stop forwarding packets? Take any existing protocol as baseline for IPFIX? NetFlow, LFAP, CRANE? Network Laboratories, Heidelberg 18
IPFIX Outlook Good support from IESG High interest from equipment manufacturers Cisco intend(ed) to have NetFlow version 9 compliant to IPFIX standards Highly skilled design team approx. 15 people from Cisco, NEC, Riverstone, CAIDA, XACCT, Progress on schedule Requirements almost agreed Completion in planned in 2002 More information at http://ipfix.doit.wisc.edu Further help is very welcome! Please join us! Network Laboratories, Heidelberg 19
IETF PSAMP Working Group Establishment under discussion Focus on sampling and capturing packets and on transferring them to data collectors Target applications traffic profiling monitoring network behavior Closely related to IPFIX Preparation meeting planned for March Initial document <draft-duffield-framework-papame-00.txt> Network Laboratories, Heidelberg 20