CSCD 443/533 Advanced Networks

Transcription

1 CSCD 443/533 Advanced Networks Lecture 10 Usage and Network Measurement Spring 2016 Reading: See References at end 1

2 Topics Internet Usage Measurement overview Why measure? What to measure? Where to measure? Challenges of Measurement Measurement tools and Devolopers Active: ping, traceroute Passive: packet, and flow monitoring Useful Data Sets

3 Internet Use Over Time Mentioned before... Internet has changed over time last 40 years Began with evolved towards our use of it for entertainment and education Internet architects and network engineers have tried to quantify this change in usage

4 Why study network use patterns? Why do we need to know this?

5 Internet Use is Important How people use Internet Important for predicting potential performance issues ISP's and other providers must plan to satisfy demand Trends indicate that more and more traffic appears to have real-time characteristics Gaming, video, Skype and other VOIP technologies

6 Application Preferences Change Over Time

7 Another Graph of Applications

8 Itunes up to August 2008

9 Mobile App Revenue Billions $ Six Year Growth Rate is 850%

10 Video Explodes

11 Online Gaming Explodes

12 On-line Social Network Games Games played on Social Network sites Like Facebook Facebook most popular destination for online games, with 83% of respondents saying they have played games there 28% have purchased in-game currency with realworld money One hundred million people are playing these games and about $1 billion in revenue is expected this year

13 Some Interesting Statistics YouTube Statistics

14 YouTube Statistics YouTube Statistics Since its inception

15 Change in Internet Use 2008 to 2009

16 Internet Measurement

17 Why Measure the Internet The Internet is a man-made system, so why do we need to measure it?

18 Why Measure the Internet The Internet is a man-made system, so why do we need to measure it? Because we still don t really understand it Because sometimes things go wrong Measurement for network operations Detecting and diagnosing problems What-if analysis of future changes Measurement for scientific discovery Characterizing a complex system as organism Creating accurate models that represent reality Identifying new features and phenomena

19 Why Measure the Internet - Continued Measurement of Internet Will Help us to better understand why it works Help us to diagnose known problems Help us to design new features that the Internet should provide to enable next-generation application requirements Internet Measurements is key to the design of the next-generation Internet

20 What to Measure Take a few minutes to jot down How would you measure the Internet? What types of things would you measure? Where would you take measurements?

21 What Can be Measured Traffic Load statistics Packet or flow traces Performance of paths Application performance, e.g,. Web download time Transport performance, e.g., TCP bulk throughput Network performance, e.g., packet delay and loss Network structure Topology, and paths on the topology Dynamics of the routing protocols

22 Where to Measure Short answer Anywhere you can! End hosts Application logs, e.g., Web server logs Sending active probes to measure performance Individual links/routers Load statistics, packet traces, flow traces Configuration state Routing-protocol messages or table dumps Alarms

23 Challenges of Internet Measurement Given list of AS's, is there a built-in tool/function, which outputs the topology of the Internet? Given a path from source to destination, is there built-in a tool/function, which can determine how long a packet will take to travel to the destination? Given a set of routers along the path of a packet, is there a built-in tool/function, which can determine the delays introduced by each of the routers? The answer to all of these questions is NO!!!

24 Why don t we have such functions? Main reason, not designed for measurement Result of this is Poor Observability Reasons for this Core Simplicity Layered architecture Hidden Pieces Administrative Barriers

25 Core Simplicity Keep It Simple Stupid (KISS) Design Principle Stateless nature - connections/flows End-to-End argument Network elements do not track packets individually Interaction of traffic with network is hard to observe

26 Layered Architecture IP hourglass model hides details of lower level layers While this abstraction improving interoperability, it impedes detailed visibility of lower layers Hence, even detailed measurements such as packet capture cannot detect differences between two types of links

27 Hidden Pieces - Middleboxes Firewalls provide security Traffic Shapers assist in traffic management Proxies improve performance NAT boxes utilize IP address space efficiently Each of these impedes visibility of network components. E.g.: Firewalls may block active probing requests NATs hide away the no. of hosts and structure of the network on the other side

28 Administrative Barriers Owing to the competition-sensitive nature of the data required (topology, traffic etc.), ISPs actively seek to hide these details from outside discovery Information that they do provide are often simplified. E.g.: Instead of publishing router-level topologies, ISPs often publish PoP-level topologies

29 Measurement Tools

30 Measurement Tools Classification Active Measurement Passive Measurement Fused/Combined Measurement Bandwidth Measurement Latency Measurement Geolocation Others

31 Active Measurement - Ping Adding traffic for purposes of measurement Trade-offs between accuracy and overhead Need careful methods to avoid introducing bias Ping Host sends an ICMP ECHO packet to a target and captures the ICMP ECHO REPLY Useful for checking connectivity, and RTT Only requires control of one of the two end-points

32 Active Measurement - Ping Issues Many routers filter out ICMP packets RTT includes end system processing time Not accurate for network performance Was not designed for performance ICMP 32

33 Active Measurement - Traceroute Used to find forward path to a host Algorithm Send an IP datagram with TTL=1 First router sends back ICMP time exceeded Then send a datagram with TTL=2 Continue till destination is reached/ttl expired 33

34 Active Measurement - Traceroute Time-To-Live field in IP packet header Source sends a packet with a TTL of n Each router along the path decrements the TTL TTL exceeded sent when TTL reaches 0 Traceroute tool exploits this TTL behavior TTL=1 source TTL=2 Time exceeded destination Send packets with TTL=1, 2, 3, and record source of time exceeded message

35 Active Measurement Challenges of Traceroute Non-participating network elements Some routers and firewalls don t reply Inaccurate delay information Includes processing delays on the router CPU Round-trip vs. one-way measurements Paths may have asymmetric properties

36 Active Measurement - Challenges of Traceroute Measuring multiple paths Successive probes may traverse different paths Inferred path: A -> B -> Y Y: time exceeded Dest = D TTL = 2 A X Y D B: time exceeded Dest = D TTL = 1 B C

37 More Active Measurement Other Tools iperf netperf bing ttcp 37

38 Passive Measurement Passive Measurement Monitoring that uses equipment that taps into a network and does not interfere with the flow of network traffic Hardware Example: Network Tap Software Example: Traffic sniffer 38

39 Passive Measurement - Packets Traffic Sniffer Older program, tcpdump, for Unix-based hosts Wireshark now for many OS's Dedicated measurement systems DAGMON (up to 10GE) 39

40 Passive Measurement - Packets Filter for subset of packets IP addresses/prefixes (e.g., to/from specific Web sites, client machines, DNS servers, mail servers) Protocol (e.g., TCP, UDP, or ICMP) Port numbers (e.g., HTTP, DNS, BGP, Napster) Collect first n bytes of packet (snap length) IP header (typically 20 bytes) IP+UDP header (typically 28 bytes) IP+TCP header (typically 40 bytes) Application-layer message (entire packet)

41 Passive Measurement - IP Flows Aggregate Traffic into IP Flows For larger levels of network not enough to monitor individual packets, too many... Defined traffic as flows All industrial routers support flow data 41

42 Passive Measurement - IP Flow An IP flow is a unidirectional series of IP packets Given protocol (and port where applicable), Travelling between a source and destination Within a certain period of time

43 Aggregating Packets into IP Flows flow 1 flow 2 flow 3 flow 4 Set of packets that belong together Source/destination IP addresses and port numbers Same protocol Same input/output interfaces at a router (if known) Packets that are close together in time Maximum spacing between packets (e.g., 15 sec, 30 sec) Example: flows 2 and 4 are different flows due to time

44 Passive Measurement - IP Flow Look at Cisco's Netflow NetFlow creates a NetFlow cache entry contains information for all active flows NetFlow cache is built by processing first packet of flow through standard switching path A flow record is maintained within NetFlow cache for each active flow

45 Passive Measurement - IP Flow - Each flow record is identifies packets with similar flow characteristics and counts or tracks packets and bytes per flow - Flow details or cache information is exported to flow collector server(s) periodically based upon flow timers - The collector contains a history of flow information that was switched within Cisco device

46 CISCO Netflow Collector

47 NetFlow Data Characteristics What can you measure? Source and Destination addresses Input and Output interface numbers Source and Destination port numbers Layer 4 protocol Number of packets in the flow Total Bytes in the flow Time stamp in the flow

48 What can you do with Flow Data Aggregates traffic for monitoring Historical Traffic Flow Graphs of network utilization Graphs by protocol Graphs by IP address Tools have been developed around Netflow concept

49 Free Tools for Netflow Number of tools created to analyze Netflow data Many are free Most allow you to visualize traffic over time or by traffic type

50 FlowScan from U. Wisconsin FlowScan Developed by Dave Plonka University of Wisconsin Freely-available network traffic reporting and visualization tool Its development began in December 1998, and it was first released in March 2000 There are hundreds of users today, many campuses and ISPs FlowScan analyzes data exported by IP based routers

51 What does FlowScan do? FlowScan counts IP flows by protocol, application, user population, or Internet connection Protocols include TCP and UDP Applications include (SMTP), file sharing (P2P) User populations are subnets such as schools or departments. Internet connections are transit and peering links between Autonomous Systems

52 Interpreting Graphs Horizontal axis is time, current time to the right. Vertical axis indicates magnitude of measurement, usually in bits, packets, or flows per second Outbound traffic is upwards, Inbound traffic is downwards Colored bars show traffic classification and are stacked (not overlayed) to show the total

53 Interpreting FlowScan Graphs

54 Ethernet Flow Probe Workstation A Flow probe connected to switch port in traffic mirror mode Workstation B Campus Diagram by Mark Fullmer (author of flow-tools), 2002

55 Flow Data Useful for Anomalies Flow data useful in spotting traffic events Used heavily for mapping security incidents in addition to normal network events DoS events Popular software release

56 Outbound Distributed DoS flood from 30+ Campus Hosts ICMP DoS Traffic

57 The Same ICMP DDoS flood was also observed by FlowScan at another campus...

58 The Knight IRC Robot Coordinated via Internet Relay Chat (IRC) using "robots". Independent observations reported aggregates over 500Mbs The Same DDoS flood was also observed by FlowScan at other campuses...

59 Linux Release Events

60 Measurement Resources There are groups of academics/government trying to measure the Internet Have made data sets available to the public One notable group is CAIDA

61 What is CAIDA? Cooperative Association for Internet Data Analysis Goals include measuring and understanding the global Internet. Develop measurement and analysis tools Collect and provide Internet data: topology, header traces, routing, network security, DNS Visualization of the network

62 Walrus 62

63 Resources FlowScan: Argus: flow-tools: cflowd, CoralReef: tools/measurement/cflowd/ tools/measurement/coralreef/ CAIDA Tools Internet Fact Book (Really nice resource)

64 Measurements are anything but straightforward Internet Measurement is key to designing the next generation communication network Fundamental design principles of the current internet make it harder for measuring various aspects of it Preliminary research has resulted in a set of basic tools and methods to measure aspects like topology, traffic etc. Accuracy of such methods is still an open question There is still a lot of ground to cover in this direction and this is where researchers like you come into the equation!

65 Next time Discuss papers in class Next Gen Internet 65

66 1

67

68

69

70

71

72

73

74 Mobile App Revenue Billions $ Six Year Growth Rate is 850% 9

75

76

77

78

79

80

81 Internet Measurement 16

82

83

84

85 What to Measure Take a few minutes to jot down How would you measure the Internet? What types of things would you measure? Where would you take measurements? 20

86

87

88

89

90

91

92

93

94 Measurement Tools 29

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111 CISCO Netflow Collector 46

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130