CSCD 443/533 Advanced Networks

Transcription

1 CSCD 443/533 Advanced Networks Lecture 10 Usage and Network Measurement Winter 2017 Reading: See References at end 1

2 Topics Internet Usage Measurement overview Why measure? What to measure? Where to measure? Challenges of Measurement Measurement tools and Devolopers Active: ping, traceroute Passive: packet, and flow monitoring Useful Web Sites

3 Internet Use Over Time Mentioned before... Internet has changed over time last 40 years Began with evolved towards pretty much everything... entertainment, commerce, education, life Internet architects and network engineers have tried to quantify this change in usage Capture what changes mean to network management

4 Internet Use is Important How people use Internet Important for predicting potential performance issues ISP's and other providers must plan to satisfy demand Trends indicate that more and more traffic appears to have real-time characteristics Gaming, video, Skype and other VOIP technologies

5 Application Preferences Change Over Time

6 Another Graph of Applications

7 Itunes up to August 2008

8 Mobile App Revenue Billions $ Six Year Growth Rate is 850%

9 Video Explodes

10 Online Gaming Explodes

11 On-line Social Network Games Games played on Social Network sites Like Facebook Facebook most popular destination for online games, with 83% of respondents saying they have played games there 28% have purchased in-game currency with real-world money One hundred million people are playing these games and about $1 billion in revenue is expected this year

12 Some Interesting Statistics YouTube Statistics

13 YouTube Statistics YouTube Statistics Since its inception

14 Change in Internet Use 2008 to 2009

15 Internet Measurement

16 Why Measure the Internet The Internet is a man-made system, so why do we need to measure it?

17 Why Measure the Internet The Internet is a man-made system, so why do we need to measure it? Because we still don t really understand it Because sometimes things go wrong Measurement for network operations Detecting and diagnosing problems What-if analysis of future changes Measurement for scientific discovery Characterizing a complex system as organism Creating accurate models that represent reality Identifying new features and phenomena

18 Why Measure the Internet - Continued Measurement of Internet Will Help us to better understand why it works Help us to diagnose known problems Help us to design new features that the Internet should provide to enable next-generation application requirements Internet Measurements is key to the design of the next-generation Internet

19 What to Measure Take a few minutes to jot down How would you measure the Internet? What types of things would you measure? Where would you take measurements?

20 What Can be Measured Traffic Load statistics Packet or flow traces Performance of paths Application performance, e.g,. Web download time Transport performance, e.g., TCP bulk throughput Network performance, e.g., packet delay and loss Network structure Topology, and paths on the topology Dynamics of the routing protocols

21 Where to Measure Short answer Anywhere you can! End hosts Application logs, e.g., Web server logs Sending active probes to measure performance Individual links/routers Load statistics, packet traces, flow traces Configuration state Routing-protocol messages or table dumps Alarms

22 Challenges of Internet Measurement Given list of AS's, is there a built-in tool/function, which outputs the topology of the Internet? AS = Autonomous System Given a path from source to destination, is there built-in a tool/function, which can determine how long a packet will take to travel to the destination? Given a set of routers along the path of a packet, is there a built-in tool/function, which can determine the delays introduced by each of the routers? The answer to all of these questions is NO!!!

23 Why don t we have such functions? Main reason, not designed for measurement Result of this is Poor Observability Reasons for this Core Simplicity Layered architecture Hidden Pieces Administrative Barriers

24 Core Simplicity Keep It Simple Stupid (KISS) Design Principle Stateless nature - connections/flows End-to-End argument Network elements do not track packets individually Interaction of traffic with network is hard to observe

25 Layered Architecture IP hourglass model hides details of lower level layers While this improves interoperability, it impedes detailed visibility of lower layers Detailed measurements such as packet capture cannot detect differences between two types of links

26 Hidden Pieces - Middleboxes Firewalls provide security Traffic Shapers assist in traffic management Proxies improve performance NAT boxes utilize IP address space efficiently Each of these impedes visibility of network components. E.g.: Firewalls may block active probing requests Proxies hide real servers, shapers direct traffic behind scene NATs hide real number of hosts and structure of the network on the other side

27 Administrative Barriers Competition-sensitive nature of data, (topology, traffic etc.), ISPs actively seek to hide these details from outside discovery Information that they do provide often simplified E.g.: Instead of publishing router-level topologies, ISPs often publish PoP-level topologies

28 Measurement Tools

29 Measurement Tools Classification Active Measurement Passive Measurement Fused/Combined Measurement Bandwidth Measurement Latency Measurement Geolocation Others

30 Active Measurement - Ping Adding traffic for purposes of measurement Trade-offs between accuracy and overhead Need careful methods to avoid introducing bias Ping Host sends an ICMP ECHO packet to a target and captures the ICMP ECHO REPLY Useful for checking connectivity, and RTT Only requires control of one of the two end-points

31 Active Measurement - Ping Issues Many routers filter out ICMP packets RTT includes end system processing time Not accurate for network performance Was not designed for performance ICMP 31

32 Active Measurement - Traceroute Used to find forward path to a host via routers Algorithm Send an IP datagram with TTL=1 First router sends back ICMP time exceeded Then send a datagram with TTL=2 Continue till destination is reached/ttl expired 32

33 Active Measurement - Traceroute Time-To-Live field in IP packet header Source sends a packet with a TTL of n Each router along the path decrements the TTL TTL exceeded sent when TTL reaches 0 Traceroute tool exploits this TTL behavior TTL=1 source TTL=2 Time exceeded destination Send packets with TTL=1, 2, 3, and record source of time exceeded message

34 Active Measurement Challenges of Traceroute Non-participating network elements Some routers and firewalls don t reply Inaccurate delay information Includes processing delays on the router CPU Round-trip vs. one-way measurements Paths may have asymmetric properties

35 Active Measurement - Challenges of Traceroute Measuring multiple paths Successive probes may traverse different paths Inferred path: A -> B -> Y Y: time exceeded Dest = D TTL = 2 A X Y D B: time exceeded Dest = D TTL = 1 B C

36 More Active Measurement Other Tools iperf netperf bing ttcp 36

37 Passive Measurement Passive Measurement Monitoring uses equipment that taps into a network and does not interfere with flow of network traffic, just captures and observes it Hardware Example: Network Tap Software Example: Traffic sniffer 37

38 Passive Measurement - Packets Traffic Sniffer Older program, tcpdump, for Unix-based hosts Wireshark freely available for most OS's Dedicated measurement systems DAGMON - old 38

39 Passive Measurement - Packets Filter for subset of packets IP addresses/prefixes (e.g., to/from specific Web sites, client machines, DNS servers, mail servers) Protocol (e.g., TCP, UDP, or ICMP) Port numbers (e.g., HTTP, DNS, BGP, Napster) Collect first n bytes of packet (snap length) IP header (typically 20 bytes) IP+UDP header (typically 28 bytes) IP+TCP header (typically 40 bytes) Application-layer message (entire packet)

40 Passive Measurement - IP Flows Aggregate Traffic into IP Flows For larger levels of network not enough to monitor individual packets, too many... Defined traffic as flows All industrial routers support flow data 40

41 Passive Measurement - IP Flow An IP flow is a unidirectional series of IP packets Given protocol (and port where applicable), Travelling between a source and destination Within a certain period of time

42 Aggregating Packets into IP Flows flow 1 flow 2 flow 3 flow 4 Set of packets that belong together Source/destination IP addresses and port numbers Same protocol Same input/output interfaces at a router (if known) Packets that are close together in time Maximum spacing between packets (e.g., 15 sec, 30 sec) Example: flows 2 and 4 are different flows due to time

43 Passive Measurement - IP Flow Look at Cisco's Netflow NetFlow creates a NetFlow cache entry contains information for all active flows NetFlow cache is built by processing first packet of flow through standard switching path A flow record is maintained within NetFlow cache for each active flow

44 Passive Measurement - IP Flow - Each flow record has packets with similar flow characteristics and counts or tracks packets and bytes per flow - Flow details or cache information is exported to flow collector server(s) periodically based upon flow timers - The collector contains a history of flow information that was switched within Cisco device

45 CISCO Netflow Collector

46 NetFlow Data Characteristics What can you measure? Source and Destination addresses Input and Output interface numbers Source and Destination port numbers Layer 4 protocol Number of packets in the flow Total Bytes in the flow Time stamp in the flow

47 What can you do with Flow Data Aggregates traffic for monitoring Historical Traffic Flow Graphs of network utilization Graphs by protocol Graphs by IP address Tools have been developed around Netflow concept

48 Free Tools for Netflow Number of tools created to analyze Netflow data Many are free Most allow you to visualize traffic over time or by traffic type

49 FlowScan from U. Wisconsin FlowScan Developed by Dave Plonka University of Wisconsin Freely-available network traffic reporting and visualization tool Its development began in December 1998, and it was first released in March 2000 There are hundreds of users today, many campuses and ISPs FlowScan analyzes data exported by IP based routers

50 What does FlowScan do? FlowScan counts IP flows by protocol, application, user population, or Internet connection Protocols include TCP and UDP Applications include (SMTP), file sharing (P2P) User populations are subnets such as schools or departments. Internet connections are transit and peering links between Autonomous Systems

51 Interpreting Graphs Horizontal axis is time, current time to the right. Vertical axis indicates magnitude of measurement, usually in bits, packets, or flows per second Outbound traffic is upwards, Inbound traffic is downwards Colored bars show traffic classification and are stacked (not overlayed) to show the total

52 Interpreting FlowScan Graphs

53 Ethernet Flow Probe Workstation A Flow probe connected to switch port in traffic mirror mode Workstation B Campus Diagram by Mark Fullmer (author of flow-tools), 2002

54 Flow Data Useful for Anomalies Flow data useful in spotting traffic events Used heavily for mapping security incidents in addition to normal network events DoS events Popular software release

55 Outbound Distributed DoS flood from 30+ Campus Hosts ICMP DoS Traffic

56 The Same ICMP DDoS flood was also observed by FlowScan at another campus...

57 The Knight IRC Robot Coordinated via Internet Relay Chat (IRC) using "robots". Independent observations reported aggregates over 500Mbs The Same DDoS flood was also observed by FlowScan at other campuses...

58 Linux Release Events

59 Measurement Resources There are groups of academics/government trying to measure the Internet Have made data sets available to the public One notable group is CAIDA Center for Applied Internet Data Analysis Mission Center for Applied Internet Data Analysis (CAIDA) is a collaborative undertaking among organizations in commercial, government, and research sectors aimed at promoting greater cooperation in engineering and maintenance of robust, scalable global Internet infrastructure

60 What is CAIDA? Goals include measuring and understanding the global Internet. Develop measurement and analysis tools Collect and provide Internet data: Topology, header traces, routing, network security, DNS Visualization of the network Projects CAIDA deploys and maintains a globally distributed measurement platform we call Archipelago (Ark) Archipelago consists of distributed hardware measurement nodes

61 CAIDA Arc Based Projects Internet Topology Discovery: Using multiple teams of geographically distributed Ark monitors, dynamically and strategically divide probing work among teams to conduct coordinated, large-scale traceroute-based topology measurements. Congestion: Connectivity and Congestion, running measurements to detect congestion on interdomain links of the networks hosting the Ark monitors. These measurements will inform analysis of traffic congestion dynamics induced by evolving interconnection and traffic management practices of CDNs and ISPs.

62 CAIDA Visualization Research Plotting the positions of Autonomous System (AS) nodes in polar coordinates allows a visualization of the dimensions of peering richness and geographic information, revealing the most highly connected Autonomous Systems Walrus tool uses 3D hyperbolic geometry to display graphs under a fisheye-like distortion, providing an interactive visualization of large directed graps in threedimensional space.

63 Walrus 63

64 Regional Internet Registry for Europe RIPE NCC Regional Internet Registry for Europe, Middle East and parts of Central Asia. As such, allocate and register blocks of Internet number resources to Internet service providers (ISPs) and other organizations. Performs a great deal of statistics gathering and has created tools and visualizations that assist their users and the Internet community in general. One major Internet mapping effort is Atlas.

65 Regional Internet Registry for Europe RIPE NCC RIPE Atlas is global network of probes that measure Internet connectivity and reachability, providing an unprecedented understanding of the state of Internet in real time Thousands of active probes in RIPE Atlas network... Atlas probes are small, USBpowered hardware devices that hosts attach to an Ethernet port on their router Go to url below to see kinds of things RIPE researchers are looking at

66 Measurement Lab (M-Lab) M-Lab is a consortium of research, industry and publicinterest partners dedicated to: Providing an open, verifiable measurement platform for global network performance Hosting largest open Internet performance dataset on the planet Creating visualizations and tools to help people make sense of Internet performance Measurement Lab (M-Lab) is the largest open source Internet measurement effort in the world. M-Lab provides performance tests that help consumers develop an accurate picture of their Internet service by offering a state-of-the-art server platform that supports many measurement needs.

67 Stanford List of Network Tools SLAC List of tools used for Network (both LAN and WAN) Monitoring Audience is mainly network administrators. You are welcome to provide links to this web page. Please do not make a copy of this web page and place it at your web site since it will quickly be out of date. Network Monitoring Tools grew out of a list that were reported to be in use at 11 ESnet sites in a survey made by the ESnet Network Monitoring Task Force (NMTF) and completed in October 1995

68 Internet Measurement Workshops and Conferences AIMS workshops are held to discuss the current and future state of active Internet measurement and analysis. An evolution of the Internet Statistics and Metrics Analysis (ISMA) workshop series, the intent of these workshops are to facilitate discussion among communities of academia, equipment vendors, and service providers, who share an interest in and incentive to understand one another's interests and concerns with Internet statistics and analysis. Network Traffic Measurement and Analysis Conference Network Traffic Measurement and Analysis Conference (TMA) highly selective venue for the presentation of both early-stage and mature research as well as controversial works on all the aspects of network measurements. The focus is on improving the practice or application of measurements, across the entire network stack up to the application layer htttp://tma.ifip.org/

69 More Resources FlowScan: Argus: flow-tools: CAIDA Tools Internet Fact Book (Really nice resource)

70 Network Measurement is anything but straightforward Internet Measurement is key to designing the next generation communication network Fundamental design principles of the current internet make it harder for measuring various aspects of it Preliminary research has resulted in a set of basic tools and methods to measure aspects like topology, traffic etc. Accuracy of such methods is still an open question There is long way to go to have solid Internet measurement!!!

71 Next time Discuss papers in class 71

72 1

73

74

75

76

77

78

79 Mobile App Revenue Billions $ Six Year Growth Rate is 850% 8

80

81

82

83

84

85

86 Internet Measurement 15

87

88

89

90 What to Measure Take a few minutes to jot down How would you measure the Internet? What types of things would you measure? Where would you take measurements? 19

91

92

93

94

95

96

97

98

99 Measurement Tools 28

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116 CISCO Netflow Collector 45

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132 CAIDA Arc Based Projects Internet Topology Discovery: Using multiple teams of geographically distributed Ark monitors, dynamically and strategically divide probing work among teams to conduct coordinated, large-scale traceroute-based topology measurements. Congestion: Connectivity and Congestion, running measurements to detect congestion on interdomain links of the networks hosting the Ark monitors. These measurements will inform analysis of traffic congestion dynamics induced by evolving interconnection and traffic management practices of CDNs and ISPs. 61

133 CAIDA Visualization Research Plotting the positions of Autonomous System (AS) nodes in polar coordinates allows a visualization of the dimensions of peering richness and geographic information, revealing the most highly connected Autonomous Systems Walrus tool uses 3D hyperbolic geometry to display graphs under a fisheye-like distortion, providing an interactive visualization of large directed graps in threedimensional space. 62

134

135 Regional Internet Registry for Europe RIPE NCC Regional Internet Registry for Europe, Middle East and parts of Central Asia. As such, allocate and register blocks of Internet number resources to Internet service providers (ISPs) and other organizations. Performs a great deal of statistics gathering and has created tools and visualizations that assist their users and the Internet community in general. One major Internet mapping effort is Atlas. 64

136 Regional Internet Registry for Europe RIPE NCC RIPE Atlas is global network of probes that measure Internet connectivity and reachability, providing an unprecedented understanding of the state of Internet in real time Thousands of active probes in RIPE Atlas network... Atlas probes are small, USBpowered hardware devices that hosts attach to an Ethernet port on their router Go to url below to see kinds of things RIPE researchers are looking at 65

137 Measurement Lab (M-Lab) M-Lab is a consortium of research, industry and publicinterest partners dedicated to: Providing an open, verifiable measurement platform for global network performance Hosting largest open Internet performance dataset on the planet Creating visualizations and tools to help people make sense of Internet performance Measurement Lab (M-Lab) is the largest open source Internet measurement effort in the world. M-Lab provides performance tests that help consumers develop an accurate picture of their Internet service by offering a state-of-the-art server platform that supports many measurement needs. 66

138 Stanford List of Network Tools SLAC List of tools used for Network (both LAN and WAN) Monitoring Audience is mainly network administrators. You are welcome to provide links to this web page. Please do not make a copy of this web page and place it at your web site since it will quickly be out of date. Network Monitoring Tools grew out of a list that were reported to be in use at 11 ESnet sites in a survey made by the ESnet Network Monitoring Task Force (NMTF) and completed in October

139 Internet Measurement Workshops and Conferences AIMS workshops are held to discuss the current and future state of active Internet measurement and analysis. An evolution of the Internet Statistics and Metrics Analysis (ISMA) workshop series, the intent of these workshops are to facilitate discussion among communities of academia, equipment vendors, and service providers, who share an interest in and incentive to understand one another's interests and concerns with Internet statistics and analysis. Network Traffic Measurement and Analysis Conference Network Traffic Measurement and Analysis Conference (TMA) highly selective venue for the presentation of both early-stage and mature research as well as controversial works on all the aspects of network measurements. The focus is on improving the practice or application of measurements, across the entire network stack up to the application layer htttp://tma.ifip.org/ 68

140

141

142