NGN Progress Report. Table of Contents

Transcription

1 NGN Progress Report Title: Simulator Scalability Testing Prepared by: Richard Nelson Date: 08 February, 2006 Table of Contents Introduction...2 Simulators...2 Test Method...2 Simulation Model...2 CPU Utilisation...2 Memory Utilisation...3 Results...4 NS2...4 J-Sim...5 Omnet Comparisons...7 Limitations of the Tests...8 Conclusions...8 References...8 Page 1 1/06/2006

2 Introduction In anticipation that simulations of significant portions, or possibly all of Telecom NZ's network might be required, the scalability properties of simulators was investigated. Literature review showed that there has been little work done in this area [1] and so direct testing of suitable simulators was undertaken. Simulators The requirements for simulators to be selected were that: tools Source code be available so that they could be linked to memory allocation profiling The simulations be script able so that multiple runs could be automated The simulators be currently supported The simulators that met these criteria and were tested were NS2, J-Sim and Omnet++. Test Method Simulation Model A basic dumbell model was used with two routers connected by a single bottleneck link. Each router has an equal number of end nodes directly connected to it. In the simulations unidirectional TCP connections were formed from each node at one end of the dumbell to the corresponding node at the other end and greedy agents sent as much data as possible for the duration of the simulation run. To test performance scaling two sets of runs were performed, one with a varying number of nodes on each end and a fixed bandwidth on the bottleneck link. This kept the traffic approximately equal. The second set used a fixed number of nodes and links and varied the bandwidth of the bottleneck link. The parameters for the simulation runs were: Number of nodes on each end of the dumbell: fixed runs 200, variable runs Bandwidth of router - router link: fixed runs 100 Mb/s, variable runs 1, 10, 100, 1000, Mb/s Delay of router-router link: all runs 10ms Bandwidth of router end node links: all runs 10 Mb/s Delay of router end node links: all runs 1ms Router queue sizes: 250 packets Router queue discipline: Drop-tail Simulation time for each run: 10sec CPU Utilisation CPU utilisation was measured using the Linux time command. This provides system and user CPU time utilisation. The number of packets transmitted on the network was counted and the packets per CPU second calculated. Page 2 1/06/2006

3 To count the number of packets sent, an extra simulation was performed where a trace file detailing every simulation event across the bottleneck link was created. This file was then parsed, counting every 'enqueue' event at the client-side router, which means that a packet has passed the bottleneck successfully. The final count was then divided by the CPU time previously calculated to get packets per CPU second. Memory Utilisation Memory utilisation was measured with two heap profiling tools; one for Java and one for C. NS2's memory usage was profiled using a tool developed within WAND that replaced the standard C memory management functions, e.g. malloc, free, realloc, with custom functions that kept track of the current heap size and a list of individual allocations. Whenever new memory is allocated, a new allocation is added to the list and the current heap size is increased by the appropriate amount. An allocation is identified by the address of the first byte and the size. Whenever memory is deallocated, the original allocation is looked up in the list and the current heap size is decreased by the value of the size attribute. Approximately every second, the current heap size is written out to a file. At the conclusion of the simulation, the file is parsed to find the maximum heap usage which is used to construct the graphs below. J-sim's memory usage was profiled with SimpleProfiler a java based heap profiler ( Page 3 1/06/2006

4 Results NS2 Memory Utilisation CPU Utilisation Packets per CPU second Page 4 1/06/2006

5 J-Sim Memory Utilisation CPU Utilisation Packets per CPU second Page 5 1/06/2006

6 Omnet++ Memory Utilisation CPU Utilisation Packets per CPU second Page 6 1/06/2006

7 Comparisons CPU usage Memory Usage Packets per CPU second Page 7 1/06/2006

8 Limitations of the Tests Model limitations No starting and stopping of connections. No control or deliberate selection of the routing system used. Routing tables may occupy significant memory when large numbers of nodes are used. This will be dependent on the topology. Test tool limitations CPU utilisation is limited by Linux process accounting which (in common with most OS) is limited to jiffy resolution. Due to use of 1ms jiffy's on 2.6 series Linux kernels, this is not considered a significant limitation. Memory testing only checks heap allocation, not stack allocation. Counting of packets only takes place at a fixed location rather than across the entire topology as this was too difficult to do for some simulators. Conclusions The results show that J sim is very efficient on memory utilisation. This is due to the automatic garbage collection provided by Java. The price paid for this is the very high CPU usage to perform the garbage collection. As a result J-sim is reduced to a very small packet per second rate as the number of connections climbs towards Of the C-based simulators Omnet++ is significantly more memory efficient than NS-2 as the simulation size grows to 1000 connections. NS2 is more CPU efficient for small simulations, but not large. The differences appear to be due to the default routing algorithm used by NS2. It allocates an NxN table of routes with N being a power of 2. This provides a direct route lookup for every node in the simulation but requires significant amounts of memory as N grows large. The effect of this can be seen in the steps in the graphs of memory utilisation for N. Looking up larger tables requires more processing resulting in similar steps in the CPU usage and steps down in the packets per CPU second. Alternative routing algorithms are available in the NS2 source, but they are poorly documented and our efforts to get them working have failed. They should remove the steps from the graph, but the slope of the memory utilisation graph is higher for NS2 than Omnet++ so it will likely still use more memory. It may also use more CPU although this may scale better. References [1] Simulating Large Networks How Big is Big Enough? George F. Riley and Mostafa H. Ammar. In First International Conference on Grand Challenges for Modeling and Simulation, ICGCMS, January Page 8 1/06/2006