Introduction to NS-2 Part 6. Confidence Interval Min Chen School of Computer Science and Engineering Seoul National University 1
Outline Definitions Normal Distribution Confidence Interval Central Limit Theorem Error-Bar An Example 2
Definitions Population The set of elements we are inteseted in Sample A subset of the Population Population Mean μ Always stands for the everage value Sample Mean X = X i N The Mean for the Sample subset 3
Definitions (2) Variance X 2 = i 2 N Sample Variance X S 2 = i X 2 N Standard Deviation : = 2 Sample Standard Deviation : S=S 2 4
Normal Distribution A continuous probability distribution that often gives a good descriptio of data that cluster around the mean For X ~N, 2 Probability Density Function(PDF) x= 1 x 2 2 e 2 2 2 Cumulative Distribution Function(CDF) x= 1 t 2 2 e 2 dt 5
Normal Distribution : PDF 6
Normal Distribution : CDF 7
Confidence Interval Confidence Interval (CI) is a particular kind of interval estimate of a population parameter. Instead of estimating the reliability of a single value, an interval likely to include the parameter is given. Thus, CIs are used to indicate the reliability of an estimate. If Probability{c1<=µ<=c2} = 1-α Then we call (c1,c2) Confidence Interval 1-α is the Confidence Coefficient 8
Central Limit Theorem Central Limit Theorem (CLT) states conditions under which the mean of a sufficiently large number of independent random variables, each with finite mean and variance, will be approximatedly normally distributed. This distribution is X ~N, M Where M is the Sample Size 9
Central Limit Theorem According to CLT, we can calculate the Confidence Interval by a given Confidence Coefficient Define Z value: Frequently used Z: Z(90%) = 1.645 Z(95%) = 1.96 Z(99%) = 2.33 X Z = n 10
Calculate Confidence Interval With the value of Z, the confidence interval can be calculated as follow: X Z S M, X Z S M Where X denotes mean, S denotes sample variance and M denotes the sample size 11
Error-Bar Error Bar is used on graphs to indicate the error, or uncertainty in a reported measurement. In simulation result figures, error bar represents the confidence of the simulation The error bar in the figure are the variance of confidence intervals for each mean value in the simulation work Z S M 12
Example: Network Topology TCP: TCP: Throughput Throughput in in 5s 5s ~ ~ 10s 10s Random: [3,4] s n1 1Mb 1Mb 10ms 10ms n6 0.0 s n2 n4 n5 n7 0.0 s n3 Each simulation will be repeated for 30 times with different seeds n8 UDP: UDP: Controlable Controlable Data Data Rate Rate 100 100 ~ 100 ~ 100 500 500 ~ ~ step 500 step 500 100 100 10Mb 10Mb 1ms 1ms 13
OTCL Script if {$argc!= 2} { puts "Usage: ns errorbar.tcl rate_ no_" exit } # Simulation parameters setup set par1 [lindex $argv 0] ;# date rate for the interference flow set par2 [lindex $argv 1] ;# the number of the experiments set val(stop) 12.0 ;# time of simulation end # Initialization #Create a ns simulator set ns [new Simulator] #Open the NS trace file set tracefile [open out$par1-$par2.tr w] $ns trace-all $tracefile #Open the NAM trace file set namfile [open out.nam w] $ns namtrace-all $namfile 14
OTCL Script (2) # Nodes Definition #Create 8 nodes set n0 [$ns node] set n1 [$ns node]... set n7 [$ns node] # Links Definition #Createlinks between nodes $ns duplex-link $n0 $n3 10.0Mb 1ms DropTail $ns queue-limit $n0 $n3 50... $ns duplex-link $n3 $n4 1.0Mb 10ms DropTail $ns queue-limit $n3 $n4 50 #Give node position (for NAM) $ns duplex-link-op $n0 $n3 orient right-down... $ns duplex-link-op $n4 $n7 orient right-down $ns duplex-link-op $n3 $n4 orient right 15
OTCL Script (3) # Agents Definition #Setup a TCP connection set tcp0 [new Agent/TCP] $ns attach-agent $n0 $tcp0 set sink3 [new Agent/TCPSink] $ns attach-agent $n5 $sink3 $ns connect $tcp0 $sink3 #Setup a TCP connection set tcp1 [new Agent/TCP] $ns attach-agent $n1 $tcp1 set sink4 [new Agent/TCPSink] $ns attach-agent $n6 $sink4 $ns connect $tcp1 $sink4 #Setup a UDP connection set udp [new Agent/UDP] $ns attach-agent $n2 $udp set null [new Agent/Null] $ns attach-agent $n7 $null $ns connect $udp $null 16
OTCL Script (4) # Applications Definition #Setup a FTP Application over TCP connection set ftp0 [new Application/FTP] $ftp0 attach-agent $tcp0 $ftp0 set type_ FTP set ftp1 [new Application/FTP] $ftp1 attach-agent $tcp1 $ftp1 set type_ FTP set traffic [new Application/Traffic/Exponential] $traffic set packetsize_ 1000 $traffic set burst_time_ 0.5 $traffic set idle_time_ 0 $traffic set rate_ [expr $par1*1000] $traffic attach-agent $udp 17
OTCL Script (5) # Traffic Running Time set rng [new RNG] $rng seed 0 set RVstart [new RandomVariable/Uniform] $RVstart set min_ 3 $RVstart set max_ 4 $RVstart use-rng $rng set startt [expr [$RVstart value]] puts "startt $startt sec" $ns at 0.0 "$ftp1 start" $ns at 0.0 "$traffic start" $ns at $startt "$ftp0 start" $ns at 11.0 "$ftp0 stop" $ns at 11.5 "$ftp1 stop" $ns at 11.5 "$traffic stop" 18
OTCL Script (6) # Termination #Define a 'finish' procedure proc finish {} { global ns tracefile namfile $ns flush-trace close $tracefile close $namfile # exec nam out.nam & exit 0 } $ns at $val(stop) "$ns nam-end-wireless $val(stop)" $ns at $val(stop) "finish" $ns at $val(stop) "puts \"done\" ; $ns halt" $ns run 19
AWK for Throughput BEGIN { Init = 0; startt = 0; endt = 0; } { Action = $1; time= $2; From = $3; to = $4; Type = $5; pktsize = $6; flow_id = $8; src = $9; dst = $10; seq_no = $11; packet_id = $12; if ( action == "r" && type == "tcp" && time>=5.0 $$ time <=10.0 && from == 4 && to == 5 ) { if(init == 0) { start_time = time; Init =1; } pkt_byte_sum += pktsize; endt = time; } } END{ time=endt-startt; throughput=pkt_byte_sum*8/time/1000; printf("%f\n",throughput); } 20
AWK for ErrorBar with 95% CI BEGIN { FS="\t"; } { } ln++; throughput[ln]=$1; sum=sum+throughput[ln]; END{ average=sum/ln; for(i=1;i<=ln;i++) { gap=throughput[ln]-average; sum2=sum2+gap*gap; } } variance=sqrt(sum2/(ln-1)); confidence_interval=1.96*variance/sqrt(ln); printf("%f %f %f\n",t,average,confidence_interval); 21
The result file The use of parameters awk -v t=100 -f errorbar.awk result100 >> errorbar.txt Three columns 1 : the data rate 2 : the mean throughput 3 : the confidence interval variance 22
GPL script set title "Impact of Data Rate on the Throughput" set xlabel "On-Off Flow Rate Setting (kbps)" set ylabel "Flow Throughput (kbps)" set xrange [0:600] set terminal png set output "throughput.png" plot "errorbar.txt" using 1:2:3 with errorbars 1, "errorbar.txt" with lines 1 23
Perl Script in Linux for($i=100; $i<=500; $i=$i+100) { for($j=1; $j<=30; $j++) { system("ns errorbar.tcl $i $j"); $f1="out$i-$j.tr"; $f2="result$i"; system("awk -f throughput.awk $f1 >> $f2"); print "\n"; } print "\n"; } for($i=100; $i<=500; $i=$i+100) { $f1="result$i"; $f2="errorbar.txt"; system("awk -v t=$i -f errorbar.awk $f1 >> $f2"); print "\n"; } system("gnuplot errorbar.gpl"); 24
To run perl script in Linux Grant the privilege to the script file chmod 777 errorbar.pl Run the script perl errorbar.pl 25
Results: 26