ELECTRONIC COPY SAMKNOWS ANALYSIS OF ROGERS BROADBAND PERFORMANCE IN FEBRUARY 2015 ELECTRONIC COPY. Delivered by to: Shane Jansen.

Transcription

1 ELECTRONIC COPY SAMKNOWS ANALYSIS OF ROGERS BROADBAND PERFORMANCE IN FEBRUARY 2015 Delivered by to: Shane Jansen Rogers Dated: February 25, 2015 ELECTRONIC COPY

2 [THIS PAGE LEFT INTENTIONALLY BLANK] SamKnows 2

3 Contents A EXECUTIVE SUMMARY 5 B TESTING METHODOLOGY 6 B.1 Recruitment 6 B.2 Sample Composition 8 B.3 Test Configuration and Schedule 9 B.4 Data Analysis 11 C KEY PERFORMANCE INDICATORS 12 C.1 Download Throughput 12 C.2 Upload Throughput 15 C.3 Latency 18 C.4 Packet Loss 19 C.5 US providers vs Rogers comparison 20 SamKnows 3

4 About This Document This document is a summary report, which has been commissioned by Rogers and completed by SamKnows. For the purposes of this study 440 households across Rogers s customer base were given a specially configured hardware device (SamKnows Whitebox), which runs a series of purpose-built tests to measure Internet performance. The analysis in this report is carried out on data collected over the date period of 20 th January to 20 th February Any comments on the analysis in this document should be directed to Roxanne Robinson (Roxanne@samknows.com). SamKnows 4

5 A Executive Summary This report presents the findings of a project commissioned by Rogers to study broadband performance across their network. This study was conducted during 20 th January to 20 th February 2015 using a sample of 440 fixed measurement devices ( Whiteboxes ) deployed to Rogers customers. SamKnows has been engaged with other regulators and broadband providers around the world and the methodology used has been developed to meet the needs of both regulators and operators to enable them to better understand and characterize their networks. Tests were performed in a similar fashion to the FCC s Measuring Broadband America project, gathering statistics on upload and download speeds. This report focuses on both the upload and the download tests specifically. These were run every other hour each day for a duration of 30 seconds. SamKnows created the panel using standard recruitment methodology. Potential volunteers were contacted via an campaign run by Rogers. Recipients of this were then directed to a SamKnows operated website where they could register for the project. Following specific stages SamKnows then selected the final panel of participants who could receive a Whitebox. The study found: - All Rogers current in market packages exceed 100% of advertised download and upload speeds based on 24/7 results. - Speedboost pushes download speeds far beyond the advertised speed for the majority of packages based on 24/7 results, particularly those with lower advertised rates. - Rogers peak-hour download average of 108% of advertised speed exceeds the US average of 101% and US cable operator average of 103%. SamKnows 5

6 B Testing Methodology B.1 Recruitment For the purposes of this study, SamKnows sought to recruit 440 panellists using the following method: 1) Potential volunteers were contacted via an campaign run by Rogers. This was targeted at all their customers on current in market service packages which were representative of the overall subscriber base. Following this SamKnows assumed full control over volunteer selection. 2) Recipients of this were directed to a SamKnows-operated website to register for the project. This involved completing an online form, requesting information about which Rogers package they were subscribed to, and where they resided. 3) SamKnows reviewed the demographics of this initial panel to ensure a representative sample and to identify any deficiencies or shortfalls that would prevent us achieving the goals of the project. 4) A sub-set of the volunteers were then selected based on the statistical requirements of the panel. Selected volunteers were then asked to complete an acknowledgment of User Terms and Conditions that outlined the permissions to be granted by the volunteer in key areas such as privacy. 5) Of those volunteers that completed the User Terms and Conditions, SamKnows selected the final panel of participants, each of whom received a Whitebox for self-installation. SamKnows arranged for dispatch of the Whitebox, and SamKnows provided full support during the Whitebox installation phase. 6) Panellists were asked to install the Whitebox per the provided instructions and leave it connected for the duration of the study. SamKnows 6

7 NETWORK SCHEMATIC The installation of the SamKnows Whitebox is as follows: Option A: Option B: To home computers To home computers SamKnows Whitebox SamKnows Whitebox Personal router SamKnows 7

8 B.2 Sample Composition 440 Rogers customers were provided with a Whitebox, which they were instructed to leave powered on and connected to their Internet Modem/Gateway. 151 panellists were excluded for the following reasons: 1) The Whitebox hadn t run tests for at least 5 days in the month (due to it not being turned on, or the panellist was actively using the broadband connection continuously) 2) The panellist s service package could not be reliably determined 3) The results were highly erratic and/or intermittent 4) On a legacy Rogers speed package. 5) As this report focuses on new products, most of the excluded panellists belong to older products not suitable for the purpose of this study. These exclusions were made to ensure the integrity of the panel and ensure each speed package was statistically representative. As a result the report is based on 289 panellists. SamKnows 8

9 The table below depicts the breakdown of the sample: ISP Package (Download x Upload) Sample Size Rogers 15Mbps x 1Mbps 12 Rogers 30Mbps x 5Mbps 48 Rogers 60Mbps x 10Mbps 93 Rogers 70Mbps x 10Mbps 41 Rogers 100Mbps x 10Mbps 46 Rogers 250Mbps x 20Mbps 49 TOTAL 289 Table 1: Sample size split by Download Throughput service package SamKnows typically recommends a minimum of anything between 40 and 45 samples per service package being reported upon, as this is what is considered statistically significant based on research carried out by regulators around the world. This is also the minimum sample size used by the FCC in their Measuring Broadband America program. The 15Mbps plan therefore does not have a sufficient representative sample size to be statistically significant, although it remains included in this study. B.3 Test Configuration and Schedule All measurements for download and upload were conducted on a series of offnet test nodes. The same metrics, test parameters and testing schedule as the FCC s Measuring Broadband America project were used. A detailed description of the methodology underpinning each metric can be found at Four servers were used in total and they were hosted at a data centre in Toronto. SamKnows 9

10 A summary of the speed related parameters and measurement frequency for each metric is detailed below: Test Schedule (Fixed) for Rogers Test Target(s) Frequency Duration Download speed 1 off-net test node Every other hour, 24/7 Fixed 30 seconds*** Upload speed 1 off-net test node Every other hour, 24x7 Fixed 30 seconds*** Latency/Loss 1 off-net test node Every hour, 24x7 Video Streaming 1 off-net test node Every hour, 24x7 Web Browsing speed 1 off-net test node Every hour, 24x7 DNS response time 1 off-net test node Every hour, 24x7 Jitter 1 off-net test node Every hour, 24x7 *** The speed tests run for a fixed duration of 30 seconds, but output the cumulative average transfer speed every 5 seconds (i.e. the first average is for seconds 0-5, the second is for 0-10, etc.). This allows us to measure performance over multiple durations, whilst running only one test. Below is also an outline of the number of tests carried out for each of the major metrics along with the volume of data used for these tests: Test Number of Tests Volume of data Consumed Download speed 845, TB Upload speed 830, TB Latency/Loss 277, GB Web Browsing speed 2,210, TB SamKnows 10

11 B.4 Data Analysis A large raw dataset was collected over the period 20 th January to 20 th February The following process describes how the data was analysed to produce the charts and tables in this report: 1. Failed measurements and significant outliers were identified and removed from the raw dataset. A significant outlier is one where there has been a clear error and failed measurements occur when a test fails (this could be due to the user being online when the test was trying to run or something similar). The FCC define the removal of any problem units as: removed measurements for any unit that exhibited greater than or equal to 10% failures in a particular one hour period (the purpose is to remove periods where units were unable to reach the Internet). 2. Panellists that changed service package intra-month were identified and data was removed for the service package that they spent the shortest duration on. This is in line with SamKnows standard global methodology. 3. Per-Whitebox averages were computed for the period of the day for each metric. 4. Per-service-package averages were computed from the per-whitebox averages calculated in step 3. Please note that the analysis is not weighted by Rogers s customer distribution. However a sample plan was built to mitigate this based on the customer base. SamKnows 11

12 C Key Performance Indicators C.1 Download Throughput Sustained download throughput is a measure of the speed a broadband connection is able to deliver for sustained periods of time. This metric is the one most commonly used by regulators when reporting on speed. All of Rogers' packages exceed their advertised rates significantly. The 100Mbps product outperforms all other products, achieving 118% of the advertised speed. All other packages achieve between 106% and 110% of advertised speed. The 250Mbps plan also displays a considerable improvement since March 2014, increasing from 105% to 110% of headline speed. The 30Mbps and 60Mbps plan instead show a slightly lower level of throughput. Sustained Download Throughput, Percentage of Advertised Speed 140% 120% % % % % % % Throguhput (%) 100% 80% 60% 40% 20% 0% 15x1 30x5 60x10 70x10 100x10 250x20 Package Figure a.1: Sustained Download Speed as a percentage of Advertised Speed, by package SamKnows 12

13 Burst download throughput is a measure of the short-term speed that can be achieved on broadband connections. This is commonly much faster than the sustained rate for cable service providers thanks to technologies such as SpeedBoost. Rogers employs SpeedBoost for their packages, whose effects can clearly be seen in the figure below. Most broadband services show a considerable increase in throughput thanks to burst technology with the exception of the 250Mbps product, although it also displays a slight increase from 110% to 118% of advertised speed, again showing an improvement since March In contrast, the effects of burst technology are less significant in this study s testing period for the 30Mbps and 60Mbps plans, although they still display a considerable increase from sustained throughput. The 15Mbps product shows the largest increase from sustained throughput, rising to 183% from 108% of advertised speed. Generally, packages with lower advertised rates benefit the most from burst technology as it is designed to use spare bandwidth. Therefore, it is to be expected that plans with higher advertised rates display less significant improvements on throughput performance, as seen with the 250Mbps plan. Burst Download Throughput, Percentage of Advertised Speed Throughput (%) 200% 180% 160% 140% 120% 100% 80% 60% 40% 20% 0% % % % % % % 15x1 30x5 60x10 70x10 100x10 250x20 Package Figure a.2: Burst Download Speed as a percentage of Advertised Speed, by package SamKnows 13

14 All of Rogers s packages experience a very small decline during the peak period, with the 250Mbps package showing the most significant decrease in throughput compared to all other plans. This decline in performance still remains very small, with throughput dropping by 3.32% points. All packages continue to perform noticeably above the advertised speeds. Throughput (%) 140% 120% 100% 80% 60% 40% 20% 0% Off-peak vs Peak Sustained Download Throughput, Percentage of Advertised Speed % % % % % % 15x1 30x5 60x10 70x10 100x10 250x % Package % % % % % 24hr Peak Figure a.3: Peak vs Off-peak Sustained Download Speed as a percentage of Advertised Speed, by package SamKnows 14

15 C.2 Upload Throughput Sustained upload throughput is a measure of the speed a broadband connection is able to send for sustained periods of time. As with download speed, all plans exceed their advertised rates, achieving a similar performance ranging between 104% and 108% of headline speeds. The 30Mbps and 60Mbps packages exhibit the highest level of throughput as a percentage of advertised speed compared to all other plans, both achieving 108% of headline speeds. This is a considerable improvement for the 60Mbps plan, rising from 98% in March 2014, contrasting with the 30Mbps plan which dropped from 124%. Sustained Upload Throughput, Percentage of Advertised Speed 120% % % % % % % 100% Throughput (%) 80% 60% 40% 20% 0% 15x1 30x5 60x10 70x10 100x10 250x20 Package Figure b.1: Sustained Upload Speed as a percentage of Advertised Speed, by package SamKnows 15

16 Burst upload throughput is a measure of the burst speeds attained during an upload measurement. Some service providers enable Boost technologies only in the downstream direction whilst others enable it in both directions (this is less common). Most of Rogers packages deliver burst upload throughput closely resembling sustained throughput, with all plans exhibiting a small increase, showing only small improvements unlike what is seen in the downstream direction. The 15Mbps package shows a more significant increase in throughput, although it is also a much smaller improvement compared to burst download speed. Burst Upload Throughput, Percentage of Advertised Speed Throguhput (%) 140% 120% 100% 80% 60% 40% 20% % % % % % % 0% 15x1 30x5 60x10 70x10 100x10 250x20 Package Figure b.2: Burst Upload Speed as a percentage of Advertised Speed, by package SamKnows 16

17 As is the case with download speed, upload speed of all of Rogers s packages display a very small decline in average performance during peak hours. All plans continue to exceed headline speeds during the peak period as well. 120% Off-peak vs Peak Sustained Upload Throughput, Percentage of Advertised Speed % % % % % % % % % % % % Throughput (%) 100% 80% 60% 40% 20% 24hr Peak 0% 15x1 30x5 60x10 70x10 100x10 250x20 Package Figure b.3: Peak vs Off-peak Sustained Upload Speed as a percentage of Advertised Speed, by package SamKnows 17

18 C.3 Latency Latency is a measure of the time it takes for a single packet of data to reach its destination from its point of origin and back. It is also known as round-trip latency. The latency of Rogers' packages ranges from 24.52ms to 39.76ms, achieved by the 15Mbps and 70Mbps plans respectively. The average level of latency across all plans is slightly higher than in March 2014, although this is mostly due to the high latency of the 70Mbps product. The 60Mbps plan exhibits a higher level of latency than in the previous testing period, contrasting with the 250Mbps package which shows a noticeably lower latency. Latency Latency (ms) x1 30x5 60x10 70x10 100x10 250x20 Package Figure c: Round-trip latency, in milliseconds, by package SamKnows 18

19 C.4 Packet Loss Packet loss is a measure of the percentage of data packages sent from the home to the destination that are lost during the transmission of data. On average, packet loss is considerably lower than in March 2014, ranging from 0.04% to 0.098% with the 30Mbps, 60Mbps and 250Mbps products all displaying significant improvements. Contrasting with latency results, the 70Mbps package displays the lowest level of packet loss compared to all other products. Packet Loss 0.12% 0.10% 0.098% Packet Loss (%) 0.08% 0.06% 0.04% 0.050% 0.071% 0.040% 0.077% 0.078% 0.02% 0.00% 15x1 30x5 60x10 70x10 100x10 250x20 Package Figure d: Packet loss, by package SamKnows 19

20 C.5 US providers vs Rogers comparison The chart below shows a comparison between the sustained download performance of Rogers to US ISPs. The US data is sourced from the FCC s April 2014 Measuring Broadband America report. The measurement methodology used by the FCC and Rogers is directly comparable, although some small differences exist (such as the measurement servers in use). All US broadband providers achieve an average of 105% of advertised speed during the 24-hour period, with US cable operators reaching 108% of advertised speed during this period. These results are exceeded by Rogers 110% for this study s measuring period. During the peak period, Rogers achieves an average of 108%, continuing to exceed US performance with the latter s overall average of 101%. Rogers s peak period performance also exceeds that of cable operators, which achieve an average of 103% of advertised speed. Rogers overall average drops by 1.33% between the 24-hour and peak periods. This is a better show of consistency for throughput compared to US cable services, which exhibit a decline of 4.62%. Sustained Download Throughput, Percentage of Advertised Speed 160% 140% 120% Throughput (%) 100% 80% 60% 40% 20% 0% 24hr Peak Figure e: Sustaned Download Speed as a percentage of Advertised Speed comparison between Rogers and US providers, by ISP ISP SamKnows 20

21 US cable providers show a very slightly higher average latency of 31.49ms (compared to Rogers's 31.19ms). The following is a breakdown by tier for Rogers and US cable providers. Speed Tier Rogers US cable 15Mbps Mbps Mbps Mbps Table 2: Latency comparison between Rogers and US cable operators, in milliseconds, by speed tier As seen above, individual tiers perform much better on Rogers's side, particularly the Mbps tier. Packet loss sees similar results, with the average across individual tier being slightly higher for US cable operators with 0.12% compared to the 0.07% average of Rogers's speed tiers where applicable. Packet loss of all of Rogers s packages achieves an average of 0.08%. US cable operators only outperform Rogers in the 50-60Mbps tier. Speed Tier Rogers US cable 15Mbps 0.04% 0.12% 30Mbps 0.08% 0.09% 50-60Mbps 0.10% 0.08% Table 3: Packet Loss comparison between Rogers and US cable operators, by speed tier SamKnows 21