ELECTRONIC COPY SAMKNOWS LONG FORM ANALYSIS OF UPC S BROADBAND PERFORMANCE IN NOVEMBER 2014 ELECTRONIC COPY

Transcription

1 ELECTRONIC COPY SAMKNOWS LONG FORM ANALYSIS OF UPC S BROADBAND PERFORMANCE IN NOVEMBER 2014 ELECTRONIC COPY

2 [THIS PAGE LEFT INTENTIONALLY BLANK] Confidential SamKnows 2

3 Contents A BACKGROUND AND OVERVIEW 5 B MAJOR FINDINGS 6 C TESTING METHODOLOGY 7 D TEST SCHEDULE 15 E DATA ANALYSIS 16 F EXPLANATION OF STATISTICAL ANALYSIS; INCLUDING CONFIDENCE INTERVAL 19 G RECRUITMENT 20 H KEY PERFORMANCE INDICATORS 22 H.1 Download Throughput 22 H.2 Upload Throughput 26 H.3 Latency 30 H.4 Webpage Loading Time 31 H.5 Packet Loss 32 Confidential SamKnows 3

4 About This Document This document is a summary report, which has been commissioned by UPC and completed by SamKnows. For the purposes of this study 52 households across UPC customer base were given a specially configured hardware device (SamKnows Whitebox), which runs a series of purpose-built tests to measure Internet performance. The analysis in this report is carried out on data collected in the month of November Any comments on the analysis in this document should be directed to Neil Campbell (Neil@samknows.com) Confidential SamKnows 4

5 A Background and Overview This report presents the findings of a project commissioned by UPC to study broadband performance across their network. The main objective of this study is to measure broadband performance delivered by UPC s 120Mbps product to its users. Many factors influence consumer broadband performance, but this report focuses on those factors under the control of UPC and its network, direct or indirect. Bandwidth limitations or delays occurring in the consumer s home or outside UPC s network would not be shown in the analysis this study s results. Introduction This report contains the first published set of results from the analysis of UPC s broadband connections in Measurements for this study were taken in the testing period between the 1 st and the 31 st of November This report is the first of a series of an ongoing study of residential broadband performance for individuals using UPC s services. This study looks at the results of active performance tests for 52 UPC subscribers. Measurements in this report were taken during the testing period of November The report relies on measurement hardware and software installed in the homes of 52 UPC subscribers. These are the SamKnows Whitebox devices whose software conducts automated measurements of broadband performance throughout the year. The report focuses on UPC s 120Mbps product and contains the key results of the analysis conducted. All figures and metrics depict broadband performance during the full day (24 hours) unless otherwise specified. A line chart depicting hourly broadband performance also shows broadband usage during individual times, providing the reader with the opportunity to examine peak period broadband performance. Throughput performance is measured as a percentage of advertised speed for most of the report. The term advertised speed, or headline speed, is the speed UPC refers to when advertising their products. Confidential SamKnows 5

6 B Major Findings UPC's 120Mbps plan performs just below the advertised download throughput, delivering an average of 87.73% of the headline speed during the peak period. This follows a small decline in average performance from 90.39% during the 24- hour overall measurement period. 50% of users achieved 92% or better of the advertised download speed. Only 8% of users achieved 100% of the advertised speed. In contrast, upload speed exceeds the advertised rate considerably, achieving % of the headline speed. This also follows a small decline from % during the 24-hour measurement period. Every user achieved at least 100% of the advertised upload speed. The behaviour of latency, packet loss and webpage loading times reflect the movement of throughput in the downstream and upstream directions, as all metrics experience a decline in average performance from the 24-hour period into the peak period. Confidential SamKnows 6

7 C Testing Methodology This section describes the system architecture and network programming features of the tests, and other technical aspects of the methods employed to measure broadband performance during this study. A. Selection of Hardware Approach A fundamental choice when developing a solution to measure broadband performance is whether to use a hardware or software approach. Software approaches are by far the most common and allow a very large sample to be reached relatively easily. Web-based speed tests fall into this category. These typically use Flash or Java applets, which execute within the context of the user s web browser. When initiated, these clients download content from remote web servers and measure the throughput of the transfer. Some web-based speed tests also perform upload tests, while others perform basic latency checks. Other less common software-based approaches to performance measurement involve installing applications on the user s workstation which periodically run tests while the computer is switched on. All software solutions implemented on a consumer s computer, smart phone, or other Internet access device suffer from the following disadvantages for the purposes of this study: - The software may itself affect broadband performance; - The software typically does not account for multiple machines on the same network; - The software may be affected by the quality and build of machine; - Potential bottlenecks (such as wireless equipment, misconfigured networks, and older computers) are generally not accounted for and result in unreliable data; - A consumer may move the computer or laptop to a different location which can affect performance; - The tests may only run when the computer is actually on, limiting the ability to provide a 24-hour profile; - For manually-performed software tests, panellists may introduce a bias by when they choose to run the tests (e.g., may only run tests when they are encountering problems with their service). Confidential SamKnows 7

8 In contrast, hardware approaches involve placing a device inside the user s home that is physically connected to the consumer s Internet connection, and periodically running tests to remote targets on the Internet. These hardware devices are not reliant on the user s workstation being switched on, and so allow results to be gathered throughout the day and night. The primary disadvantages of a hardware approach are that this solution is much more expensive than a software approach and requires installation of the hardware by the consumer or a third party. A hardware approach was used in this instance. B. Design Principles and Technical Approach For this test of broadband performance, SamKnows used design principles that were previously developed in conjunction with their study of broadband performance in the U.K and USA. The design principles comprise sixteen technical objectives: Technical Objective 1. Must not change during the monitoring period. 2. Must be accurate and reliable. 3. Must not interrupt or unduly degrade the consumer s use of the broadband connection 4. Must not allow collected data to be distorted by any use of the broadband connection by other applications on the host PC and other devices in the home. 5. Must not rely on the knowledge, skills and participation of the consumer for its on-going operation once installed. Methodological Accommodations The Whitebox measurement process is designed to provide automatic and consistent monitoring throughout the measurement period. The hardware solution provides a uniform and consistent measurement of data across a broad range of participants The volume of data produced by tests is controlled to avoid interfering with panellists overall broadband experience, and tests only execute when consumer is not making heavy use of the connection The hardware solution is designed not to interfere with the host PC and is not dependent on that PC. The Whitebox is plug-and-play. Instructions are graphics-based and the installation process has been substantially field tested. Confidential SamKnows 8

9 Technical Objective Methodological Accommodations 6. Must not collect data that might be deemed to be personal to the consumer without consent. 7. Must be easy for a consumer to completely remove any hardware and/or software components if they do not wish to continue with the research program. 8. Must be compatible with a wide range of DSL, cable, satellite and fiber-to-the-home modems. 9. Where applicable, must be compatible with a range of computer operating systems, including, without limitation, Windows XP, Windows Vista, Windows 7, Mac OS and Linux. 10. Must not expose the volunteer s home network to increased security risk, i.e., it should not be susceptible to viruses, and should not degrade the effectiveness of the user s existing firewalls, antivirus and spyware software. The data collection process is explained in plain language and consumers are asked for their consent regarding the use of their personal data as defined by any relevant data protection legislation. Whiteboxes can be disconnected at any time from the home network. As soon as the route is reconnected the reporting is resumed as before. Whiteboxes can be connected to all modem types commonly used to support broadband services in Canada. Whiteboxes are independent of the PC operating system and therefore able to provide testing with all devices regardless of operating system. Most user firewalls, antivirus and spyware systems are PC-based. The Whitebox is plugged in to the broadband connection before the PC. Its activity is transparent and does not interfere with those protections. 11. Must be upgradeable from the remote control center if it contains any software or firmware components. 12. Must identify when a user changes broadband provider or package (e.g., by a reverse look up of the consumer s IP address to check provider, and by capturing changes in modem connection The Whitebox can be completely controlled remotely for updates without involvement of the consumer PC, providing the Whitebox is switched on and connected. Ensures regular data pool monitoring for changes in speed, ISP, IP address or performance, and flags when a panellist should notify and confirm any change to their broadband service since the last test execution. Confidential SamKnows 9

10 Technical Objective speed to identify changes in package). 13. Must permit, in the event of a merger between ISPS, separate analysis of the customers of each of the merged ISP s predecessors. 14. Must identify if the consumer s computer is being used on a number of different fixed networks (e.g., if it is a laptop). 15. Must identify when a specific household stops providing data. 16. Must not require an amount of data to be downloaded which may materially impact any data limits, usage policy, or traffic shaping applicable to the broadband service C. SamKnows Methodology Each deployed Whitebox performs the following tests: Methodological Accommodations Data are stored based on the ISP of the panellist, and therefore can be analyzed by individual ISP or as an aggregated dataset. The Whiteboxes are broadband dependent, not PC or laptop dependent. The Whitebox needs to be connected and switched on to push data. If it is switched off or disconnected its absence is detected at the next data push process. The data volume generated by the information collected does not exceed any policies set by ISPs. Panelists with bandwidth restrictions can have their tests set accordingly. Test Download Speed Upload Speed Web Browsing UDP Latency UDP Packet Loss Video Streaming Voice over IP DNS Resolution Primary Measure (s) Throughput in Megabits per second (Mbps) utilizing three concurrent TCP connections Throughput in Megabits per second (Mbps) utilizing three concurrent TCP connections Total time to fetch a page and all of its resources from a popular website Average round trip time of a series of randomly transmitted UDP packets distributed over a long timeframe Fraction of UDP packets lost from UDP Latency test. Initial time to buffer, number of buffer under-runs and total time for buffer delays. Upstream packet loss, downstream packet loss, upstream jitter, round trip latency Time taken for the ISP s recursive DNS resolver to return a record for a popular website domain name. Confidential SamKnows 10

11 The following subsection details the methodology used in each of the individual tests. DOWNLOAD SPEED AND UPLOAD SPEED These tests measured the download and upload throughput by performing multiple simultaneous HTTPGET and HTTPPOST requests to a target test node. Binary, non-zero content herein referred to as the payload was hosted on a web server on the target test node. The test operated for a fixed duration of 30 seconds. It also recorded average throughput at 5 second intervals during the test. The client attempted to download as much of the payload as possible for the duration of the test. The test used three concurrent TCP connections (and therefore three concurrent HTTP requests) to ensure that the line was saturated. Each connection used in the test counted the numbers of bytes transferred and was sampled periodically by a controlling thread. The sum of these counters (a value in bytes) divided by the time elapsed (in microseconds) and converted to Mbps was taken as the total throughput of the line. Three TCP connections were selected for the speed tests because of the following reasons: 1) Over a relatively short RTT (<50ms), we found little to no benefit in moving to a larger number of TCP connections. Over much larger RTTs (e.g. when we carry out international transfers over 250ms) there can be significant benefit to moving to 8 or more concurrent connections. 2) UDP was discounted because the majority of the real-world applications rely upon TCP, and middleboxes may behave differently with TCP and UDP. The choice of three concurrent TCP connections is not the dominating factor in the measurement of "burst speed". The most important factor here is the duration / size of transfer that the burst measurement accounts for. A figure of 5 seconds was selected as it represented (a) a short amount of time (thus fitting the "burst" definition) and (b) it tallied reasonably closely with the actual durations that were experienced on various US cable operators' packages. It is worth noting that all of the above criteria are based upon the extensive testing we have carried out with the FCC and North American ISPs. It is designed to be ISP and technology agnostic, as far as reasonably possible. Confidential SamKnows 11

12 Factors such as TCP slow start and congestion were taken into account by repeatedly transferring small chunks (256 kilobytes, or kb) of the target payload before the real testing began. This warm up period was said to have been completed when three consecutive chunks were transferred at within 10% of the speed of one another. All three connections were required to have completed the warm up period before the timed testing began. The warm-up period was excluded from the measurement results. Downloaded content was discarded as soon as it was received, and was not written to the file system. Uploaded content was generated and streamed on the fly from a random source. WEB BROWSING The test recorded the averaged time taken to sequentially download the HTML and referenced resources for the home page of each of the target websites, the number of bytes transferred, and the calculated rate per second. The primary measure for this test was the total time taken to download the HTML front page for each web site and all associated images, JavaScript, and style sheet resources. This test did not test against the centralized testing nodes; instead it tested against real websites, ensuring that the effects of content distribution networks and other performance enhancing factors could be taken into account. Each Whitebox tested against the following 8 websites The results include the time taken for DNS resolution. The test used up to eight concurrent TCP connections to fetch resources from targets. The test pooled TCP connections and utilized persistent connections where the remote HTTP server supports them. Confidential SamKnows 12

13 The client advertised the user agent as Microsoft Internet Explorer 8. Each website was tested in sequence and the results summed and reported across all sites. UDP LATENCY AND PACKET LOSS These tests measured the round trip time of small UDP packets between the Whitebox and a target test node. Each packet consists of an 8-byte sequence number and an 8-byte timestamp. If a packet was not received back within three seconds of sending, it was treated as lost. The test recorded the number of packets sent each hour, the average round trip time and the total number of packets lost. The test computed the summarized minimum, maximum, and mean from the lowest 99% of results, effectively trimming the top (i.e., slowest) 1% of outliers. The test operated continuously in the background. It was configured to randomly distribute the sending of the echo requests over a fixed interval of one hour, reporting the summarized results once the interval had elapsed. Approximately 600 packets were sent within a one hour period, with fewer packets sent if the line was not idle. This test was started when the Whitebox booted and ran permanently as a background test. VIDEO STREAMING For the purpose of the video streaming test, the intent was to simulate an end user viewing a streaming video online. This test emulated live video streaming rather than a service such as YouTube that employs a progressive download approach. The test operated over TCP and used proprietary client and server side components. The client and server negotiated the test parameters at the start of each test. A three-second playout buffer was configured and the client attempted to download data from the server at the maximum rate necessary to ensure that this buffer was never empty. A separate client-side thread consumed data from this buffer at a fixed rate, looking for buffer under-runs (which would manifest themselves to users as a pause in video). The Whitebox recorded the time to initial buffer, the total number of buffer under-runs and the total delay in microseconds due to these under-runs. Confidential SamKnows 13

14 The test operated at four bit rates: 768 kilobits per second (kbps), 1.25 Mbps, 2.25 Mbps, and 3.75 Mbps. VOICE OVER IP The Voice over IP (VoIP) test operated over UDP and, unlike the video streaming test, utilized bidirectional traffic, as is typical for voice calls. The Whitebox would handshake with the server, and each would initiate a UDP stream with the other. The test used a 64 kbps stream with the same characteristics and properties (i.e., packet sizes, delays, bitrate) as the G.711 codec. The test measured jitter, delay and loss. These metrics were measured by subdividing the stream into blocks, and measuring the time taken to receive each block (as well as the difference between consecutive times).jitter was calculated using the PDV approach described in section 4.2 of RFC5481. The 99th percentile was recorded and used in all calculations when deriving the PDV. DNS RESOLUTION AND DNS FAILURES These tests measured the DNS resolution time of an A record query for the domains of the websites used in the web browsing test, and the percentage of DNS requests performed in the DNS resolution test that failed. The DNS resolution test was targeted directly at the ISP s recursive resolvers. This circumvented any caching introduced by the panellist s home equipment (such as another gateway running in front of the Whitebox) and also accounted for panellists that might have configured the Whitebox (or upstream devices) to use non-isp provided DNS servers. ISPS provided lists of their recursive DNS servers for the purposes of this study. Confidential SamKnows 14

15 D Test Schedule Measurements were conducted and data collected during the month of November The same metrics, test parameters and testing schedule as the European Commission broadband performance project were used. A detailed description of the methodology underpinning each metric can be found here: A summary of the parameters and measurement frequency for each metric is detailed in the table below: Test Schedule for UPC Test Target(s) Frequency Duration Web browsing 8 popular Irish websites Hourly, 24/7 Video streaming* 1 off-net test node Hourly, 24/7 Est. 30 seconds Fixed 10 seconds at 768kbps, 1.25Mbps, 2.25Mbps, 3.75Mbps Voice over IP 1 off-net test node Hourly, 24/7 Fixed 30 seconds at 64k Download speed Upload speed 1 off-net test node 1 off-net test node Once every 6 hours until 6pm, then hourly thereafter Once every 6 hours until 6pm, then hourly thereafter UDP latency 1 off-net test node Hourly, 24/7 Permanent UDP packet loss 1 off-net test node Hourly, 24/7 Permanent DNS resolution 8 popular Irish websites Hourly, 24/7-6MB if speed < 30Mbit/s - 12MB if speed >= 30Mbit/s and < 50Mbit/s - 10 seconds if speed >= 50Mbit/s - 3MB if speed < 10Mbit/s - 6MB if speed >= 10Mbit/s and < 20Mbit/s - 10 seconds if speed >= 20Mbit/s Est. 3 seconds Jitter 1 off-net test node Every 2 hours Est. 5 seconds * Video streaming rates: Lines will only stream the rates they are capable of, according to the latest speed test results. If a rate is deemed unreachable (e.g. a 3.75Mbps rate on a 1Mbps line), then it will be skipped. Confidential SamKnows 15

16 E Data Analysis This section describes the methods employed to collect and analyse the test results. A. Background i. Time of Day One of the key factors that affect broadband performance is usage-based congestion. At peak hours, defined for this study as the period on weekdays between 8:00 pm and 10:00 pm local time, there are more people attempting to use the Internet simultaneously, giving rise to the potential for congestion if any of these points are provisioned on a contended basis. When congestion occurs, users performance will suffer. ii. Service Tier A sufficient sample size is necessary to allow meaningful statistical analysis and the ability to robustly compare the performance of specific packages. B. Data Collection and Analysis Methodology i. Data Integrity As the Whiteboxes ran tests consistently from homes across Ireland, it was important to check the data to ensure that any anomalies were removed. To ensure the integrity of the large amount of data collected, the following protocols were developed: 1. Change of ISP intra-month: found units that changed ISP intra-month (determined by performing daily WHOIS query using the panellist s IP address), and removed data. 2. Change of service tier intra-month: found units that changed service tier intramonth by isolating the difference between the average throughput observed for the first three days in the reporting period from the average throughput observed for the final three days in the reporting period. If a unit was not online at the start or end of that period, then the first/final three days that they were actually online were taken. If this difference was over 50%, the downstream and upstream charts for this unit were individually reviewed. Where an obvious step change was observed (e.g., from 30 Mbps to 110 Mbps), the data for the shorter period was flagged for removal. Confidential SamKnows 16

17 3. Removal of any failed or irrelevant tests: removed any failed or irrelevant tests by removing measurements that exhibited greater than or equal to 10% failures in a specific one hour period. 4. Removal of any problem units: removed measurements for any unit that exhibited greater than or equal to 10% failures in a particular one hour period (the purpose was to remove periods where units were unable to reach the Internet). ii. Collation of Results and Outlier Control All measurement data were collated and stored for analysis purposes as monthly trimmed averages during three time intervals (24 hours, 8:00 pm to 10:00 pm local time Monday through Friday, 12:00 am to 12:00 am local time Saturday and Sunday). Only participants who provided a minimum of one week (seven days) of valid measurements and had valid data in each of the three time intervals were included in the November 2014 test results. In addition, the top and bottom 1% of measurements were dropped to control for outliers that may have been anomalous or otherwise not representative of actual broadband performance. All statistics were computed on the trimmed data. iii. Peak Hours Adjusted to Local Time Peak hours were defined as weekdays between 8:00 pm to 10:00 pm (inclusive) for the purposes of the study. Due to some tests that only took place once every two hours on an individual Whitebox, the period used for aggregating peak performance had to be a multiple of two. iv. Congestion in the Home Not Measured Download, upload, latency, and packet loss measurements were taken between the panellist s home gateway and the test nodes provided by SamKnows. Web browsing measurements were taken between the panellist s home gateway and eight popular Irish-hosted websites. Any congestion within the user s home network is therefore not measured by this study. The web browsing measurements are subject to possible congestion at the content provider s side, although the choice of nine highly trafficked websites configured to serve high traffic loads may have mitigated the effects of temporary congestion. v. Latencies Attributable to Propagation Delay The speeds at which signals can traverse networks are limited at a fundamental level by the speed of light. While the speed of light is not believed to be a significant limitation in the context of the other technical factors addressed by the testing methodology, a delay of 5ms per 1000 km of distance travelled can be Confidential SamKnows 17

18 attributed solely to the speed of light. The geographic distribution and the testing methodology s selection of the nearest test servers are believed to minimize any significant effect. However, propagation delay is not explicitly accounted for in the results. Confidential SamKnows 18

19 F Explanation of Statistical Analysis; Including Confidence Interval The below depicts how raw data was processed for the production of the UPC report. Raw data for the chosen period is collected from the measurement database. The products that panellists were on are exported to a unit profile file, and those that changed during the period are flagged. Data is cleaned. This includes removing measurements when a user changed ISP/product during the period. Anomalies and significant outliers are also removed at this point. Per-unit results are generated for each metric. It is at this point that the 8-10pm averages are computed and the trimmed mean calculated for each metric. The per-unit CSV data is processed by SPSS scripts. This process removes products with low sample sizes and computes statistical averages for the remainder that can be used in the report. Summary data tables and charts in Excel are produced from the statistical averages. These are used directly in the report. Confidential SamKnows 19

20 G Recruitment For the purposes of this study, SamKnows sought to recruit panellists using the following method: How does the recruitment process work? 1. UPC sent a recruitment to its customer base. SamKnows assists and reviews this as well as suggesting best practices. Each consumervolunteer is directed to a web form that is built specifically for each project. Once there, they are asked to complete a short form which gives SamKnows a minimum amount of personal information so that SamKnows can determine whether each volunteer fits the sample plan requirements. 2. Once selected the volunteer is then sent an which asks for further information and requests that the volunteer complete a speed test. This speed test has been developed in-house by SamKnows and is actually the official test for the Brazilian government. On the basis of the information provided by the customer and the results of the speed test, it is then decided whether the consumer is eligible for the next stage. 3. If successful, the volunteer is sent an End User License Agreement that details key considerations such as the responsibilities of the volunteer and SamKnows respectively, data ownership and duration of the project. On completion of the EULA SamKnows will then organize for the dispatch of a Whitebox. 4. A spreadsheet is sent via encryption to our distribution partner of over 4 years. Ours is a long-standing relationship built on shared foundations of trust, expertise and dedication. SamKnows has built a number of automatic systems that the distribution partner can pull data from and also push data to. It is in this way that SamKnows is able to automatically check the progress of Whiteboxes as they are dispatched from the warehouse and ensure they are delivered to the volunteer in a timely fashion. 5. Once they receive confirmation that the Whitebox has been delivered, the SamKnows support team will contact the volunteer in order to provide assistance with installing the Whitebox. Over four years SamKnows has developed a sophisticated support team infrastructure that ensures our volunteers receive the best possible support during installation and throughout the project. 6. Once installed the Whitebox calls home to request the SamKnows test suite. Once received it will start to run tests according to the pre-defined Confidential SamKnows 20

21 testing schedule. The volunteer is then sent an automatically by the reporting system with details of how to access the results data from the SamKnows reporting system. The entire process and system has been developed over the course four years and is constantly being refined. As a consequence of this focus and development the volunteer recruitment through to on-boarding and ongoing support is an extremely efficient process. This methodology has been used successfully in the following continents: Asia, North America, South America and Europe. Confidential SamKnows 21

22 H Key Performance Indicators H.1 Download Throughput Actual vs. Advertised Throughput Figure a.1 plots actual throughput during the peak period against advertised throughput for UPC s 120Mbps package. As mentioned previously in this study, the 120Mbps plan performs just below the advertised rate, achieving Mbps. Throughput (Mbps) Actual vs Advertised Download Speed x10 (IE) Advertised Actual Figure a.1: Actual vs Advertised Download Speed, by package Confidential SamKnows 22

23 24-Hour vs Peak Period Throughput Figure a.2 below shows download throughput as a percentage of advertised speed for the 120Mbps package during the 24-hour measurement period against download speed as a percentage of advertised speed during the peak period. Throughput experiences a small decline in average performance between the two periods, falling from 90.39% to 87.73%. Throughput (%) 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 24-hour vs Peak Period Download Speed as a Percentage of Advertised Speed 90.39% 120x10 (IE) 87.73% 24hr Peak Figure a.2: 24-hour vs peak period Download Speed as a percentage of Advertised Speed, by package Confidential SamKnows 23

24 Throughput by Time of Day Download throughput as a percentage of advertised throughput varies by time of day. The behaviour of throughput during the day is depicted in Figure a.3 below, which shows download speed experiences very little variation during the day with no significant changes. As shown in Figures a.1 and a.2 above, throughput performs just under the advertised speed at all times, achieving over 90% at all times before 6pm, and exhibits a small but sharp decline in performance during the peak period. 24-Hour Download Throughput Time of as Day a Percentage of Advertised Speed Throughput (%) 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 120x10 (IE) Time of Day Figure a.3: Hourly Download Throughput as a percentage of Advertised Speed, by package Confidential SamKnows 24

25 Cumulative Distribution of Throughput This report also includes cumulative distribution charts for throughput in the downstream and upstream directions to illustrate how broadband performance varies within the sample population for each product. This allows us to understand what proportion of users achieved at least some speed. For instance, looking at the 80 th percentile value (the y-axis) tells us what speed at least 80% of users achieved. Figure a.4 below depicts the CDF plot for download speeds, as a percentage of advertised speed, amongst panelists. The long tail on the chart shows us that there are a noticeable proportion of users that achieve significantly lower than advertised speeds. We observe that only 7.69% of users achieve 100% of advertised speed. 50% of users achieve 92% of advertised speeds or better. Percentile 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Download Speed Cumulative Frequency Distribution 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% 105% Percentage of Advertised Speed 120x10 (IE) Figure a.4: Cumulative Distribution of Download Speed as a percentage of Advertised Speed, by package Confidential SamKnows 25

26 H.2 Upload Throughput Actual vs. Advertised Throughput Figure b.1 shows actual upload throughput during the peak period against advertised throughput for UPC s 120Mbps package. UPC s 120Mbps package exceeds its advertised upload speed considerably, achieving 12.08Mbps against the advertised upload speed of 10Mbps. Throughput (Mbps) Actual vs Advertised Upload Speed x10 (IE) Advertised Actual Figure b.1: Actual vs Advertised Upload Speed, by package Confidential SamKnows 26

27 24-Hour vs Peak Period Throughput Figure b.2 below shows upload throughput as a percentage of advertised speed for the 120Mbps package during the 24-hour measurement period against upload speed as a percentage of advertised speed during the peak period. Upload speed experiences almost no change between the 24-hour and peak periods, displaying a very small decline of only 0.33%. This indicates upload speed is very stable during the day and always exceeds the headline speed. 140% 120% 24-hour vs Peak Period Upload Speed as a Percentage of Advertised Speed % % Throughput (%) 100% 80% 60% 40% 20% 24hr Peak 0% 120x10 (IE) Figure b.2: 24-hour vs peak period Upload Speed as a percentage of Advertised Speed, by package Confidential SamKnows 27

28 Cumulative Distribution of Throughput Figure b.4 below depicts the cumulative distribution of upload speed during the peak period. In contrast to the download speed CDF, there is almost no tail on this chart at all. 100% of users achieve at least 100% of the upload speed. Half of the panellists achieve at least 120% or better. In general, there is a far tighter distribution of results in the upstream than the downstream, and the results are generally better too. Percentile 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Upload Speed Cumulative Frequency Distribution 0% 6% 12% 18% 24% 30% 36% 42% 48% 54% 60% 66% 72% 78% 84% 90% 96% 102% 108% 114% 120% 126% 132% Percentage of Advertised Speed 120x10 (IE) Figure b.3: Cumulative Distribution of Upload Speed as a percentage of Advertised Speed, by package Confidential SamKnows 28

29 Download vs Upload Throughput Broadband services are typically offered with advertised asymmetric download and upload rates, with the former usually being much higher than the latter. As seen above, download speed performs just below the advertised rate during the peak period, with upload speed greatly exceeding the headline speed. It is also noted that upload speed experiences a far less significant decline between the 24- hour and peak periods. Download vs Upload Speed as a Percentage of Advertised Speed 140% % 120% Throughput (%) 100% 80% 60% 40% 20% 0% 87.73% 120x10 (IE) Download Upload Figure c: Download vs Upload Speed as a percentage of Advertised Speed, by package Confidential SamKnows 29

30 H.3 Latency Figure d shows the average round trip latency to Irish measurement servers for each package. The 120Mbps package delivers a latency of 13.73ms during the peak period. This is a very small increase from 13.69ms during the 24-hour measurement period. This change in the behaviour of latency reflects the behaviour of throughput in the downstream and upstream directions. The round-trip latency measured here should be adequate for common latencysensitive Internet applications such as VoIP Hour vs Peak Period Latency Latency (ms) hr Peak 0 120x10 (IE) Figure d: Round-trip latency, in milliseconds, by package Confidential SamKnows 30

31 H.4 Webpage Loading Time Figure e below shows the average webpage loading time during the peak and 24- hour measurement periods in milliseconds for UPC s 120Mbps product. The behaviour of webpage loading time is almost identical to that of latency, experiencing an increase from ms in the 24-hour to the peak period. 24-Hour vs Peak Period Webpage Loading Time Webpage Loading Time (ms) hr Peak 0 120x10 (IE) Figure e: Webpage Loading time, in milliseconds, by package Confidential SamKnows 31

32 H.5 Packet Loss Figure f below depicts packet loss of UPC's 120Mbps plan during the 24-hour and peak periods. Packet loss is relatively low across both measurement periods, exhibiting a slight increase from 0.107% in the 24-hour period to 0.134% during peak hours. This is also in line with the behaviour of latency and webpage loading time. Packet loss, like latency, is not determined by line speed. It is instead caused by other intrinsic factors such as signal degradation in the access technology, network congestion and corrupted packets being rejected during transmission. The low level of packet loss indicates a well-engineered network as well as good connectivity between consumers and the measurement server. 24-Hour vs Peak Period Packet Loss 0.16% 0.14% 0.134% Packet Loss (%) 0.12% 0.10% 0.08% 0.06% 0.04% 0.107% 24hr Peak 0.02% 0.00% 120x10 (IE) Figure f: Packet loss, by package Confidential SamKnows 32

33 [DOCUMENT ENDS NO MORE PAGES AFTER THIS ONE] Confidential SamKnows 33