Lab Validation Report

Transcription

1 Lab Validation Report Limelight Orchestrate Performance Examining the Performance of Dynamic Content Delivery Between Leading Content Delivery Networks By Aviv Kaufmann, ESG Lab Analyst and Brian Garrett, Vice President of ESG Lab February 2014

2 Lab Validation: Limelight Orchestrate Performance 2 Contents Abstract... 3 The Challenges... 3 The Solution: Orchestrate Performance... 4 How the Solution Was Tested... 5 The Results... 7 Dynamic HTML Response Time... 7 Content Download Times ESG Lab Validation Highlights Issues to Consider The Bigger Truth ESG Lab Reports The goal of ESG Lab reports is to educate IT professionals about data center technology products for companies of all types and sizes. ESG Lab reports are not meant to replace the evaluation process that should be conducted before making purchasing decisions, but rather to provide insight into these emerging technologies. Our objective is to go over some of the more valuable feature/functions of products, show how they can be used to solve real customer problems and identify any areas needing improvement. ESG Lab's expert third-party perspective is based on our own hands-on testing as well as on interviews with customers who use these products in production environments. This ESG Lab report was sponsored by Limelight Networks. All trademark names are property of their respective companies. Information contained in this publication has been obtained by sources The Enterprise Strategy Group (ESG) considers to be reliable but is not warranted by ESG. This publication may contain opinions of ESG, which are subject to change from time to time. This publication is copyrighted by The Enterprise Strategy Group, Inc. Any reproduction or redistribution of this publication, in whole or in part, whether in hard-copy format, electronically, or otherwise to persons not authorized to receive it, without the express consent of The Enterprise Strategy Group, Inc., is in violation of U.S. copyright law and will be subject to an action for civil damages and, if applicable, criminal prosecution. Should you have any questions, please contact ESG Client Relations at

3 Abstract Lab Validation: Limelight Orchestrate Performance 3 ESG Lab evaluated the results of tests that compared the global performance of the industry s leading CDNs when delivering dynamic HTML objects from an active origin server for a well-known online shopping website. The tests were designed to show the importance of middle-mile optimization technologies in the delivery of dynamic content, and to benchmark Limelight Orchestrate Performance against four industry leading CDN service offerings. The Challenges In order to offer end-users a better and more customizable experience, organizations have increasingly incorporated dynamic content into their websites. Dynamic content provides a far more unique experience for each user. But each page containing dynamic content requires additional processing, and web browsers need to receive an HTML framework before the page can be rendered. The framework is non-cacheable and must be retrieved from an origin server that, in many cases, is not located near to the user and may even be located on the other side of the world. This adds to the response time of every request and the user must wait longer for pages to load. But end-users are no longer willing to wait for pages to load or for videos to buffer. While the rule of thumb used to be that users are willing to wait for up to two seconds for a page to load, 1 recent studies by Microsoft have shown that people will visit a website less often if it is slower than a close competitor by more than 250ms (one quarter of a second). 2 Amazon.com has reported that for every 100ms increase in load time of its pages, sales will decrease by 1%, and a Google study found that a 500ms increase in load time (half a second) resulted in a 20% drop in traffic. 3 To make matters worse, a steadily increasing number of users are accessing web content with their mobile devices and across wireless networks, resulting in higher latency and more unpredictable response times for the delivery of dynamic content. To help speed connectivity to their origin servers, most website administrators have optimized their Internet speed to their data centers and may employ various technologies to help improve aspects of website performance. Many companies, however, have not yet optimized the delivery of dynamic content across the segment of data transfer that covers the longest distance required to reach global customers: the middle-mile. Figure 1. The Cost of Increased Page Load Times to an Online Business 1 Source: Nah, F., Behaviour & Information Technology, A study on tolerable waiting time: how long are Web users willing to wait?, Source: The New York Times, For Impatient Web Users, an Eye Blink Is Just Too Long to Wait, February Source: Google VP Marissa Mayer, quoted at 2006 Web 2.0 Conference.

4 Lab Validation: Limelight Orchestrate Performance 4 The Solution: Orchestrate Performance To help ensure a predictable level of performance and minimize the latencies associated with the delivery of dynamic content, many companies choose to leverage the acceleration services offered by some content delivery network (CDN) providers. Limelight Orchestrate Performance is a product that leverages Limelight s global CDN for the delivery of website content to endpoint devices (mobile or desktop). Orchestrate Performance can be used to optimize the delivery of both static and dynamic content and ensures that delivery is optimized at all three of the major data transfer segments: first-mile, middle-mile and last-mile. First-mile Optimization: First-mile optimizations focus on minimizing latency between the origin server (the server hosting the website) and the CDN service provider. Orchestrate Performance optimizes the first-mile using technologies like WAN acceleration, TCP acceleration, and connection control technologies. Middle-mile Optimization: In a global network, the middle-mile may be the most important segment that a CDN controls. Most of the distance that content must travel is across the middle mile, between the origin server and the point that content is passed on to the end-user s access network. Limelight Networks is one of the few CDNs that privately operates, and can therefore optimize, its own dedicated high-speed network. Limelight has leveraged years of experience, expertise, and product development to ensure that middle-mile latency is kept to a minimum. Last-mile Optimization: The last-mile refers to network segment where data is transferred from the end user s access network to the client device. This can include wired and wireless networking as well as mobile network delivery. Performance in the last-mile can be difficult to benchmark in a reproducible manner because each client s situation is different, producing variables that would create inaccurate test results. For instance, latencies can be introduced from many uncontrollable sources such as network congestion and poor signal strength. Orchestrate Performance optimizes the last-mile performance through WAN and TCP acceleration as well as front-end acceleration (which optimizes the presentation of content within the end-user app or browser). Figure 2. Limelight Networks Orchestrate Performance Three-segment Optimization for Dynamic Content

5 Lab Validation: Limelight Orchestrate Performance 5 How the Solution Was Tested A benchmark test was designed to accurately assess the real-world delivery speed of a dynamic HTML framework from a live e-commerce origin server to clients located around the globe. The test analyzed results from Orchestrate Performance and four competing content delivery network services, including two established companies who have been in business for the last five to ten years, and two newer services introduced within the last five years. The test was designed to be easily auditable and easily reproducible for the purposes of accuracy and transparency. The test focused on measuring performance across the first-mile and middle-mile. Although the last-mile is very important to the delivery of dynamic content, it was not a focus of this test due to the variables that last-mile access networks introduce. The test was performed double-blind, meaning that neither the e-commerce site nor the CDN companies (with the exception of Limelight) were made aware of the test, in order to ensure the most accurate results. Gomez Backbone Testing The well-known and well-respected Gomez backbone test was used for this test. 4 The Gomez backbone test, a service from Compuware, utilizes agents physically located all over the world to generate requests and measure the response of website performance from each location. As is shown in Figure 3, Limelight selected agents located globally in countries and regions in which Internet traffic is high and where Limelight maintains a point of presence (POP). Figure 3. Location of Gomez Backbone Agents Used in Global Testing of CDN Performance The backbone test was designed to measure each service provider s ability to respond to a request, forward that request, retrieve the requested dynamic content from the origin server, and offload the requested dynamic content to the requesting network. A respected e-commerce website was chosen to serve as the source of the dynamic content. The e-commerce website was chosen due to its well-publicized, highly optimized infrastructure and optimal configuration, which 4 For more information please visit and/or

6 Lab Validation: Limelight Orchestrate Performance 6 provides dynamic content to CDNs. It should be noted that all CDNs tested would be expected to benefit equally from the e-commerce site s own optimizations, a fact which helped to minimize the complexity of the test. The object selected for the test was served from a single URL, on a page with ideal dynamic properties. Examples of e-commerce URLs containing dynamic content similar to the URL tested in the test are listed in Table 1. Table 1. e-commerce URLs Containing Dynamic Content e-commerce Website toysrus.com asos.com etsy.com darty.com URL Containing Similar Dynamic Content as Used in Test dartyclic=x_tv-vide-home-cine_cate_ampl-home-cine The URL of the dynamic HTML page used in the test was chosen because it contains content that is not cacheable, and thus must be retrieved from the origin server upon each new request. The test object was chosen because its size required multiple TCP segments to be requested from the origin server and transferred across the CDN upon each request from the backbone agent. The multiple round-trips required across a global network helps to highlight the performance benefits achievable through first-mile and middle mile optimizations. To conduct the test, the selected website was analyzed to determine the hostnames used by any CDNs already configured to deliver it. Configurations were created for any CDNs tested that offered a self-ordering and configuration interface. The Gomez script recorder was then used to create scripts for each CDN being tested. To avoid introducing variance in DNS lookup times that could impact the test results, the CDN hostnames for all service providers were used as the primary hostnames in the test, with the real website hostname being introduced using a HOST-HEADER override in the Gomez script recorder. Three of the service providers tested, including Limelight Orchestrate Performance, explicitly stated that they offer middle-mile optimizations for dynamic content delivery, while two of the service providers offered no explicit middle-mile optimization service. Neither the website nor the CDN companies tested, other than Limelight Networks, were aware of the test taking place. ESG Lab verified that the optimizations used in the test were deployed on the standard Limelight Orchestrate Performance service footprint, with no special tuning parameters or priorities given beyond those available to a typical Limelight customer. To ensure that results were consistent and repeatable, the request for dynamic content was repeated from every agent every five minutes over a period of 38 hours for a total of 456 data points per location for each CDN tested. The test was also first performed against the website origin server, using no CDN service provider as a baseline for performance measurement. The origin hostname was established using deduction, and the origin server assessed as being configured to accept requests from Gomez and all of the CDNs being tested. The results were compiled by grouping each of the backbone agent locations into a geographical region. The global results included all Gomez backbone agents, while only those agents located in North America, South America, EMEA, and Asia contributed to the respective regional geographical results. In some cases where a particular POP presented an ideal use case (such as the traversal of major oceans for instance), a single location was tested on its own for use as a metro region study.

7 The Results Lab Validation: Limelight Orchestrate Performance 7 ESG Lab examined the Gomez backbone test results with a focus on the two key performance indicators that have the largest impact in the delivery of dynamic content: dynamic HTML response times and content download times. Dynamic HTML Response Time Dynamic HTML response times represent the amount of time taken (in milliseconds) for the service provider to receive a request from the testing agent, forward it across geographical regions to the origin server, and then return the requested object to the testing agent. Results of the test are easily auditable because they were consistently reported in readily available Gomez performance charts and logs. All results were well documented over the entire 38-hour period including time and date stamps. Figure 4 shows an example of one of the Gomez test reports. Figure 4. Sample Gomez Performance Report Gomez reports were checked to ensure that every CDN was measuring response times while downloading the same amount of data from the test URL. The reports verify that all but one of the CDNs downloaded 40KB of data during the test. The other CDN (CDN D) was consistently downloading 246KB of data a behavior that may be due to CDN D not requesting compressed data from the origin server. This discrepancy almost certainly contributes to the higher response times seen from CDN D. Figure 5 shows a summary of the results of global response time testing reported in the test. The response times shown in the chart represent the average response times recorded over the 38-hour test period. The baseline average response time of the website with no acceleration is also depicted as a dotted red line. With the exception of CDN D, the benefit of using any CDN to minimize response time is clearly visible in the chart.

8 Lab Validation: Limelight Orchestrate Performance 8 Figure 5. Test Results of Global Average Response Time Testing What the Numbers Mean Limelight Orchestrate Performance was faster than all other CDNs in the global test at providing dynamic content by 9-51%. Using Orchestrate Performance, the response time of providing dynamic content to a client from an e- commerce site was cut in half, eliminating more than 400ms of response time. The response times of CDNs that deploy middle-mile optimizations (Orchestrate Performance, CDN A, and CDN B) were substantially lower than those without middle-mile optimizations (CDN C and CDN D). The results clearly indicate that while nearly all CDNs will help to improve the delivery of dynamic content, Limelight Networks dedicated network and expertise in optimization helped deliver dynamic content faster than all of the CDNs tested. ESG Lab also validated that Orchestrate Performance had the fastest response times when delivering dynamic content between the origin server and each of the four geographic regions tested: North America, South America, Europe/Middle East/Africa (EMEA), and Asia. For this portion of the testing, only the results reported by agents physically located in the geographic region were included in the average response time. While the results for each CDN vary by region and location, Orchestrate Performance had the lowest average response time in every geographic region tested. The results of this testing can be seen in Figure 6.

9 Lab Validation: Limelight Orchestrate Performance 9 Figure 6. Test Results of Regional Average Response Time Testing Why This Matters The old adage time is money applies to any business, but when it comes to delivering dynamic content on an e- commerce site, a little bit of time is a lot of money. The adage may as well be rewritten: Milliseconds are worth millions. A recent study stated that a page load slowdown of only one second could cost Amazon.com $1.6 Billion in sales each year. 5 But the benefit of faster delivery of dynamic content is not limited to e-commerce sites. Websites that depend on dynamic content such as those with directed advertising, highly personalized content, and social media will enjoy similar advantages. Using a content delivery network to accelerate the middle-mile can help shave valuable response time from every request for dynamic content frameworks, resulting in higher customer satisfaction, higher customer retention, and ultimately higher sales. ESG Lab validated that Limelight Networks Orchestrate Performance was able to cut the average response time for the delivery of dynamic content to clients across the globe for a live e-commerce site in half. When compared with the average response times of other leading CDNs, Limelight Orchestrate Performance was 9-51% faster at delivering dynamic content globally, and had the lowest average response times in every geographic region tested. 5 Source: One second delay on Amazon, $1.6 billion loss a year, January 2013.

10 Lab Validation: Limelight Orchestrate Performance 10 Content Download Times Content download times are an important metric because they are a good indication of how well a service provider is optimizing the flow of data from the origin server, across the middle-mile, and to the testing client. A standard download process consists of four main stages: DNS time, connect time, time to first byte (TTFB), and content download time. While the first two stages are important to the overall response time, it is the last two steps that contribute to the majority of the response time spent waiting for an object download. Figure 7 compares Orchestrate Performance to a CDN capable of middle-mile optimization and a CDN not capable of middle-mile optimization. Time to first byte (TTFB) was relatively constant for all CDNs, while middle-mile optimization can be seen to have greatly improved the content download time across the middle-mile. Note how content download times yielded a glaring difference between the CDN providers tested and reveals two distinct categories: those CDNs capable of middle-mile optimization (Limelight, CDN A, and CDN B) and those not capable of middle-mile optimization (CDN C and CDN D). Figure 7. Download Process Time Comparison for Agent in Seoul, Korea Time to first byte for each CDN was relatively consistent during this series of tests due mainly to two factors. First, TTFB is largely a function of distance, and the different routes taken by each service provider to get to the origin server were relatively the same distance from the backbone agent in most cases. Second, TTFB can be significantly impacted by the origin server's performance, which we can determine was consistent for each service provider. To illustrate this point, Limelight Networks presented ESG Lab the download process times recorded from the Gomez backbone agent located in Seoul, Korea for all CDNs tested. This particular agent was chosen because it is a long distance from the origin server, is located in a region with a high population of online users, and requires good connectivity and acceleration in order to effectively deliver content. The benefits of middle-mile acceleration can be clearly seen in the content download measurements in Table 2. Table 2. Download Process Time Comparison Data for Agent in Seoul, Korea CDN DNS Time Connect Time Time to First Byte Content (TTFB) Download Non-optimized CDN 78ms 167ms 383ms 965ms Optimized CDN 7ms 2ms 391ms 145ms Limelight Orchestrate Performance 5ms 4ms 436ms 93ms

11 Lab Validation: Limelight Orchestrate Performance 11 Figure 8 shows the average dynamic object content download times for each CDN throughout the 38-hour test period, as reported by all globally dispersed backbone agents. It is clear that the service providers that include middle-mile optimization (Orchestrate Performance, CDN A, and CDN B) were able to deliver content to the agents much faster than the service providers that do not optimize transfers over the middle-mile (CDN C and CDN D). This clearly shows the correlation between middle-mile optimization and the speed of dynamic content delivery. Figure 8. Global Content Download Time By CDN What the Numbers Mean Limelight Orchestrate Performance downloaded dynamic content 16-88% faster than the four other CDNs tested in the test CDNs with middle-mile optimization capabilities (Limelight, CDN A, and CDN B) downloaded dynamic content hundreds of milliseconds (up to 8.5x) faster than those with no middle-mile optimization (CDN C and CDN D). Download times for dynamic objects were eight times longer for CDN D (a CDN with no middle-mile optimization) than for Orchestrate Performance. The results show the advantage of Limelight Networks dedicated and highly optimized content delivery platform even when compared with the other CDNs that perform middle-mile optimization. In addition to being the fastest globally, Limelight Orchestrate Performance was the fastest in every geographic region tested. Being able to download content quickly is only beneficial to the end-user if the download performance is consistent and predictable. ESG Lab validated that response time and content download time for all phases of the test for Orchestrate Performance were not only faster compared with the other CDNs, but in nearly every case, far more predictable and consistent than the others. In Figure 9, ESG Lab compared content download times reported for agents in the EMEA region for Orchestrate Performance, a CDN with middle-mile optimization, and a CDN with no middle-mile optimization. The EMEA region was chosen to illustrate the point because performance in this region proved to be highly variable among CDNs tested.

12 Lab Validation: Limelight Orchestrate Performance 12 Figure 9. Variability in Content Download Times Reported over 38-hour Test Duration Table 3. Measured Variance in Average Content Download Time over 38-hour EMEA test Limelight Orchestrate Performance Middle-mile Optimized CDN Non-optimized CDN Average Content Download Time 78ms 142ms 268ms Measured Variance +/- 11% +/- 29% +/- 42% What the Numbers Mean Orchestrate Performance had the fastest content download time of all CDNs tested. The samples reported over the 38-hour test period varied much less for Limelight Networks Orchestrate Performance (11%) than all other CDNs tested. A website using a CDN that is not optimized to accelerate dynamic content delivery showed content download times two to six times slower than Orchestrate Performance, depending on when the content was accessed. Why This Matters The key to success in nearly any business is getting the product to the customer as quickly as possible while maintaining the highest levels of customer satisfaction. The same is true for operators of any website with dynamic content. Optimizing the download time of dynamic content across the middle-mile is absolutely critical because it allows websites to be built with higher quality images and video, get more items in front of customers, and offer a much better shopping experience. This ultimately results in higher customer retention and increased sales. ESG Lab validated that Orchestrate Performance downloads dynamic content faster than other CDNs with middlemile optimization, and much faster than CDNs with no optimization. In addition to offering the fastest content download times, Orchestrate Performance offered the most predictable content download times, ensuring that every customer will enjoy the same level of performance.

13 Lab Validation: Limelight Orchestrate Performance 13 ESG Lab Validation Highlights ESG Lab audited the results of a double-blind test, which proved that Limelight Orchestrate Performance is 9-51% faster than other leading CDNs at delivering dynamic content globally, and was the fastest in every region tested. Orchestrate Performance was able to cut the response times of delivering dynamic content for an active production website in half (compared with not using a CDN). Using a CDN capable of middle-mile optimization results in significant reduction in the delivery of dynamic content across the globe. Due to superior middle-mile optimization capabilities, Orchestrate Performance downloaded dynamic content 16-88% faster than all other CDNs tested. Orchestrate Performance delivered the most predicable level of performance for the duration of the test, with far less variability in the results when compared with all other CDNs tested. Issues to Consider The test was designed to test performance across the first-mile and middle-mile of the delivery path, and achieved this objective in a fair and unbiased manner. Last-mile performance is an important consideration. Although Orchestrate Performance offers last-mile performance optimizations, testing last-mile performance is a logistical challenge and the results are difficult to interpret and normalize among CDNs. The exclusion of last-mile testing in this test was not intended to gain a competitive advantage in Limelight s favor, but to produce accurate and transparent results. Every organization s requirements and customer base are different. Before choosing a content delivery service provider, buyers should understand their requirements and the particular strengths of the CDNs being considered. Some questions to ask of your current or potential CDN provider may be: o How does your CDN scale to deliver your content to a globally distributed audience? o What methods and layers of intelligent cache management are employed by your CDN? o How does the architecture of your CDN ensure high availability and throughput? o Does your CDN offer a high enough cache-hit ratio to deliver real return-on-investment? o What performance optimizations can your CDN offer, in addition to caching and content delivery? o How does your CDN help you deliver your complete digital experience online? o Is your CDN capable of accelerating dynamic content, or merely delivering it? o In performance tests, did your CDN test delivery across the same footprint that you as a customer will have access to? For additional information and access to more comprehensive results on Limelight Networks middle-mile optimization performance test, please contact your Limelight Networks sales representative.

14 The Bigger Truth Lab Validation: Limelight Orchestrate Performance 14 Websites containing dynamic content enable organizations to offer their customers an easily navigable, unique, and customizable experience surrounded by stunning graphics, quick access to their favorite products and services, and an endless amount of relevant content. Dynamic content results in higher customer satisfaction, customer retention, and ultimately increased sales. But this is only true if the dynamic content can be delivered quickly and seamlessly to a global customer base that simply is not willing to wait for pages to load. It is absolutely critical that website administrators take advantage of all technologies that can ensure the quickest possible delivery of dynamic content to the end-user. Most website administrators are aware of the importance of using hardware to accelerate WAN performance to and from their data centers, using edge-caching techniques and designing software for optimal delivery to the client. While many organizations understand the necessity of leveraging a CDN provider to optimize their global performance, they may have trouble differentiating between the technological capabilities of the leading CDN providers. When evaluating content delivery networks for the delivery of dynamic content, IT leaders need to find the right metrics to evaluate performance in order to determine if different vendors truly offer more than standard CDN services, specifically optimization in the middle mile for better dynamic content delivery. This capability is a critical necessity that can reduce response times by hundreds of milliseconds or even seconds. As current research shows, this time is enough to seriously impact customer retention and can translate to hundreds of thousands of dollars in lost sales to a typical e-commerce site. ESG Lab audited the results of Limelight Networks real-world site acceleration benchmark test. ESG Lab validated that Orchestrate Performance is capable of optimizing the delivery of dynamic content across the middle-mile, and was the fastest at delivering dynamic content both globally and within every region tested. Because Limelight Networks owns and optimizes its own private, highly reliable network, its customers can be assured that their dynamic content will be delivered consistently with high speed. Middle-mile optimization is critical in the delivery strategy of dynamic content. As the results of real-world testing showed, Orchestrate Performance was the fastest CDN tested at optimizing the delivery across the middle-mile. Organizations should perform their own studies to determine how quickly they are able to deliver dynamic content to their global customers and how this time impacts their business. If this time is in the hundreds of milliseconds, ESG Lab recommends considering whether Orchestrate Performance can help accelerate the delivery of your dynamic content.

15 20 Asylum Street Milford, MA Tel: Fax: