Benchmark Results. Date: 14/03/2011 Version: 1.0 Author: Philippe Humeau. Naming convention

Transcription

1 Benchmark Results Date: 14/03/2011 Version: 1.0 Author: Philippe Humeau Naming convention Demostore Nitrostore : Basic demostore : a basic demostore with all Nitrogento optimization features turned on

2 SOMMAIRE Test environment 3 Test server configuration 3 Server software setup 3 Test box 3 Firebug testing 4 Demostore result 4 Nitrostore result 5 Funkload test 6 The scripts 6 Homepage test configuration 6 Full scenario test configuration 6 Homepage test explanations 7 Full run scenario explanations 7 Goals 7 Funkload results 8 Homepage 8 Homepage delivered per second 8 Homepage loading time (average & max) 8 APDEX indices (User experience rating) 9 Explanations 9 Results 9 Raw data 10 Full scenario 11 Pages delivered per second 11 Average & max time to load a page during the complete scenario 11 APDEX indices (User experience rating) 12 Explanations 12 Results 12 Raw data 12 Yslow / Pagespeed / GTmetrix tests 13 What is it? 13 Basic demostore 13 More details on demostore ranking 14 Pagespeed notation 14 Yslow ranking 14 Nitrostore 15 More details on demostore ranking 15 Pagespeed notation 15 Yslow ranking 15 What is left in Pagespeed 15 What is left in Yslow 16 Conclusion 17

3 Test environment Test server configuration A virtual machine (Xen) with 4 cores & 4 GB CPU: 4 * 5650 (2.67 Ghz) core on a virtual machine RAM: DDR3, 4 Go Server software setup Kernel: Linux xen Demostore Apache: Mysql: a APC: Test box FunkLoad a r54261

4 Firebug testing Demostore result A basic demostore loads in 2,61s on a good browser (Firefox) running on a good station (i7, 6 Go, Windows 7). On the first line of Firebug test, the full load weights 570,9 Ko of data and we can see that the core loads in 483 ms. The server is basically optimized, involving a decent configuration and APC.

5 Nitrostore result The Nitrostore (demostore with Nitrogento installed) core loads in 80ms. Actually it s a static HTML file that is served instead of the HTML resulting from a PHP computation. This is 400 ms less and far less server CPU load since it s only a static file to serve instead of a PHP run. We can also see the full page loads in 1,227s which is due also to the sprite feature, the Etags & expires settings and the minifying if CSS/JS. (The strange looking file names are the compressed & minified CSS / JS) The whole economy is ~1,4 seconds and 279 Ko. Time to load the homepage: 53% Data transferred: 47%

6 Funkload test The scripts Homepage test configuration [main] title=demostore Home Hammering description=demostore homepage hammering url= [test_simple] description=access %(nb_time)s times the main url nb_time=1 [credential] host=localhost port=8007 [monitor] hosts=localhost [localhost] port=8008 description=the benching and benched machine [ftest] log_to = console file log_path = simple test.log result_path = simple test.xml sleep_time_min = 0 sleep_time_max = 0 [bench] cycles = 10:30:50:70:90:110:130:150:170:190:200 duration = 60 startup_delay = 0.01 sleep_time = 0.01 cycle_time = 130 log_to = log_path = simple bench.log result_path = simple bench.xml sleep_time_min = 0 sleep_time_max = 0.5 Full scenario test configuration [main] title=demostore Full Hammering description=demostore Full visit benchmark url= [test_demostore_test_script] description=access %(nb_time)s times the main url nb_time=1 [credential] host=localhost port=8007 [monitor] [localhost] port=8008 description=the benching machine [ftest] log_to = console file log_path = demostore_test_script test.log result_path = demostore_test_script test.xml sleep_time_min = 0 sleep_time_max = 0 [bench] cycles = 10:20:30:40:50 duration = 200 startup_delay = 0.01 sleep_time = 0.01 cycle_time = 240 log_to = file log_path = demostore_test_script bench.log result_path = demostore_test_script bench.xml sleep_time_min = 0 sleep_time_max = 0.5

7 Homepage test explanations The principles are to test with 10 simultaneous visitors all starting the same scenario at the same time. In time, the different users will succeed to achieve N scenario run. Those scenarios are simple fetch, meaning we don t care to download all static resources; we only load the homepage php/html. The second cycle then involves 30 users, and then 50, all the way up to 200 simultaneous users. Between each cycles, a wait time of 130 seconds is made for the server to return to a normal state and load. This is made to avoid cache or swap flush, apache daemon stopping or parasites in the Bench. Even if using the most exhaustive list of precautions, a single bench cannot be considered as realistic. We did 30 runs to get a reliable statistic result. The Funkload test scripts & configurations are available to download so you can reproduce the tests by yourself ( Full run scenario explanations This test is made in the same way as the basic homepage loading except that the scenario is more complicated and involves a homepage load, a product load, a search engine use, a category load, the fact that a user put a product in the cart and then gets back to navigation and checkout. This is a very CPU intensive test. Goals The goal is to test the Full Page Cache (FPC) capability and limits. Basically, when active, the FPC is serving static pages instead of requiring any kind of PHP interpretation. This is lightning fast since almost no operation is required to serve a static page. When the user put something in its cart or if a special dynamic data is present on the page, the FPC can t do its job and then, the performances get almost back to normal. In the Full run scenario, the product add to cart arrive very soon (it s the 5 th operation on 25) so this can be considered the worst case scenario.

8 Funkload results Homepage Homepage delivered per second Successfull Pages Per Second served (the higher the best) Demostore Pages Served Per Second Nitrostore Pages Served Per Second Homepage loading time (average & max) 18 Full scenario average and max load time (the lowest the best) Demostore worst load time Demostore average load time Nitrostore worst load time Nitrostore average load time

9 APDEX indices (User experience rating) Successfull Pages Per Second served (the higher the best, < 0,6 : bad experience) 1 0,8 0,6 0,4 0, Demostore Apdex (user experience) indice Nitrostore Apdex (user experience) indice Explanations We can easily see the difference. First, the number of pages served per second reaches 300 with a Nitrostore whereas the demostore is delivering a maximum of 20 pages per second. This is a time 15 factor. On very high load (180+ simultaneous users), the demostore crashed where the Nitrostore was still easily delivering pages. (We could have tested it further to see the limit, but it has not been done yet). Next come the average and maximum load time of the home page. When under heavy load (180, after it became unstable), the demostore can reach up to 16,5 second and on average, homepage load takes 9 second. The Nitrostore never deliver the home page HTML in more than 1 second and on average, under heavy load, second. Last but not least, the APDEX indices of the tests. It s recommended to be always at a better level than 0,9 and never less than 0,6 since it s a really dirty user experience. More can be found about Apdex here. We can see that, during the load test, the demostore is delivering a poor user experience as soon as 40 simultaneous users come in whereas the Nitrostore is still delivering a perfect (1) Apdex indice, even with 200 simultaneous users. Considering the Homepage usually represent 40% of the whole traffic, this is good to get! Results 15x more homepage delivered in the same time 8 time less CPU load Homepage HTML delivered 30 times faster (0,375 ms on average instead of 9s under heavy load) Continuous top quality user experience rating (especially visible under heavy load)

10 Raw data Want to make your own graph or make a measure of yours and compare? Here are the raw data: Simultaneous users SPPS demostore max SPPS nitrostore max SPPS demostore med 0 4,45 17,30 20,72 21,32 20,17 19,27 20,75 21,08 20,45 10,34 3,80 SPPS nitrostore med Apdex demostore 1 0,878 0,808 0,506 0,5 0,498 0,479 0,269 0,054 0,02 0,01 0,005 Apdex Nitrostore loadtime demostore 0 1,822 1,857 2,162 3,031 4,163 4,909 5,861 6,693 7,536 8,6 12 loadtime nitrostore 0 0,016 0,019 0,050 0,044 0,058 0,104 0,166 0,215 0,282 0,329 0,375 worst time demostore 0 4,1 4,2 4,3 5,1 6,4 7,2 9,102 11,2 12,55 14,2 16,41 worst time nitrostore 0 0,052 0,074 0,135 0,279 0,412 0,45 0,489 0,76 0,813 1,006 1,023

11 Full scenario Pages delivered per second Max Successfull Pages Per Second served (the higher the best) Demostore Max Pages served per second Nitrostore Max Pages served per second Average & max time to load a page during the complete scenario Full scenario average and max load time (the lowest the best) Demostore worst load time Nitrostore worst load time Demostore average load time Nitrostore average load time

12 APDEX indices (User experience rating) Successfull Pages Per Second served (the higher the best, < 0,6 : bad experience) 1 0,8 0,6 0,4 0, Demostore Apdex (user experience) indice Nitrostore Apdex (user experience) indice Explanations On the Page served per second, we can see that the Nitrostore is delivering two times more pages than the Demostore. Obviously, since the product is put in the cart very early in the scenario, the benefits of FPC are quickly cut. This is especially true on the APDEX test, only a 20% difference is visible under a 40/50 simultaneous user load. It s still good to take and we are making a lot of work actually on this, we should get 35% soon The Average and worst load time are also better in the Nitrostore, with also ~20% better timings. Results Two times more pages served. 20% better on APDEX test and load time Raw data Want to make your own graph or make a measure of yours and compare? Here are the raw data: Simultaneous users SPPS demostore max SPPS nitrostore max SPPS demostore med 0 11,85 14,065 13,945 14,055 14,06 SPPS nitrostore med 0 15,57 14,71 14,83 16,08 15,67 Apdex demostore 1 0,997 0,832 0,698 0,607 0,549 Apdex Nitrostore 1 0,998 0,837 0,718 0,682 0,634 loadtime demostore 0 0,627 1,207 1,925 2,606 3,331 loadtime nitrostore 0 0,29 0,942 1,390 2,103 2,601 worst time demostore 0 8,502 9,108 9,595 9,608 10,258 worst time nitrostore 0 5,918 7,058 7,963 8,125 8,331

13 Yslow / Pagespeed / GTmetrix tests What is it? Google and Yahoo published some testing plugins for Firefox. The eldest one is Yslow, a quite decent all arounder that Yahoo keep maintaining. Google came up with Pagespeed a couple of years later. The testing rules are not exactly the same but the overall goal is the same: make the sites load faster, optimize navigation and give the world the taste for best practices. A very clever company came with the idea that: 1. Not everybody is using Firefox 2. This can be pretty convenient to be able to test a website from anywhere without having to install anything and to be able to compare at once Yslow & Pagespeed results They decided to create a site called that allows for a full testing of Yslow & Pagespeed rule set at once. The only problem is that this site is hosted in US and thus the transfer speed and load time are dependent on the distance between the location of the tested server and GTmetrix servers. Many different points are evaluated in theses plugins, CDN usage, CSS grammar, browser cache leveraging etc. The plugins then give you a rating for the tested site. We know that Google take load speed as a very important ranking factor, it s very likely that also all other factor are used also to rate a site. Ths rating can sometimes change a bit since the rule set are changing. Basic demostore We can see that a Basic demostore is not getting very good ratings.

14 More details on demostore ranking Pagespeed notation Yslow ranking Here, only problems are reported. The good scores have been stripped off since they don t help us to get better. The items are quite explicit by themselves, so we know where to act.

15 Nitrostore The Nitrogento plugin has been made to address almost all of Yslow & Pagespeed requests since they are very legitimate. We did our best to help your site achieve the best possible ranking in those two performance benchmark. Here is a store with all Nitrogento performances boost enabled & configured: In Yslow rule set, you have to declare your CDN (either in Firefox or in the GTmetrix site) if you want it to be considered, if not, the ranks should be displayed 94/92%. More details on demostore ranking Pagespeed notation Yslow ranking What is left in Pagespeed Here we can see that the optimizations that are not 100% are mainly not doable. Many points are doing very better, the rankings overall is far better. Actually there are two or three optimizations that we intend to put in the next release, like inline small CSS, optimize image, specify image dimensions.

16 The minify HTML can be made 100% but it s not worth the CPU cost. This optimization is either 100% for minify, either more throughput on the front server. We ll probably come up with a solution but actually, using minify HTML cost a bit of performances. Remove unused CSS and choose efficient CSS selector is something that should not happen in your sites if you develop properly. What is left in Yslow Make fewer HTTP requests Tricky point. Either you spread your resources across a large number of hosts and leverage browser parallelizing download capability (CDN multiple simultaneous GET), but this imply to have many hostnames (like cdn1.nitrostore.com, cdn2.nitrostore.com etc.) but this make more DNS requests. Or you use less hosts, thus less DNS resolutions. The good way to proceed is to take the amound of static files, divide it by 8 (the average number of simultaneous download capability of the average browser) and fix this amount of CDN name pointers. For example, if you take a demostore, it has 49 HTTP GET requests, one for the main HTML file and then CSS/JS/images = 48. Divide this number by 8, this make 6, so the best way to proceed is to have 6 hostname for your CDN to optimize resource download. Actually 5 would be the best plus the basic www hostname, making 6 at all. The problem is, is you make more than 3 in Yslow you get a penalty, if you don t put all static resources on a CDN you get a penalty If you want speed, proceed like said here, if you care about your Yslow ranking, make CDN1/2/3 and no more, Nitrogento will automatically spread your resources accordingly. Make fewer HTTP requests. Once you have your Sprite, you already have divided your requests by almost two on a demostore, going down from 49 to 28 It s possible to do even less request, but by design, Nitrogento can t do more about this. Already 28 requests is not that much. Minify JS and CSS, it s done but probably the rating is not maximum since we could do more about it like stripping all carriage return and stuff like this. This won t really help a lot but render the code unreadable so that s not really a choice here. Finally, put all JS at bottom, this is not always possible and here 1 is left before bottom, but here again, this is a by design point, we can t do better here.

17 Conclusion When looking at this benchmark, we can see that we did our best. Some efforts are still to be made, some minor gain could be added but the very big point is already there. Nitrogento will make your site way faster, your visitor will get the page at a really increased rate and all studies show that a better speed is directly related to: a better SEO, a more efficient SEM, a far more efficient sale process, less cart drop, less users bailing out, more returning users etc. There are a lot of studies around those points so you ll easily find why Nitrogento can help you achieve better results with your Magento Website! When doing a large ing, the homepage is the most loaded page and this one will be far more efficient using Nitrogento, leaving also more resources to your server(s) to take care of the other users going deeper in your site and completing a buy process. Your website will perform faster, your server load will be lowered, you will consume less bandwidth and all this will lead to more sales. The figures shown here are based on a demostore but we ll be really glad to hear about your stories and gain, to see how Nitrogento improved your Magento experience and your visitor conversion rate, keep us posted! About Nitrogento is a Magento extension edited by NBS System on an original idea from the staff, specifications and project development process has been handled by NBS System. The Extension has been developed by Mathieu Bouchot (Academy E commerce) and Aymeric Aytamer (Agence DnD). Magento and Magento Inc are trademarks from the Magento Inc. Company. This document is free to copy and spread on any support, part can be quoted as long as the sense is not altered. No modification can be done without any direct agreement from NBS System NBS System, All right reserved. Benchmark Nitrogento from NBS System is made available following the terms of the Creative Commons Paternity No Modification 3.0 Unported licence.