MBB Robot Crawler Data Report in 2014H1

Size: px

Start display at page:

Download "MBB Robot Crawler Data Report in 2014H1"

Myles Burke
6 years ago
Views:

1 MBB Robot Crawler Data Report in 2014H1

2 Contents Contents 1 Introduction Characteristics and Trends of Web Services Increasing Size of Web Pages Increasing Average Number of Access Requests per Page Images Taking the Highest Proportion of Contents on Each Page, Followed by JS Texts Different Page Characteristics in Different Areas Increasing Average Number of Hosts per Page Growing Trend of OTT Privacy Encryption Characteristics and Trends of Video Services Basic Characteristics of Youku and YouTube Most YouTube 720p Videos Featuring a Bit Rate of 1.5 to 3 Mbit/s Stable Average Video Duration Videos Shorter Than 5 Minutes Taking a Proportion of 50% Increasing Proportions of HD Videos and Ultra HD Videos Possible Impact of YouTube Video Stream Encryption on Video Optimization Solutions Characteristics and Trends of Microblog Services Common Users Taking a Proportion of Over 90% Individual VIP Users Having the Largest Influence Chinese Characters per Microblog On Average % of Microblogs Containing Images % of Microblogs Being Forwarded Appendix About This Report Data Sources Contact Disclaimer A Acronyms and Abbreviations ii

3 Figures Figures Figure 2-1 Trend of the average size of transmitted content per iphone-based page... 3 Figure 2-2 Trend of the average size of transmitted content per PC-based page... 3 Figure 2-3 Trend of the average number of access requests per iphone-based page... 4 Figure 2-4 Trend of the average number of access requests per PC-based page... 4 Figure 2-5 Proportion of each type of content transmitted for each iphone-based page in the Top100 websites on average... 5 Figure 2-6 Proportion of each type of content transmitted for each iphone-based page in the Top5000 websites on average... 6 Figure 2-7 Proportion of each type of content transmitted for each PC-based page in the Top100 websites on average... 6 Figure 2-8 Proportion of each type of content transmitted for each PC-based page in the Top websites on average... 7 Figure 2-9 Characteristics of iphone-based pages in the Top50 websites in different areas (KB) (April, 2014).. 7 Figure 2-10 Trend of the average number of hosts per page... 8 Figure 2-11 Trend of the proportion of HTTPS connections per page on average (%)... 9 Figure 3-1 Resolutions and basic characteristics of videos provided by Youku and YouTube Figure 3-2 Bit rates of YouTube 720p videos Figure 3-3 Trend of the average video durations in Youku and YouTube Figure 3-4 Proportions of videos of different durations in YouTube in 2014H Figure 3-5 Trends of the proportions of HD videos and ultra HD videos provided by Youku Figure 3-6 Video streams unencrypted in an earlier version of YouTube Figure 3-7 Video streams encrypted in the latest version of YouTube Figure 4-1 Proportions of different types of sampled microblog users Figure 4-2 Average number of microblogs posted by each type of user Figure 4-3 Average number of fans for each type of user Figure 4-4 Trend of the average number of Chinese characters per microblog for each type of user Figure 4-5 Trend of the proportion of microblogs containing images under each type of user Figure 4-6 Trend of the proportion of microblogs forwarded under each type of user iii

4 1 Introduction 1 Introduction This report mainly consists of three parts: Chapter 2 "Characteristics and Trends of Web Services", which describes the size trend, proportions of different types of contents, regional differences, and number of Hypertext Transfer Protocol Secure (HTTPS) connections for each page on average at Alexa Top-listed websites. Alexa.com is a website at which the rankings of global websites are published. Chapter 3 "Characteristics and Trends of Video Services", which describes the code rate trend, duration distribution, proportion of high definition (HD) videos, and encryption methods for videos provided by Youku and YouTube. Chapter 4 "Characteristics and Trends of Microblog Services", which describes the average number of microblogs, average number of fans, average number of times that users are followed, average number of Chinese characters per microblog, proportion of images, and forwarding behaviors for different types of Sina Weibo users. Data used in this report comes from web, video, and microblog service samples collected by the MBB Robot. Based on the mass samples, this report aims to describe the characteristics and trends of web, video, and microblog services objectively. However, it is not guaranteed that information in this report is accurate and complete. Key findings of this report are as follows: Web services The average page size is increasing year by year at an annual growth rate of about 28%. Among the traffic generated from page content transmission, 65% is image data and 23% is JavaScript (JS) text data. Due to continuous improvement of the distributed Internet architecture, the number of hosts contained in a single page is increasing. For privacy security concern, the number of HTTPS connections contained in a single page is also increasing. Video services In the new version of YouTube client, video streams can be transmitted through HTTPS. This affects the video solution. The proportion of video sources that provide HD and ultra HD videos is increasing. The average video duration is about 12 minutes, and half of the videos are shorter than 5 minutes. 1

5 1 Introduction Microblog services Individual VIP users take a lower proportion. However, they publish more microblogs, have more fans, and are followed more than other types of users, and therefore have greater influence. Currently, a single microblog contains 71 Chinese characters on average. 74% of microblogs contain images and 57% microblogs are forwarded. 2

6 Jan-12 May-12 Sep-12 Jan-13 May-13 Sep-13 Jan-14 May-14 Transfer Size(KB) Jan-12 Apr-12 Jul-12 Oct-12 Jan-13 Apr-13 Jul-13 Oct-13 Jan-14 Apr-14 Transfer Size(KB) MBB Robot Crawler Data Report in 2014H1 2 Characteristics and Trends of Web Services 2 Characteristics and Trends of Web Services 2.1 Increasing Size of Web Pages Figure 2-1 Trend of the average size of transmitted content per iphone-based page Top 100 Top 5000 Figure 2-2 Trend of the average size of transmitted content per PC-based page Top 100 Top

7 Jan-12 Apr-12 Jul-12 Oct-12 Jan-13 Apr-13 Jul-13 Oct-13 Jan-14 Apr-14 Requests Jan-12 Apr-12 Jul-12 Oct-12 Jan-13 Apr-13 Jul-13 Oct-13 Jan-14 Apr-14 Requests MBB Robot Crawler Data Report in 2014H1 2 Characteristics and Trends of Web Services The average size of each page in the Top5000 websites is increasing at an annual growth rate of 28%, and has reached 940 KB by June, The average size of each page in the Top100 websites is increasing at an average annual growth rate of 20%, and has reached 520 KB by May, The average size and growth rate of each page in the Top100 websites are lower than those of each page in the Top5000 websites. This is because websites that rank higher are more inclined to provide pages with fewer elements for mobile devices in most cases. The average size per PC-based page is increasing. The average size of each page in the Top websites is increasing at an average annual growth rate of 30%, and has reached 1.8 MB by June, Some websites do not provide mobile device-based pages. Therefore, the growing size of PC-based pages also drives the growth in the size of iphone-based pages to some extent. 2.2 Increasing Average Number of Access Requests per Page Figure 2-3 Trend of the average number of access requests per iphone-based page Top 100 Top 5000 Figure 2-4 Trend of the average number of access requests per PC-based page Top 100 Top

8 2 Characteristics and Trends of Web Services The number of access requests per iphone-based page does not increase significantly. The number of access requests per PC-based page in the Top100 websites, however, is increasing significantly at an annual growth rate of about 14%. The average number of access requests per page is increasing at a lower growth rate than the average size per page, which means that the average size per element on web pages is increasing. 2.3 Images Taking the Highest Proportion of Contents on Each Page, Followed by JS Texts The following figures show the proportion of each type of content transmitted for each page in Alexa Top-listed websites in 2014H1 from different dimensions: Figure 2-5 Proportion of each type of content transmitted for each iphone-based page in the Top100 websites on average CSS 4% Html 7% Other 0% JS 24% Image 65% 5

9 2 Characteristics and Trends of Web Services Figure 2-6 Proportion of each type of content transmitted for each iphone-based page in the Top5000 websites on average Html CSS 5% 5% JS 23% Image 66% Figure 2-7 Proportion of each type of content transmitted for each PC-based page in the Top100 websites on average Html 4% CSS 3% Other 16% JS 20% Image 57% 6

10 2 Characteristics and Trends of Web Services Figure 2-8 Proportion of each type of content transmitted for each PC-based page in the Top websites on average CSS 3% Html 4% Other 9% JS 17% Image 67% According to analysis on the proportion of each type of transmitted page content in 2014H1, among iphone-based pages, 65% of the traffic is generated from image transmission and 23% of the traffic is generated from JS text transmission. In this report, it is assumed that the average size of iphone-based pages in 2014H1 is 880 KB, the size of images per page is 580 KB and that of JS data per page is 200 KB. 2.4 Different Page Characteristics in Different Areas Figure 2-9 Characteristics of iphone-based pages in the Top50 websites in different areas (KB) (April, 2014) Other Html CSS JS Image 7

11 Jan-12 Mar-12 May-12 Jul-12 Sep-12 Nov-12 Jan-13 Mar-13 May-13 Jul-13 Sep-13 Nov-13 Jan-14 Mar-14 May-14 MBB Robot Crawler Data Report in 2014H1 2 Characteristics and Trends of Web Services Russia features the largest average page size, which is 830 KB. Indonesia features the smallest average page size, which is only 390 KB. The average page size in most other countries is about 600 KB. Of all countries, images in websites in China take the highest proportion, whereas JS texts take the lowest proportion. Web page characteristics vary in different countries due to factors, such as the web service development history, Internet maturity, and smartphone popularity. However, no fixed relationships have been found between the preceding factors and the characteristics. 2.5 Increasing Average Number of Hosts per Page Figure 2-10 Trend of the average number of hosts per page PC Top100 PC Top iphone Top 100 iphone Top From 2012 till now, the average number of hosts per page has been increasing at an annual growth rate of about 20%. Due to the continuous improvement of the Internet architecture, the number of hosts per page increases if data, such as images and JS texts, are stored on dedicated servers. 8

12 2 Characteristics and Trends of Web Services 2.6 Growing Trend of OTT Privacy Encryption Figure 2-11 Trend of the proportion of HTTPS connections per page on average (%) PC Top iphone Top5000 This section uses the trend of OTT privacy encryption for iphone-based pages as an example. The average proportion of HTTPS connections per iphone-based page is increasing year by year, from 5.5% in January, 2012 to 12.8% in June, 2014 at an annual growth rate of about 40%. The average proportion of HTTPS connections per PC-based page is rapidly increasing, too, at an annual growth rate of about 50%. It is a general tendency to encrypt data in order to protect OTT privacy. The increasing number of HTTPS connections affects some solutions for Hypertext Transfer Protocol (HTTP) traffic identification and optimization. 9

13 3 Characteristics and Trends of Video Services 3 Characteristics and Trends of Video Services 3.1 Basic Characteristics of Youku and YouTube Figure 3-1 Resolutions and basic characteristics of videos provided by Youku and YouTube Data source: Huawei mlab The latest version of YouTube provides two optional resolutions, including 240p and 360p, for UMTS, and provides two optional resolutions, including 360p and 720p, for an LTE and WiFi. The H.264 coding format remains unchanged. Therefore, the average bit rates for videos of different resolutions provided by the two websites are distributed in the same interval, and are generally stable from 2012 till now. 10

14 Bitrate(Kbps) MBB Robot Crawler Data Report in 2014H1 3 Characteristics and Trends of Video Services 3.2 Most YouTube 720p Videos Featuring a Bit Rate of 1.5 to 3 Mbit/s Figure 3-2 Bit rates of YouTube 720p videos According to the scatter chart, most video bit rates are between 1.5 Mbit/s and 3 Mbit/s within an interval ranging from 65% of the average video bit rate 2.2 Mbit/s to 135% of the average video bit rate 2.2 Mbit/s. 95% of 720p videos provided by YouTube feature a bit rate lower than 3 Mbit/s. Videos feature a bit rate of 1 to 2 Mbit/s and those feature a bit rate of 2 to 3 Mbit/s take a proportion of 45%, respectively. 11

15 3 Characteristics and Trends of Video Services 3.3 Stable Average Video Duration Figure 3-3 Trend of the average video durations in Youku and YouTube Youku The average duration of videos provided by YouTube is about 700s, and has remained stable since The average duration of videos provided by Youku is increased from 300s in 2012 to the current 700s. It is speculated that the increase is caused by the introduction of mass serial plays, especially American serial plays, and the expiry of the copyrights on some movies and serial plays in

16 3 Characteristics and Trends of Video Services 3.4 Videos Shorter Than 5 Minutes Taking a Proportion of 50% Figure 3-4 Proportions of videos of different durations in YouTube in 2014H1 5% 3% 7% 15% 25% 45% <1min 1-5mins 5-10mins 10-30mins 30-60mins >60mins Most videos are short. Among samples in 2014H1, videos shorter than 5 minutes take a proportion of 52%, and those shorter than 10 minutes take a proportion of 77%. From 2012 till now, videos shorter than 5 minutes have been decreasing, whereas 5- to 10-minute videos have been increasing slightly. 13

17 3 Characteristics and Trends of Video Services 3.5 Increasing Proportions of HD Videos and Ultra HD Videos Figure 3-5 Trends of the proportions of HD videos and ultra HD videos provided by Youku HD Ultra HD The proportion of HD videos provided by Youku has been growing rapidly since 2012, and has reached 85%. The proportion of ultra HD videos provided by Youku is growing at an annual growth rate of about 50%. This means that one out of two videos can be played in ultra HD mode. 3.6 Possible Impact of YouTube Video Stream Encryption on Video Optimization Solutions Video uniform resource locators (URLs) and contents can be identified when video streams are not encrypted in an earlier version of YouTube. 14

3 Characteristics and Trends of Video Services Figure 3-6 Video streams unencrypted in an earlier version of YouTube Data source: Huawei mlab Video URLs and contents cannot be identified when video

Figure 3-7 Video streams encrypted in the latest version of YouTube Data source: Huawei mlab The latest version of Android-based YouTube (version number: 5.

18 3 Characteristics and Trends of Video Services Figure 3-6 Video streams unencrypted in an earlier version of YouTube Data source: Huawei mlab Video URLs and contents cannot be identified when video streams are encrypted in the latest version of Android-based YouTube. Figure 3-7 Video streams encrypted in the latest version of YouTube Data source: Huawei mlab The latest version of Android-based YouTube (version number: 5.6) has adopted the video stream encryption mechanism, and user interaction information has been encrypted since earlier versions. This means that only images are transmitted through HTTP. Video stream encryption has significant impacts on most of the current video solutions: Most video solutions are designed based on video URL matching or video file content matching. After video stream encryption is implemented, however, video URLs and video files become invisible to pipes, and therefore these video solutions do not function. 15

19 4 Characteristics and Trends of Microblog Services 4 Characteristics and Trends of Microblog Services 4.1 Common Users Taking a Proportion of Over 90% Figure 4-1 Proportions of different types of sampled microblog users Individual users 91.43% Institutional VIP users 0.68% Individual VIP users 0.71% Expert users 7.18% Individual users take a proportion of over 90%. Most individual users are common users. Individual users also include some users whose have not obtained VIP authentication but have a large number of fans. 16

20 4 Characteristics and Trends of Microblog Services Expert users are active individual users who have obtained expert identity and have many more fans than common individual users. Individual VIP users are those authenticated by Sina Weibo. Most individual VIP users are users who have significant influence in various industries and have a large number of fans. Institutional VIP users are authenticated organizations, such as government agencies, enterprises, and websites. 4.2 Individual VIP Users Having the Largest Influence Figure 4-2 Average number of microblogs posted by each type of user Individual users Expert users Individual VIP users Institutional VIP users Figure 4-3 Average number of fans for each type of user Individual users Expert users Individual VIP users Institutional VIP users 17

21 Chinese characters (number) MBB Robot Crawler Data Report in 2014H1 4 Characteristics and Trends of Microblog Services Among the four types of microblog users, individual VIP users take the lowest proportion. However, they publish more microblogs, have more fans, and are followed more than other types of users, and therefore have greater influence Chinese Characters per Microblog On Average Figure 4-4 Trend of the average number of Chinese characters per microblog for each type of user Individual users Individual VIP users The average number of Chinese characters per microblog is increasing. The average number of Chinese characters per microblog for individual VIP users is greater than that for individual users. The average number of Chinese characters per microblog now is 71 based on the user proportions and through weight calculation. 18

22 Proportion (%) MBB Robot Crawler Data Report in 2014H1 4 Characteristics and Trends of Microblog Services % of Microblogs Containing Images Figure 4-5 Trend of the proportion of microblogs containing images under each type of user Individual users Individual VIP users The proportion of microblogs containing images has been slightly increasing, showing a roughly stable trend on the whole. Individual VIP users publish more microblogs containing images than other types of individual users. The proportion of microblogs containing images now is 72.45% based on the user proportions and through weight calculation. 19

23 Proportion (%) MBB Robot Crawler Data Report in 2014H1 4 Characteristics and Trends of Microblog Services % of Microblogs Being Forwarded Figure 4-6 Trend of the proportion of microblogs forwarded under each type of user Individual users Individual VIP users The proportion of common users who forward other users' microblogs is slightly higher than that of individual VIP users who forward other users' microblogs. Nearly half of the current microblogs are forwarded ones on the whole. The proportion of forwarded microblogs now is 56.78% based on the user proportions and through weight calculation. 20

24 5 Appendix 5 Appendix 5.1 About This Report This report is prepared by Huawei mlab. Information provided in this report is for reference only. This report is prepared based on analysis of web, video, and microblog samples collected by the MBB Robot. Based on the mass samples, this report aims to describe the characteristics and trends of web, video, and microblog services objectively. However, it is not guaranteed that information in this report is accurate and complete. 5.2 Data Sources Rankings of global websites at Data collected by the MBB Robot, which consists of the following parts: The server database collects the service characteristics of industrial OTTs, such as Youku, YouTube, and Sina Weibo, through crawler scripts that are continuously executed, and completes data of the MBB Robot website by using some open databases containing page characteristics. The query system provides the function of querying the characteristics and trends of different types of services from various dimensions, including the time dimension (main dimension) and other dimensions such as website, area, and user. The query system can also be used to predict future network traffic. 5.3 Contact The author of this report iswang Feng whose employee ID is and is wangfeng.sh@huawei.com. You can contact Huawei mlab personnel ((MBB lab) through MBBlab@huawei.com. 21

25 5 Appendix 5.4 Disclaimer This report is prepared by Huawei mlab. Information provided in this report is for reference only. Due to the rapid development of OTT services, Huawei mlab reserves the rights to update this report in the future, and does not bear liabilities for any impacts that may arise due to the update. This report cannot be used as a basis for investment or research decision-making, or as a basis or proof for ethic, responsibility, or law purpose, either expressed or implied. Huawei mlab may supplement, correct and revise information in this report without notice. Every effort has been made in the preparation of this report to ensure accuracy of the information. Huawei mlab does not bear liabilities for any direct or indirect investment profit and loss due to information in this report. No part of this report may be reproduced, transmitted, published in any form or by any means without prior written consent of Huawei mlab. Shall information in this report needs to be quoted, it must be clearly stated that the information is prepared by Huawei mlab, and the quoted information must not be abbreviated or modified deliberately against its original meaning. 22

26 A Acronyms and Abbreviations A Acronyms and Abbreviations H HD HTML HTTP HTTPS high definition HyperText Markup language HyperText Transfer Protocol HyperText Transfer Protocol Secure J JS JavaScript U URL uniform resource locator 23

Nigerian Telecommunications (Services) Sector Report Q3 2016

Nigerian Telecommunications (Services) Sector Report Q3 2016 24 NOVEMBER 2016 Telecommunications Data The telecommunications data used in this report were obtained from the National Bureau of Statistics