Economics of the Interconnection Data Centre (IDC) Industry

Transcription

1 Economics of the Interconnection Data Centre (IDC) Industry 19 November 2015

2 Project Team Prof. Dr. Frank Maier-Rigaud Dr. Christopher Milde André Selke Unter den Linden Berlin, Germany Tel: Fax: Square de Meeûs Brussels, Belgium Tel: +32 (0)

3 This report was prepared at the request and with the support of Interxion Holding N.V. CONFIDENTIALITY We understand that the maintenance of confidentiality with respect to our clients plans and data is critical to their interests. NERA Economic Consulting rigorously applies internal confidentiality practices to protect the confidentiality of all client information. Similarly, our approaches and insights are proprietary and so we look to our clients to protect our interests in our proposals, presentations, methodologies and analytical techniques. Under no circumstances should this material be shared with any third party without the prior written consent of NERA Economic Consulting. NERA Economic Consulting NERA Economic Consulting

4 Contents Contents EXECUTIVE SUMMARY INTRODUCTION BACKGROUND TO THE IDC INDUSTRY THE INTERNET AS A NETWORK OF NETWORKS KEY INTERNET NETWORKS AND PLAYERS INTERCONNECTION TECHNOLOGIES THE INTERCONNECTION DATA CENTRE INDUSTRY COLOCATION SERVICES IDCS AND OTHER TYPES OF DATA CENTRES INTERCONNECTION SERVICES INTERNET EXCHANGES AND IDCS COMMUNITIES OF INTEREST FOR IDC SERVICES Electronic Communications Networks Digital Content Delivery Cloud Services Financial Services Advertising E-Commerce SUMMARY AND CONCLUSIONS ASPECTS OF THE ECONOMICS OF THE IDC INDUSTRY HORIZONTAL ASPECTS OF IDC COMPETITION Network Effects in IDC Services Magnet Customers Market Dynamics with Network Effects and Magnet Customers VERTICAL ASPECTS OF IDC COMPETITION LITERATURE GLOSSARY OF TERMS NERA Economic Consulting

5 List of Tables List of Tables TABLE 1 AMS-IX TRAFFIC VOLUME BY IDC TABLE 2 AMS-IX TRAFFIC SHARES OF IDC PROVIDERS IN AMSTERDAM METRO AREA List of Figures FIGURE 1 A STAR NETWORK... 1 FIGURE 2 THE INTERNET AS "NETWORK OF NETWORKS"... 2 FIGURE 3 THE INTERNET AND ITS IMPORTANT PLAYERS... 5 FIGURE 4 COLOCATION SERVICES OF IDCS... 9 FIGURE 5 SERVER RACKS INSIDE A COLOCATION FACILITY FIGURE 6 CROSS-CONNECTS AT IDCS FIGURE 7 INTERNET EXCHANGES ARE HOUSED AT IDCS FIGURE 8 IDCS AND THE HISTORY (AND PRESENT) OF THE DIGITAL ECONOMY FIGURE 9 VERTICAL INDUSTRY STRUCTURE OF PROVISION OF CONTENT TO CAUS FIGURE 10 ILLUSTRATION OF VALUE OF AN IDC NETWORK WITH IMPORTANT AND LESS IMPORTANT PEERS FIGURE 11 AVERAGE FIBRE CROSS-CONNECT PRICE BY METRO AREA (2013) FIGURE 12 THE FIVE LARGEST IXS IN EUROPE AND IN THE UNITED STATES NERA Economic Consulting

6 Executive Summary Executive Summary NERA Economic Consulting ( NERA ) has been commissioned by Interxion Holding N.V. ( Interxion ) to prepare a report ( White Paper ) on the economics of the interconnection data centre ( IDC ) industry with a focus on Europe. The purpose of this White Paper is to inform interested parties on salient features of this industry and to contribute to the understanding of a sector that is highly important to the functioning of the digital economy. The White Paper is based on publicly available information and on insights gained in discussions with various industry participants. The modern digital economy is characterized by high data volumes and the necessity for reliable high speed data exchange. Businesses such as Netflix, providing video on-demand, social and messaging networks that enable text, photo and video communications such as WhatsApp, Snapchat and YouTube, location based mobile applications such as Foursquare, cloud computing providers such as Amazon Web Services and Microsoft Azure, or search providers such as Google and Microsoft Bing all rely on high performance data transmission and data exchange between their own networks and the networks used to reach their customers or partners in order to provide services. The same is true for other industries e.g. high frequency trading, which relies on an extremely fast exchange of trading data from stock exchanges to traders or brokers, and online advertising, which relies on high speed exchange of data from content and service providers such as search engines and publishers to advertising platforms. High speed and high quality data exchange between the networks of different companies can be technically achieved by direct physical connections via fibre optic or copper cables over short distances between the servers of two networks (measured in metres rather than kilometres). Such direct physical interconnections are referred to as Direct Circuits or Cross- Connects. Another technology for data exchange is a Network Switch, which operates as an open exchange platform where many networks can simultaneously interconnect to many others. While interconnection via such a Network Switch has cost advantages when an organisation seeks to exchange data with many networks, each of which generates only limited traffic, it cannot provide the same performance in data exchange in terms of quality of service ( QoS ) and security as Cross-Connects. Enabling high-performance interconnection through Cross-Connects is the core value adding service offered by the IDC industry in Europe. As a prerequisite for Cross-Connect services, the IDC industry has to offer colocation services, i.e. the provision of space, power, cooling, and security, which allows customers to place their server and networking equipment in close proximity of each other, and the ability to reach those servers in the data centre via telecommunications infrastructure to the data centre. As colocation services are offered also by the traditional data centre industry, the high-performance interconnection opportunities are the distinguishing feature of the IDC industry. Cross-Connect services are related to but distinct from the services of Internet Exchanges ( IXs ). IXs provide local interconnection via Network Switches. IX services can be considered as an alternative to Cross-Connects only to a limited degree and in practice rather function as an important complementary service to Cross-Connects. In Europe IX services are typically offered by organisations that are independent from IDC providers whereas in the US IX services are typically offered by the same firms offering also colocation services and NERA Economic Consulting i

7 Executive Summary Cross-Connects. Given the European focus of this White Paper, in the following IDC services denote colocation and Cross-Connect services but not IX services. The IDC and IX industries provide essential infrastructure and services to organisations seeking to interconnect their networks and computing resources. The availability of high quality IDC services in sufficient quantities and on competitive terms is critical to the present and future increasingly interconnected digital economy. Therefore it is warranted to take a closer look at the economic features of the interconnection industry. It is important to note that the IX and IDC services of interest in this context concern the physical interconnection of networks rather than the commercial agreements, such as transit and peering, that govern the data exchange between two interconnected organisations. This White Paper thus analyses the economics of the provision of such physical interconnection but not the economics of the commercial agreements such as transit and peering. In the following, first, (horizontal) competition between IDC providers within the same IDC market is discussed. Then the (vertical) relationship between the IDC market and the IX market is analysed. Two phenomena appear to be particularly salient for horizontal competition between IDC providers within an IDC market: network effects and magnet customers. First, IDC services exhibit network effects. As interconnection is the primary motivation for customers to colocate in an IDC the value of an IDC increases with the number of interconnection opportunities, i.e. the number of relevant and attractive (e.g. in terms of traffic volumes) other customers also present in the IDC. A group of customers for which interconnection with each other generates value is defined as a community of interest. Which other customers are relevant and attractive for a given customer for interconnection is thus determined by the community of interest to which a given customer belongs. For example, a content provider is interested in the presence of internet service providers ( ISPs ) that distribute the content to end users; enterprises outsourcing their IT needs are interested in the presence of cloud computing service providers; financial services firms such as traders and brokers are interested in the presence of other traders and brokers and stock exchanges etc. Second, within a given community of interest some customers are so important that once they are present in an IDC they attract many other customers. Such customers are often referred to as magnet customers. For example, in the financial services community a stock exchange or other electronic exchanges may be such magnet customers; a large content provider or Backbone (Tier 1) ISP may be a magnet customer for local ISPs serving the end-users in a region; large cloud service providers make an IDC attractive to enterprises interested in cloud-based IT solutions etc. Network effects and magnet customers have important implications for market dynamics. First of all, network effects and magnet customers make it particularly difficult for new or smaller competitors to compete with established competitors as they will have great difficulty offering an attractive network built around one or several magnet customers. IDC markets are thus characterized by high barriers to entry. Moreover, network effects tend to be selfreinforcing, i.e. large networks tend to expand quickly while smaller networks have more difficulty to grow or will even shrink, thus leading to higher market concentration. With strong network effects the market may even tip towards the largest provider resulting in monopoly. With less strong network effects several providers may coexist in the market. Also in this scenario, however, strong asymmetries in network size between providers may result in the dominance of the leading provider. NERA Economic Consulting ii

8 Executive Summary Likewise, the existence of magnet customers may confer important competitive advantages to established IDC providers. Due to their magnet quality, winning magnet customers is a key success factor when entering new markets. Established IDC providers in one market, e.g. those with a large global footprint, have considerable advantages in incentivizing magnet customers to use their IDC services in a new geographic market. For example, they could offer volume discounts across markets to international customers. The effects of such discounts can be so strong that players only active in the new market or players with a smaller global footprint may not be able to attract magnet customers. This may result in the established IDC provider eventually becoming the leading IDC provider in the new geographic market even if initially it did not offer the most attractive interconnection opportunities. Regarding the effects of vertical relationships on IDC competition, the analysis turns to the complementary relationship between the two types of interconnection services, Cross- Connects and IX services. A complementary relationship between several goods or services has similar implications as a vertical relationship, i.e. where one good is an input for the provision of another good. In particular, integration of firms across complementary products carries similar risks as vertical integration. One common concern of integration across complementary or vertically related products is that a leading firm in one market may be able to protect (and reinforce) its leading position in that market by integrating with a leading firm in another vertical/complementary market or by creating such a leading firm in another market through foreclosure. For example, a leading IDC provider may become active also in IX services. Given its role in the IDC market the provider may be able to establish an attractive IX solution, for example by providing incentives to its IDC customers to use the integrated IX services. Given the role of IXs as an important complement to IDC services the vertically integrated firm could then have the incentive to restrict access to the IX only to customers also using the IDC services of the integrated firm. Any customer seeking IDC services and access to the IX would thus have no choice but to use the services of the integrated firm. As other IDC providers would not be able to offer access to the IX, such integration may lead to reduced competition in IDC markets. The different market outcomes in the US and Europe illustrate this. In the US, integration between IDC providers and IXs is the norm, while in Europe, so far, IXs have typically been independent of IDC providers. Potentially as a result of this, prices for Cross-Connects, for example, are much higher in the US than in Europe as would be expected with reduced competition. Similarly the network size of IXs in terms of participants appears to be systematically smaller in the US than in Europe, as would be expected by integrated firms restricting access to the leading IX to their own IDC customers. While it is difficult to vertically integrate with the typically membership association owned IXs in Europe, it is conceivable that a leading IDC provider could establish its private IX using its established large IDC customer base. This has been done by a leading IDC provider, Equinix, with considerable success in Paris. In summary, the IDC industry, by offering high-performance interconnection between networks, provides a critical infrastructure service for the digital economy. Due to network effects, competition within an IDC market may be strongly reduced if there is a large asymmetry in size between the leading and other IDC providers as network effects tend to be NERA Economic Consulting iii

9 Executive Summary self-reinforcing. Asymmetries in size between IDC providers may also stifle competition across IDC markets as the leading IDC provider can offer more powerful incentives to magnet customers to use its IDCs in a new market. Finally, the complementary relationship between IDC services and IX services implies the risk that competition can be reduced by integration across both segments. NERA Economic Consulting iv

10 Introduction 1. Introduction NERA Economic Consulting ( NERA ) has been commissioned by Interxion Holding N.V. ( Interxion ) to prepare a report on the facts and economics of the Interconnection Data Centre ( IDC ) industry with a focus on Europe ( White Paper ). The purpose of this White Paper is to inform interested parties on salient features of the industry and to contribute to the understanding of a sector that is highly important to the functioning of the digital economy. The White Paper is based on publicly available information as well as on insights gained in interviews with various industry participants. 2. Background to the IDC Industry 2.1. The Internet as a Network of Networks A communications network may be defined as consisting of points, where information is provided or consumed, and connections between those points. Figure 1 depicts a simple star network. Each point A to F in this star network is connected with all other points. Figure 1 A Star Network A B G S C D F E Source: NERA. From an economic point of view the defining feature of a network is that each additional connection between points increases the economic value of the existing connections. 1 This is illustrated above for a simple star network: for each additional participant connecting to the core S, every existing participant gains an additional communication possibility See e.g. Economides (1996). If a network is comprised of subscribers, there are ( 1)/2 connections. This relationship is also referred to as Metcalfe s law in engineering, which states that the value of a communications network is proportional to the square of the number of connected users. NERA Economic Consulting 1

11 Background to the IDC Industry The Internet can be seen as an electronic communications network consisting itself of a large number of electronic communications networks. The Internet is thus often defined as a network of networks. 3 Figure 2 provides a stylised depiction of the Internet. Figure 2 The Internet as "Network of Networks" Source: NERA. In order for the individual networks in the Internet to exchange data with other networks, these networks must come together in a given location and physically connect to each other. This physical connection between networks for the purpose of data exchange is defined as interconnection. Such interconnection typically takes place in IDCs. IDCs are not limited to interconnecting communications networks though. IDCs interconnect any network or organisation requiring the exchange of data with other networks or organisations that are colocated in the same IDC or in the same IDC campus. In order to place the IDC industry in its proper context the following section provides an overview of the key networks and players in today s Internet Key Internet Networks and Players In characterizing and defining important network types or activities of today s Internet this White Paper largely draws on the report by the Body of European Regulators for Electronic Communications ( BEREC ) of December The definitions and classifications of 3 4 See Braden (1989), page 7. See BEREC (2012). NERA Economic Consulting 2

12 Background to the IDC Industry BEREC are then complemented with insights gained in discussions with industry participants and industry research. 5 On this basis, the following types of networks are defined: Content and Application Providers ( CAPs ): CAPs create and aggregate content (e.g. web pages, blogs, movies, photos, or trading information) and applications (e.g. search engines, social networks, messaging services, or trading platforms). 6 Examples of CAPs are publishers such as the New York Times, search engine providers such as Google, e- commerce platforms such as Amazon, social networks such as Facebook and YouTube, and video services such as Netflix. Content and Application Users ( CAUs ): CAUs are residential (private) or business users mainly passively consuming content through broadband Internet access. 7 Increasingly, CAUs also produce content such as in social networks and peer-to-peer applications. They typically do so in the context of consuming an application rather than for the purpose of revenue generation. 8 Internet Service Providers ( ISPs or carriers ): ISPs are defined as operators of telecommunications infrastructure that transport data from an origin to a destination, e.g. from the servers of a CAP to the location of a CAU. 9 ISPs are regularly also referred to as carriers. From the perspective of their customers they sell network access services. Different types of ISPs can be distinguished. 10 One widely used criterion for distinction, the classification into Tier 1, 2 and 3 ISPs, is based on how close an ISP s network access services are to the Internet Backbone, i.e. to what extent ISPs are upstream wholesalers rather than retailers of network access services. 11 Another criterion for distinction is the type of customers served. On the basis of these criteria, the following categories of ISPs can be distinguished: Tier 3 ISPs: These ISPs sell network access at the retail level (called Internet access service ). They themselves need to purchase Internet access from upstream ISPs. Two types of such retail level ISPs can broadly be distinguished: CAU Eyeball ISPs: These ISPs predominantly sell Internet access to CAUs. Typical Eyeball ISPs are the former telecoms incumbents such as BT, France Telecom or Deutsche Telekom and their competitors The aim of this classification is not to allocate each actual internet firm into one of the categories discussed here but to characterize certain types of activities in today s internet. A firm may be active in one or many of those activities. See section 2.1 of BEREC (2012) for this definition and a more detailed discussion. The number of end-users of the Internet was estimated to have reached two billion in See Weller and Woodcock (2013), page 61. See section 2.2. of BEREC (2012) for this definition and a more detailed discussion. In 2011 the Internet was estimated to consist of around 5,000 ISP networks. These ISP networks were in turn connecting 37,313 Autonomous Systems (AS), a number which has grown to 51,027 as of June 2015; See Smith, P. (2011); See Smith, P. (2015); See Weller and Woodcock (2013), page 8. An AS is a network under the control of a common administrative entity such as a government agency, an academic institution or a private company. An AS is assigned a globally unique number, called an Autonomous System Number (ASN). See Arthur D. Little (2014), page 67. See section 2.3 of BEREC (2012). The definitions of ISPs in this White Paper slightly differ from those proposed in BEREC (2012) as they incorporate the market understanding gained from other sources and industry participants. See ERG (2008), page 48 or 108. NERA Economic Consulting 3

13 Background to the IDC Industry CAP Eyeball ISPs: These ISPs predominantly sell Internet access to CAPs. Examples of such ISPs are 1&1 or Strato. Tier 2 ISPs: These ISPs offer network access to Tier 3 ISPs; they themselves have to purchase network access from more upstream ISPs in order to provide connectivity to the entire Internet. Tier 1 ISPs: These ISPs offer wholesale network access to Tier 2 and Tier 3 ISPs. Due to their size and reach they need not purchase network access from any other ISP but exchange data with other Tier 1 ISPs on a reciprocal basis. Smaller ISPs pay those Tier 1 ISPs for sending and receiving traffic. 12 Examples of Tier 1 ISPs are Level3, Global Crossing, Verizon and Cogent. 13 Content Delivery Networks ( CDNs ): CDNs deliver the content of CAPs closer to the terminating network, i.e. closer to the CAUs, by maintaining servers caching this content close to the Eyeball ISPs. They often also operate their own network and therefore do not need to purchase network access or connectivity from ISPs. 14 Examples of CDNs are Akamai, which operates servers only; 15 Limelight, which also operates its own network; or CAPs such as Google and Amazon that operate their own CDNs to deliver content. Private Clouds ( PC ): Private Clouds are physical computing resources operated by a single organisation such as a business or a government organisation outside of their own facilities for various reasons including lower costs and greater scalability. These off-site computing resources can be accessed by that single organisation only. Public Cloud Providers ( PCP ): PCPs are businesses offering cloud services to private or business CAUs and to organisations using Private Clouds. In contrast to PCs PCPs use a shared physical computing infrastructure to provide cloud services to many customers. Examples of services PCPs offer are cloud storage and online office applications. Similar to CDNs, PCPs seek to maintain servers close to their customers, which may be the PCs of some organisations or the Eyeball ISPs. By doing so they can guarantee high performance in exchanging data. Examples of PCPs are Microsoft Azure and Amazon Web Services. When including the different key networks and players defined above, a more realistic and complex illustration of the Internet is obtained as shown in Figure Reciprocal traffic exchange agreements that do not involve payments from one network to another are referred to as peering. Commercial agreements that involve payment for the transmission of traffic on a metered basis are referred to as transit. There are different variants of peering and transit such as paid peering and partial transit. See e.g. BEREC (2012) for a discussion. The IDC industry provides the physical interconnection between networks. Which commercial agreements are used to exchange data between interconnected networks is not the subject of this report. See GSM (2007) for a discussion of the economics of such commercial interconnection agreements. The top ten Tier 1 ISPs ranked by AS rank are: Level3, Cogent, TeliaSonera, NTT America, Tinet Spa, TATA Communications, Telecom Italia, Hurricane Electric, XO Communications, and Bexond the Network America. The full list can be retrieved at See section 2.4 of BEREC (2012) for a similar definition and a more detailed discussion of CDNs. In operating a network of their own they still typically use services of ISPs by typically renting dedicated connection capacity from them e.g. capacity on fibre optic cables. See Nygren, Sitaraman and Sun (2010). NERA Economic Consulting 4

14 Background to the IDC Industry Figure 3 The Internet and its Important Players Source: NERA. The blue squares in Figure 3 illustrate ISPs operating in one of the three tiers. As their service is to provide internet access, ISPs within each tier and between tiers need to establish interconnections in order to be able to exchange data. The blue circles represent CDNs. CDNs seek to interconnect as closely to the Eyeball ISPs as possible in order to provide content to CAUs at high QoS. Some CDNs such as Akamai only operate servers that connect to Tier 3 ISP networks and use ISP services to transmit data to their servers. Other CDNs such as Limelight operate a network on their own as illustrated by the blue circle at the Tier 2 level. Large CAPs such as Google operate their own CDNs. PCs and PCPs are illustrated by blue ovals. PCPs seek to interconnect with Tier 3 (Eyeball) ISPs in order to reach private or business CAUs. They also seek to interconnect with PCs directly in order to ensure security and high performance. Finally, a host of different firms and organisations also seek to interconnect directly with each other via their respective PCs. Section 3.5 below provides more detail on the motivations for such interconnection in different communities of interest. Interconnection of all those players typically takes place in IDCs using one of the interconnection technologies described in section 2.3 below Interconnection Technologies There are two basic technologies to physically interconnect networks: Cross-Connects, also referred to as Direct Circuits, and Network Switches. Cross-Connects are established with fibre optic or copper cables that directly connect the servers and other equipment of two networks. Technically, a Cross-Connect needs two network interfaces that act in the same technical manner with at least one physical port at each side of the connection. 16 A Cross- 16 In the following a port is always defined as a hardware device that serves as an interface between a computer and other computers or peripheral devices. A port is therefore the counterpart of a cable that connects devices to transfer signals. NERA Economic Consulting 5

15 Background to the IDC Industry Connect allows two networks to exchange traffic exclusively between themselves and therefore do not allow connections with other networks. 17 In contrast, a Network Switch is a device that allows interconnecting more than two networks at the same time. Each network requires a physical connection to the Network Switch and can then exchange data with all other networks connected to the switch. A Network Switch allows transmitting an IP packet using the IP address information contained in each IP packet. A main concern for all networks is QoS and cost of data transmission when interconnecting with other networks. QoS in data transmission can be assessed on the basis of four indicators: 18 Capacity (also referred to as bandwidth, speed, transmission rate, or throughput): Capacity denotes the amount of data in Mbps that can be transmitted in a network connection per second; e.g. a video of size 1GB takes 10 seconds to download with a capacity of 100Mbps but 1 minute and 40 seconds with a capacity of 10 Mbps. Latency (also referred to as delay): Latency is the delay from input into a system until a response is obtained. 19 In the example above, latency is the time required from the request to watch a video until the download begins. After download begins, the time until the download is complete depends on capacity. Both latency and capacity thus affect the experienced service quality, e.g. how long it takes until a website is loaded. Latency may be negatively affected by capacity, i.e. if all capacity is used, congestion occurs and latency is increased. However, (outside of congestion) latency cannot be decreased by adding more capacity. 20 Jitter: Jitter is the variation in latency. Jitter determines how reliably a network connection can be used for certain applications. Packet loss: In IP networks, data are divided into IP packets. Each IP packet is then routed separately to its destination. Usually due to network congestion, packets can get lost. This leads to the resending of the entire data. Packet loss thus naturally increases latency. Data transmission via a Cross-Connect over a short distance performs better in each of these QoS indicators. Capacity is high and can be scaled continuously between two networks based on requirements; latency is low given that the physical lengths of the Cross-Connect is typically a few metres; likewise, jitter is low due to the dedicated transfer capacity set up with a Cross-Connect. Finally, packet loss is unlikely to occur in a Cross-Connect. An important technical characteristic of Cross-Connects is that latency increases with the length of the Recently virtualized Cross-Connects have been developed and are offered by IDC providers. Such virtualized Cross- Connects allow a dedicated private data exchange between two networks without the need to physically install additional cables between these networks. As this technology currently is employed to interconnect private clouds with PCPs they are variously labelled Cloud Connect or Cloud Exchange etc. See e.g. or See BEREC (2011), pages 25 et seq. Latency can be measured with a utility called ping which measures the round trip time (RTT) that an IP packet needs to travel from the source to the destination host and back to the source. See also Arthur D. Little (2014), page 68. See for an enlightening presentation. NERA Economic Consulting 6

16 Background to the IDC Industry physical cables. After a certain cable length additional equipment is required to ensure data transmission. 21 As Network Switches are not dedicated to a particular connection between two parties, there is less control over the capacity. For example, in times of high traffic by other networks capacity bottlenecks may occur. Both latency and jitter can be expected to be higher in an interconnection via a Network Switch than with a Cross-Connect due to the typically longer total physical lengths of cables involved and the fact that an additional transaction device is involved in the data exchange. When capacity is full, i.e. there is congestion, latency also increases. Likewise, packet loss can occur e.g. during a period of congestion. Thus Cross- Connects can generally be expected to perform better in terms of QoS in data transmission than an interconnection via a Network Switch. Despite these drawbacks, interconnection via a Network Switch may have important cost advantages if low traffic interconnection with many networks is sought and latency is not of paramount importance. Interconnection via a Network Switch only requires a single port enabling interconnection with all other participants connected to the Network Switch. In contrast, a Cross-Connect requires at least one port and the respective receiving or sending equipment on each side of the interconnection with each network. Thus, interconnection via a Network Switch entails a cost advantage. It allows the connection to a large number of networks with a single physical connection. In summary, interconnection via a Cross-Connect allows for higher and more reliable QoS in data transmission and greater flexibility in scaling capacity. 22 Interconnection via a Network Switch may offer cost advantages if interconnection with many networks is required and QoS is not the highest priority According to industry participants Cross-Connects are viable only up to at most a few hundred metres. With the typically used Ethernet cables only transmissions of several hundred meters are technically viable according to the standardised IEEE criteria for maximum cable length. See for example or Retrieved on 5 August It should be noted that there is an additional way for content providers to achieve high quality of experience for their content by CAUs. By placing servers directly into the networks of Eyeball ISPs (i.e. network data centres, see section 3.2) content providers such as Netflix achieve higher QoS in the transmission of their content to the CAUs of that network than they would be able to achieve by interconnecting to that network via a Cross-Connect. While placing servers within a network does not constitute an interconnection of networks (e.g. it does not allow high QoS of data transmission in both directions), it may constitute an alternative to Cross-Connects and IX services for some content providers. See Norton (2014), pages 92 et seq. NERA Economic Consulting 7

17 The Interconnection Data Centre Industry 3. The Interconnection Data Centre Industry In principle, two servers located anywhere in the world could be interconnected with a fibre optic or copper cable for a Cross-Connect. With increasing length, however, a Cross-Connect not only becomes more and more expensive but more importantly it loses QoS in data transmission, e.g. latency increases. For modern types of content such as video streaming and for certain business models such as high-frequency-trading only Cross-Connects over short distances are of value. This strongly motivates customers to place the servers of interconnecting networks in close proximity to each other. This is exactly the role of IDCs. This section describes the different services provided by IDC operators, distinguishing between basic colocation services and interconnection services Colocation Services IDCs provide space, power, cooling, maintenance and service, physical security and external connectivity for the server and networking equipment of hosted firms: 24 Space/Real Estate: colocation facilities enable customers to rent the space in which to operate their own server and networking equipment. 25 Power: colocation facilities typically have dual power supply from utilities and various back-up systems such as batteries and generators. As the power requirements constantly increase, IDC operators need to ensure that they employ the most up-to-date technological solutions for power supply. Cooling: as server equipment generates heat, colocation facilities offer cooling and humidity control. As the power requirements constantly increase, so does the heat in the IDC. IDC operators need to ensure that they employ the most up-to-date technological solutions for cooling. Security: colocation facilities also offer multiple layers of physical security such as trained 24/7 security staff, alarm systems, biometric access control and visitor lists etc. Maintenance and Remote Hands : technical support staff and technicians follow instructions of a remote administrator if no staff of the customer is at the colocation facility in person. Services include e.g. reboot of equipment and diagnosing or correcting technical issues. External connections: in order for customers to send data to and receive data from their servers at an IDC, they require carrier services, i.e. physical connections to and from the IDC over which their data can be transported. Therefore typically several ISPs (carriers) See for example or Retrieved 9 July Space is rented out in different formats. Cabinets are racks in which companies can install their equipment. These cabinets are located in a shared area also used by other companies but each cabinet has a dedicated power feed. Cages provide customers with their own secure space. Cabinets can be placed into cages for added security. Cooling and power distribution is shared with other customers. Suites are similar to cages but are enclosed by walls and power is supplied to suites from dedicated PDUs and the access control system is exclusive for the suite. Private rooms are separated from other customer areas by enclosed walls. Power supplies from PDUs, cooling and access control are exclusive to the private room. NERA Economic Consulting 8

18 The Interconnection Data Centre Industry operate fibre-optic cable connections to an IDC. Some large customers may also operate their own fibre cables to and from their servers in IDCs. Figure 4 illustrates these basic colocation services offered by IDCs. Figure 4 Colocation Services of IDCs Source: NERA. Each customer rents space in the form of racks and cabinets where it can place its servers and networking equipment and benefits from the cooling, power supply, and security services of the IDC. Customers can reach their servers at the IDC using the external connectivity provided by ISPs serving the IDC. Figure 5 depicts the inside of a colocation facility for illustration. NERA Economic Consulting 9

19 The Interconnection Data Centre Industry Figure 5 Server Racks inside a Colocation Facility Source: Interxion Advertisement. Colocation facilities require high upfront fixed capital investments. The IDC operator needs to have a suitable building available and also purchase the necessary equipment including the racks and cabinets, cooling and humidity control, and security equipment as well as redundant power supply, such as generators and batteries (to run the location without external electricity supply in case of a blackout). 26 For example, setting up a colocation facility of around 2000 square metres requires an investment of around EUR 30 million; equipping the facility with a dense power supply network and cooling and other quality enhancing equipment will require EUR 100 to 170 per square meter. 27 Compared to those high upfront fixed capital costs, the variable costs of running a colocation facility are relatively low. The highest variable cost is that for power that is needed to run the customers servers and their cooling. Apart from maintenance of the facility, technical support, and security the operation of a colocation facility does not require variable labour costs IDCs and other types of Data Centres From the point of view of large CAPs, CDNs or large PCPs, the following types of data centres can be distinguished: In this context a suitable building means a site that is able to carry the servers that are installed at the facility. Moreover, in the whole building the installation of resilient air conditioning systems should be possible. See Focus Investment Banking (2012), page 2. NERA Economic Consulting 10

20 The Interconnection Data Centre Industry Core data centres: These data centres are very large facilities (with about 100MW of power usage) typically located in remote places in cool climates such as Finland and Oregon, US. These core data centres are used to store non-time-sensitive information. 28 Edge data centres: These data centres are located in proximity of the networks serving CAUs, i.e. typically in major metropolitan areas. Storing data in proximity to the Eyeball ISPs allows fast download and upload from these data centres. Time sensitive data are stored there. Network data centres: These data centres are located within the Eyeball ISP networks. They are used for even faster data transmission to CAUs. In contrast to Edge data centres, Network data centres obviously cannot be used to distribute content to several ISPs as they are located within a given network. Network data centres therefore cannot provide interconnection services. Large CAPs such as Google, Facebook, and Amazon operate or use all types of data centres. In particular, they operate core data centres on their own. However, they must still purchase colocation services in Edge data centres to distribute their content. 29 IDCs operate at the level of Edge data centres, i.e. IDC colocation services are not used by customers to archive data but by customers wishing to store data that is to be exchanged with business partners or provided to CAUs on a timely basis. In addition to colocation services an indispensable requirement for data centres to generate value for their customers at the Edge level is that they can provide interconnectivity with high QoS. IDCs are also distinct from wholesale colocation data centres. 30 Wholesale colocation data centres offer large areas of private space to large companies running their own data centres or to colocation data centre firms. The customers of wholesale colocation data centres then retail that space to other customers Interconnection Services The key defining feature of IDCs as compared to other data centre types is that customers colocate in those datacenters to interconnect with other networks (customers). As described in section 2.3, one interconnection technology is Cross-Connects, i.e. a direct physical copper, fibre optic or other cable between the servers of two customers. Customers have to purchase Cross-Connects from the IDC operator as such Cross-Connect cables See Retrieved 7 July See for a discussion in favour of proximity. Retrieved 10 July See for a definition of edge data centres. Retrieved 10 July See e.g. Dupont Fabros Technologies, See Focus Investment Banking (2012), page 2. NERA Economic Consulting 11

21 The Interconnection Data Centre Industry necessarily have to leave the rented space of the customer. IDC operators also need to logistically manage the large number of Cross-Connects in their facilities. 32 The leading IDC providers increasingly offer virtualized Cross-Connect services, in particular to cloud providers and their customers. 33 Such virtualized interconnection services still require a physical presence with servers and networking equipment at the IDC. However, they allow establishing an interconnection with the same properties as a Cross-Connect without having to install an additional physical cable. They thus save costs for IDCs and customers and allow a quicker installation as well as termination of interconnection relationships. 34 Figure 6 illustrates such Cross-Connects in a colocation facility of an IDC provider. Figure 6 Cross-Connects at IDCs Source: NERA. In contrast to colocation services only, the defining feature of IDC services is that customers interconnect directly with each other within the IDC as illustrated by the red lines in the figure above Modern IDCs have meet-me rooms, i.e. rooms within the colocation facility that are dedicated to interconnection. Such meet-me rooms facilitate the logistics of Cross-Connects, as all physical cables go from the customers servers to the meet-me room rather than across the facility in arbitrary ways. See footnote 17 above. Information according to industry participants. Technologically virtualized Cross-Connects are similar to IX services, i.e. interconnections can be established between networks without additional physical equipment. It appears that so far, virtualized Cross-Connects are mainly used for interconnecting PCPs and their customers though and there currently appears to be not substitutability between IX services and virtualized Cross-Connects. NERA Economic Consulting 12

22 The Interconnection Data Centre Industry 3.4. Internet Exchanges and IDCs Interconnection via Network Switches is offered by Internet Exchanges ( IX ). 35 IXs are defined as physical network infrastructure[s] operated by a single entity with the purpose to facilitate the exchange of Internet traffic between Autonomous Systems. The number of Autonomous Systems connected should at least be three and there must be a clear and open policy for others to join. 36 In the US IX services are typically offered by the same firms that offer IDC services, i.e. there is integration of firms across the IX and the IDC markets. As part of the portfolio of profit seeking companies, IXs are operated for profit in the US. In contrast, in Europe IXs are typically not-for-profit organisations owned by associations whose members are the customers of the IX. Either way, IXs typically place their equipment in IDCs rather than running their own colocation facilities. Figure 7 illustrates an IX housed at an IDC. Figure 7 Internet Exchanges are housed at IDCs Source: NERA. Just as any other IDC customer, the IX rents space and consumes power, cooling, and security services from IDC providers. In order to exchange data at the IX, IX participants need a physical connection to the IX. Participants that also colocate in the same IDC may then simply purchase a Cross-Connect to the IX. Other IX participants may connect with physical cables from outside of the IDC (if the IDC operator permits) or may use the services IXs used to be called Network Access Points (NAPs). See BEREC (2012), page 26. See Euro-IX (2013), page 4. NERA Economic Consulting 13

23 The Interconnection Data Centre Industry of ISPs that are connected to the IX. IXs are often considered magnet customers by IDC providers, as they attract many other customers seeking to connect to them Communities of Interest for IDC Services An IDC is only of value to a given customer if there are networks in the same IDC with which the customer seeks to interconnect. For customers that can tolerate higher latencies also the networks in geographically close IDCs of the same IDC provider may be of interest and thus increase the value of the IDC provider to that customer. Which networks each customer is interested in is dictated by the business activities of that customer. The group of businesses seeking network interconnection with each other in order to create economic value are referred to as communities of interest. These communities of interest drive the demand for IDC services. An IDC hosting a community of interest is very attractive to members of that community because of the concentration of relevant interconnection opportunities that it provides. The following sections provide examples of different communities of interest Electronic Communications Networks The original need for interconnection services arose from local and regional electronic communications networks needing to interconnect with each other to form a global communications network, the global Internet. For example, in order for a CAU in the US to communicate electronically with a CAU in Italy, the US CAU requires Internet access services from an Eyeball ISP. This Eyeball ISP will then have to interconnect with a Tier 2 or Tier 1 ISP to send traffic from the US to Europe over a submarine cable. Tier 1 ISPs typically need to interconnect with other Tier 1 ISPs to transmit data over longer distances. In the destination region, Tier 1 ISPs need to interconnect with local Eyeball ISPs that transmit the data to the CAU in Italy. Each of these interconnections typically takes place in an IDC via Cross-Connects or an IX. For high traffic volumes ISPs will use Cross-Connects between them. For lower traffic volumes traffic exchange will take place via IXs. As traffic volumes are expected to increase exponentially, high QoS interconnection becomes ever more important for electronic communications networks to deliver their services Digital Content Delivery CAPs require access to CAUs in order to provide their content to users. In principle, a CAP could deliver its content to CAUs anywhere in the world by using the services of interconnected ISPs as described in the previous section. However, the increasingly high data volume of content such as photos, music and video files or music and video streaming as well as increasing demands regarding response and download times by CAUs around the world make it increasingly difficult or even impossible to provide content in this simple way of delivery. For example, websites would take too long to load, downloads of audio and video 37 Information according to industry participants and information on IDC provider websites. See e.g. homepage of Interxion: retrieved 24 July 2015, or Equinix: retrieved 24 July NERA Economic Consulting 14

24 The Interconnection Data Centre Industry files would take too long, and streaming would be impossible due to interruptions reducing the quality of experience. 38 For content to be delivered with high QoS to CAUs it must be placed close to the terminating networks serving those CAUs. By using the services of CDNs or by building their own content delivery networks CAPs seek to store content in IDCs around the world where they can interconnect with Eyeball ISPs serving the CAUs of a given region. Proximity to the terminating Eyeball ISP networks allows low latency access and high bandwidths content delivery and thus a high quality user experience. 39 CAPs thus seek interconnection with Eyeball ISPs in close proximity. Likewise, Eyeball ISPs seek close interconnection with CAPs or CDNs in order to offer high quality content to their CAUs. CAPs, CDNs, and Eyeball ISPs all benefit from interconnection in close proximity for high QoS content delivery. Case Study: Spotify Spotify allows subscribers to stream music legally. It is headquartered in Stockholm and London. In order to give their subscribers fast access to the streamed music, Spotify maintains servers at several IDCs around the world. For example, Spotify hosts content on servers at IDCs of TelecityGroup in Stockholm and uses IDCs of Equinix, via hosting provider Carpathia, in the US. 40 At these IDCs Spotify has access to many networks also colocated in these IDCs as well as to leading IXs in each region, i.e. Netnod IX in Stockholm, Sweden and Equinix IX in Ashburn, Virginia Cloud Services Cloud computing can be defined as the use of computing resources over a network connection, so that the hard- and software required for the computing services is not located at the premises of the user but at a different location. Cloud services are inter alia Infrastructure as a Service (IaaS) or Software as a Service (SaaS). IaaS allows businesses to rent hardware rather than having to buy it resulting in lower investment and cost of managing the hardware and quicker and more flexible scalability of hardware use. 43 SaaS likewise allows customers to rent software. SaaS allows the use of software applications over See Marinescu (2013), page 227. By placing their servers within in the terminating networks of Eyeball ISPs CAPs may even achieve a higher quality of experience for CAUs. Such network data centres may thus in some sense be an alternative to placing servers close to the terminating networks in Edge data centres. See Retrieved 30 July 2015; see also Retrieved 30 July See Retrieved 30 July See Retrieved 30 July See e.g. NERA Economic Consulting 15

25 The Interconnection Data Centre Industry the internet. Online , storage, social network or photo programs are all examples of SaaS. Again SaaS allows lower costs and greater flexibility in software usage. 44 Some PCPs such as Google or DropBox offer cloud services, e.g. data storage or office applications, to private CAUs. These PCP seek interconnection with Eyeball ISPs in order to serve their CAU customers. These PCPs have similar interconnection needs as CAPs delivering content to CAUs as the CAUs want to up- or download possibly high volume data quickly and reliably from their cloud drives. PCPs offer cloud services such as web hosting or online office functionalities also to business customers who have particularly high expectations regarding reliability and performance of these services. High QoS interconnection with the Eyeball ISP serving these business customers is therefore a crucial success factor. Business customers often operate Private Clouds either as IaaS, as described above, or by placing their own equipment in an IDC. The primary reason for businesses to operate Private Clouds in IDCs is to interconnect with PCPs to consume their services such as online office functionalities. 45 The close interaction of Private Clouds and PCPs is described as Hybrid Cloud. 46 In order for such Hybrid Clouds to work Private Clouds require high QoS interconnections with PCPs in an IDC. Case Study: Foursquare 47 Foursquare is an online search and discovery application that offers personalized recommendations and deals based on where the application users or their friends are. Foursquare uses a combination of a Private Cloud and the services of a PCP (Amazon Web Services). It uses its Private Cloud to analyse and manage its customer data. The PCP operates the interface accessed by the CAUs of the Foursquare mobile app. This division of labour requires high performance data exchange between Foursquare s Private Cloud and the PCP which is achieved by Cross-Connects in an IDC. Using this Hybrid Cloud only the content that has to be replicated across several IDCs within the PCP server system are hosted at the servers of the PCP. By contrast the main database used for customer data analysis is hosted within the Private Cloud. This main database, however, needs to be synchronized daily with the data stored at the PCP involving high amounts of data. This is enabled by high-performance Cross-Connects between the Private Cloud and the PCP thus reducing synchronization time from days to hours See e.g. Private clouds may also be hosted by PCPs as virtual private clouds, see e.g. AWS: See e.g. See Equinix: Retrieved 6 August NERA Economic Consulting 16

26 The Interconnection Data Centre Industry By using such a Hybrid Cloud Foursquare succeeded in reducing average transmission latency from 11 milliseconds, with traffic transmitted over the public internet, to 2.5 milliseconds, with interconnection of its Private Cloud with the PCP in an IDC Financial Services In the past 30 years, the time scale over which a trade is processed has gone from minutes to milliseconds and below. According to estimates, a one millisecond advantage in a trading application can be worth 100 million USD a year to a large brokerage firm. 48 To make highspeed trading possible, trading firms and brokers require ultra-low-latency interconnections to each other and the liquidity venues such as stock exchanges, and ultra-low-latency interconnections with information providers. These ultra-low-latency interconnections are implemented as Cross-Connects of short lengths in IDCs. Case Study: ACTIV Financial 49 ACTIV Financial is a global market data vendor supplying hedge funds, high-frequency trading firms and retail brokerages with real-time market information. In order to provide high performance and lowest latency data feeds to its clients in London ACTIV Financial needed an IDC provider with a thriving financial community. They found this community in Interxion s IDCs in London. There ACTIV Financial can connect to its clients with a Cross-Connect to provide them with highest performance and lowest latency data feeds. The Global Head of Sales of ACTIV Financial says: These high-capacity connections are simple and convenient to implement, so our clients can quickly start using our market data. Equally, ACTIV can easily connect to a wide range of European exchange feeds, such as Euronext, the London Stock Exchange, and Nasdaq OMX that have PoPs at the data centre or are so close by that we can set up high-speed fibre links Advertising According to an emarketer survey, in the U.S. alone, desktop display, mobile and video ad sales are projected to rise from $4.3 billion in 2014 to $18.2 billion by Online advertising works on the basis of real-time auctions where advertisement slots on Google, Amazon and many other platforms are auctioned off to the highest bidders. These auction mechanisms are mostly automated and auctions restart within pre-specified time intervals. In this environment bidding needs to occur within milliseconds and reducing latency creates important business advantages such as more time to analyse an advertising opportunity See Tobin (2015), page 16. See Retrieved 26 August See retrieved on 26 August NERA Economic Consulting 17

27 The Interconnection Data Centre Industry In order to reduce latency advertisers can cross-connect to leading online ad partners such as Google within an IDC. This allows bypassing the heavy traffic and risks posed by the public Internet and conducting transactions with latency in the single-digit milliseconds. 51 Case Study: Bright Roll 52 Bright Roll is an advertising platform for Web, mobile, and TV videos enabling advertisers to place ads e.g. at the beginning of a YouTube video. 53 Bidding for the advertising slot, e.g. in a YouTube video, happens in real time, i.e. a bid for the advertising slot must be submitted in only 100 milliseconds. The faster a bid is received by the ad seller the more likely it is that there will be a match between the ad seller and an ad buyer. Sending information over the public internet involves high latency and, more importantly, unpredictable variation in latency (jitter). Such high and unexpected delays in transmitting bids can prevent potential ad buyers from placing bids in time. This implies lost advertising opportunities for ad buyers and lost revenue opportunities for ad sellers. Bright Roll sought to reduce latency between bidders and its advertising exchange, the BrightRoll Exchange or BRX. To that end it placed BRX in an Equinix IDC where it could cross-connect to high-speed telecommunications networks and advertising industry partners. Compared to operating its advertising exchange over the public Internet, BrightRoll managed to reduce average network latency by 80%. In turn it succeeded in increasing successful ad response rates, i.e. completed matches between ad buyers and ad sellers, by around 50% E-Commerce Global business-to-consumer E-commerce sales are expected to climb to $2.4 trillion in 2017, according to emarketer. In order to compete in this market, E-commerce businesses have to meet user expectations of near-instantaneous response times and continuous site availability. Therefore, in particular large E-commerce enterprises seek to cross-connect to Eyeball ISPs in IDCs to ensure low latency in the interaction with CAUs and to consolidate ordering data from several online sessions of customers in a central data base system. According to some estimates, large E-commerce firms such as Amazon could increase their revenue by 10% with a one second reduction in response times Summary and Conclusions Cross-Connects over short distances allow QoS levels, in particular low latency, that cannot be achieved over longer distance transmission technologies or via an IX. In order to be able to establish Cross-Connects to important business partners, businesses must place their See Equinix: Retrieved 6 August This case study is based on a presentation by Equinix, see Retrieved on 26 August See See Arthur D. Little (2014), page 15. NERA Economic Consulting 18

28 The Interconnection Data Centre Industry servers in close proximity to the servers of their business partners. By offering colocation services and Cross-Connects IDCs enable high QoS interconnection. The following quote from a representative of a leading IDC operator provides an instructive summary of these phenomena: [ ] Data centers are playing a critical role [ ] because they bring the IT infrastructures of many different companies together. When companies are in the same facility, they can share data ultra-reliably within milliseconds, because it s traveling across meters, not miles. The data center also provides a neutral convergence point for disparate types of network services and improves user experiences by bringing systems, applications and content closer to customers. To seize the business advantages of these emerging services and business models companies also need interconnection, a form of connectivity that s instant, direct, massively scalable and increasingly more flexible than its predecessors. In contrast to traditional types of connectivity, interconnection is more secure, reliable and agile. [ ] 55 IDCs thus provide the infrastructure for high performance data exchange. Such high performance data exchange is arguably at the core of many new digital business models that are creating new and transforming old industries. Some industry participants speak of the next stage in the digital transformation of the entire economy through interconnection. 56 Figure 8 illustrates this perspective along with the historical evolution of IDCs out of the general data centre industry that was due to this development See Bishop (2015), Interconnection: History in the Making, Available at Retrieved 26 July See Bishop (2015), Interconnection: History in the Making, Available at Retrieved 26 July NERA Economic Consulting 19

29 The Interconnection Data Centre Industry Figure 8 IDCs and the History (and Present) of the Digital Economy Source: NERA, Bishop (2015). The IDC industry has been born out of the wider data centre industry in response to the transformation of the digital economy toward a more and more interconnection based economy. The changing nature of the data centres in question from general data centres to IDCs is also reflected in the predominant pricing models used over time. Starting with space and power usage based pricing models, which are suitable measures for the use of colocation services, IDCs started to charge prices for additional Cross-Connects reflecting the increasing demand for such interconnections. Industry participants expect that in the future pricing will be more and more based on the value customers derive from the interconnection opportunities at IDCs. IDCs thus provide a crucial infrastructure service for the future of the digital economy. It is therefore warranted to analyse some aspects of the economics of the IDC industry. NERA Economic Consulting 20

30 Aspects of the Economics of the IDC industry 4. Aspects of the Economics of the IDC industry This section builds on the presentation of the various elements that constitute the IDC industry laid out in the previous sections and puts these into an economic perspective. Figure 9 shows a simplified vertical industry structure from the perspective of CAUs consuming the content produced by a CAP. Figure 9 Vertical Industry Structure of Provision of Content to CAUs Source: NERA. As illustrated in the figure, the transmission of content from the site of production to the CAU involves the following layers that together comprise a chain of complementary goods and services: Backbone (Tier 1) ISPs: After having produced or acquired the content, e.g. a TV series, the CAP needs to use the services of an ISP to deliver the content to an IDC in the destination region Very large CAPs may operate their own fibre network or lease dedicated fibre from ISPs to transport the content to the destination region. Other large CAPs may have relationships directly with Tier 2 or Tier 1 carriers. Smaller CAPs, NERA Economic Consulting 21

31 Aspects of the Economics of the IDC industry Interconnection: In the destination region the data reach the servers of one or several Backbone ISPs in the IDCs chosen by the CAP. In the IDC the data could be transmitted to the servers of the CAP using either a Cross-Connect between the servers of the ISPs and the servers of the CAP, or using the Cross-Connect from the ISP to the IX and from the IX to the servers of the CAP. The CAP in turn may transmit the content to the Eyeball ISPs in the destination region using again Cross-Connects to their servers or, typically with smaller ISPs, via the local IX. Eyeball (Tier 3) ISPs: The Eyeball ISPs deliver the content to the CAUs in the destination region. The cost associated with the transmission of content from the region of origin of the content to the CAUs in the region of destination thus depends on the degree of competition on several separate vertical/complementary levels the backbone ISP level, the Eyeball ISP level, the IDC level, and (possibly) the IX level and several separate geographic markets. While there are certainly reasons for why competition at the Eyeball and Backbone ISP level should be scrutinized as well, competition problems at these levels is not the subject of this White Paper. 58 The White Paper instead focuses on the remaining vertical/complementary level of Interconnection, i.e. in Figure 9 the level where the physical interconnection between Eyeball and Backbone ISPs and between ISPs and CAPs takes place. Furthermore, the focus of this White Paper lies on the provision of the physical interconnection infrastructure rather than on the economic effects of different interconnection pricing models. 59 Competition at the Interconnection segment has a horizontal dimension only involving IDC services, which will be discussed in section 4.1. There is also a vertical dimension involving both, IDC and IX services. This vertical aspect is discussed in section Horizontal Aspects of IDC Competition In the IDC market, suppliers of colocation and interconnection services compete for ISPs and CAPs and other business networks seeking to interconnect with each other (see the discussion of communities of interest in section 3.5). As the primary service of interest is interconnection as opposed to colocation (see section 3.3), IDC services exhibit network effects, meaning that IDC customers value the number of relevant interconnection however, likely have to use the services of Eyeball CAP ISPs first. These CAP Eyeball ISPs may have to interconnect with more upstream Backbone ISPs for any transmission over a larger distance. In turn Backbone ISPs may have to interconnect with other Backbone ISPs once or several times in order to deliver the content to the destination region. Such interconnection typically takes place in IDCs and/or IXs. Competition between Eyeball ISPs in the region of destination determines whether the CAP can obtain distribution services to CAUs on competitive terms; likewise competition between Eyeball ISPs in the region of origin determines the ability to gain Internet access (or access to an IDC where the Backbone ISPs can be found). These Eyeball ISPs need to operate expensive last-mile infrastructure such as fibre optic and copper cable networks reaching every single household which may restrict entry or even exhibit the characteristics of a natural monopoly. Likewise the competitive transmission over longer distances depends on competition between Tier 1 (backbone) ISPs in different regions. This vertical level also involves high infrastructure investments, resulting in high barriers to entry. See e.g. Economides (2005), or Laffont and Tirole (2001). For a discussion of the economic implications of different interconnection pricing models, see GSM (2007). NERA Economic Consulting 22

32 Aspects of the Economics of the IDC industry opportunities found in an IDC. In order to describe competition in the IDC market, the origin and nature of network effects in IDC services are discussed in section The importance of magnet customers is discussed in section The subsequent section discusses the implications of the existence of such network effects and magnet customers on market dynamics Network Effects in IDC Services IDCs enable customers to physically interconnect with each other. Depending on the community of interest a customer belongs to, the presence of certain other customer types can be irrelevant as interconnecting with them does not generate value for either customer, or highly relevant when interconnection generates significant value. In the remainder of this section, network effects are discussed from the perspective of a given customer and the relevant and attractive other customers present in the IDC that belong to the same community of interest (in the following referred to as peers ). 60 The peers found by a customer when using the services of an IDC provider are referred to as the IDC network. In order to interconnect in an IDC, a customer needs to purchase colocation services from the IDC and install server and networking equipment there. It can then interconnect with other customers present in the IDC using Cross-Connects to the servers of those customers directly. Alternatively it can access any IX present in the IDC with a Cross-Connect and reach other customers that are also connected to the IX. The mere presence in an IDC involves costs independent of the number of interconnections a given customer may choose to have ( costs of presence ). These costs of presence are made up of, for example, the cost of own server equipment and software, the labour cost of technicians and software engineers to install the equipment and software, and the cost of colocation at the IDC. The existence of such costs of presence in an IDC and the need to interconnect with usually several valuable peers gives rise to network effects: from the perspective of a prospective customer the value of being present in an IDC increases with the number of relevant potential peers that are present in this IDC. 61 These network effects arise, inter alia, due to the savings in costs of presence from avoiding having to be present in additional IDCs. 62 For example, if the costs of presence in an IDC were X Euros per year for a CAP, the savings from using only one IDC with all desired peers instead of two IDCs each with fewer potential peers would be X Euros per year, the savings of using one IDC instead of three IDCs would be 2X The term peer here is to be understood as a firm or organisation with which a given customer may want to interconnect in order to generate economic value. It is related to but not to be confused with the term peering which in the context of interconnection denotes a commercial arrangement whereby two networks exchange traffic free of charge. In economics a good is defined to exhibit network effects if the value of the good increases with the number of users/consumers of this good. Typically direct network effects caused by physical links and indirect network effects caused by compatibility questions are distinguished. See Economides (1996) for an introduction and overview of the economics of network effects. There may be other sources of network effects, such as, for example, a higher number of hardware or software service providers when there is a large number of customers in an IDC. NERA Economic Consulting 23

33 Aspects of the Economics of the IDC industry Euros and so on. Thus, when assuming the same costs of presence at every potential IDC, the value of the IDC increases monotonically with the number of relevant peers. 63 Not all peers are equally important. For CAPs the presence of large Eyeball ISPs, Tier 1 ISPs and IXs will be particularly important whereas the presence of smaller Eyeball ISPs may play a lesser role. For example, a large Eyeball ISP with ten times as many CAU customers as a small Eyeball ISP may allow the CAP to sell ten times more content at high QoS and thus to generate more revenue. Therefore, a given CAP may only consider being present at an additional IDC if this allows interconnecting with another large Eyeball ISP but not if it allows interconnecting only with one or several smaller Eyeball ISPs. For the latter interconnections it could use an IX for example. Therefore the value of an IDC network for a particular customer does not increase in the same way for each peer present in the IDC. As explained above the value of IDC services is determined by the savings in the costs of presence from not having to use additional IDCs to satisfy interconnection needs. When only few less important peers exist in a data centre, there is no reason for the customer to use an additional IDC and thereby incur the additional costs of presence in the additional IDC. The given customer may only decide to use an additional IDC if a large number of such less important peers are present in the additional IDC. Therefore the value of an IDC increases less with the number of less important peers present at an IDC. Assuming that there are only two types of relevant peers, important and less important ones, and ranking the peers in order of importance the increase in value of an IDC as a function of the number of peers may thus follow a monotonically increasing function with a kink. Figure 10 illustrates such a function. 63 Many customers of an IDC may not need to interconnect with many peers in an IDC but only two or three peers; they do value the presence of many peers, however, with whom they could potentially interconnect in an IDC. This may be the case for content providers, which benefit from a large number of carriers to choose from for their transit needs. In this case nevertheless the value of an IDC would increase with the number of peers present at an IDC, as, again, a customer can save the cost of presence in an additional IDC if it can obtain the desired choice in one IDC. NERA Economic Consulting 24

34 Aspects of the Economics of the IDC industry Figure 10 Illustration of Value of an IDC Network with important and less important peers Source: NERA. For each peer that would induce a CAP to use an additional IDC for interconnection if the peer were not present at the IDC in question, the value of the IDC in question increases according to the amount of costs of presence saved. For each less important peer the value of the IDC in question increases less steeply when assuming that the CAP would use an additional IDC only if a sufficient number of such less important peers were present in the additional IDC. Figure 10 illustrates the role of important and less important customers for network effects by way of a simplified example. In reality, each customer will have its own individual IDC value function, resulting from a different mix of more important and less important relevant peers and different costs of presence in an IDC, and potentially other factors Magnet Customers In the extreme there could be only one important relevant peer for a given customer and all other relevant peers would not induce the given customer to be present in an IDC. Thus the entire value of the IDC is determined by the presence of a single relevant peer. This case exemplifies the concept of a magnet customer, which plays an important role in many communities of interest. 64 Such magnet customers are considered key customers to create or maintain an IDC network as their presence induces many other enterprises to become or remain customers of an IDC. 64 Information according to industry participants. NERA Economic Consulting 25

35 Aspects of the Economics of the IDC industry In this extreme case network effects may not be a suitable perspective as the network of peers comprises only one participant, the magnet customer. In reality, however, there may be more than one magnet peer, so that they are likely better described as important customers, and a continuum of more important and less important peers from the perspective of a given customer. The magnet customer may thus be of critical importance for an IDC network but other peers may still increase the value of the IDC network. Nevertheless, due to their importance, such magnet customers have a role of their own in the competitive dynamics of the IDC industry. This role is explored further in the sections that follow Market Dynamics with Network Effects and Magnet Customers The existence of network effects has implications for the competitive interaction in the IDC market. IDC providers such as Equinix, Telecity, Interxion, or Telehouse may operate one or several IDCs in each relevant market. They compete for customers seeking to interconnect with peers by offering their IDC infrastructure and services. At a given price (for colocation and interconnection services) the customer will choose the most attractive IDC network. One important factor for the attractiveness of an IDC network will be the presence of magnet customers. The other important factor is the size of the IDC network in terms of the number of attractive peers. For a customer seeking particularly low-latency interconnections, the number of peers present in the given IDC will constitute the relevant IDC network. As IDC providers typically also connect their IDCs in a metropolitan area with high performance fibre optic cables and thus enable connections across their IDCs, for customers that can tolerate also higher latency interconnections the relevant peers in all IDCs of an IDC provider will constitute the IDC network. The following discussion abstracts from this difference between IDC network at the IDC level and IDC network across IDCs in a given metropolitan area of a given IDC provider by simply referring to the IDC network of an IDC provider Network size as an entry barrier The existence of network effects implies that an IDC provider will only be able to compete if it can convince prospective customers that it can offer an attractive IDC network. For IDC providers with an existing large IDC network this is relatively simple, as the actual IDC network is the IDC network customers can expect when becoming a customer of the IDC provider. Entrants into markets with established IDC providers, however, will have great difficulty in convincing prospective customers that they will be able to offer an attractive IDC network. Joining an established IDC network environment is clearly most attractive in terms of connectivity for new customers. The existence of entry barriers can be expected to shape competition in the IDC market. 65 First, there will be fewer competitors; and second, the existing competitors are under less pressure to provide competitive services if they are under less pressure from potential entrants An alternative view is that even if entry is free in the sense that there are no sunk costs of formally entering the market, entrants with a zero or low network size will have no effect on competition. See discussion in Economides (2008). See Baumol, Panzar and Willig (1982). NERA Economic Consulting 26

36 Aspects of the Economics of the IDC industry Self-reinforcement of network size leads to concentration A consequential feature of network effects is that they are self-reinforcing. As customers value more the IDC providers that offer more peering options, in particular with magnet and other important customers, IDC providers with a larger IDC network attract more customers. After having attracted additional customers the IDC network will be even larger. Larger IDC networks will thus grow more quickly than smaller IDC networks. In turn, they will become even more attractive to additional customers. With strong network effects, small differences in network size between competitors at some point in time may lead to the larger IDC network becoming quickly larger whereas the smaller one will have more difficulty to grow or even shrink. The result may be a highly concentrated market with asymmetrical market shares, only few players and one dominant IDC network, and thus IDC provider. 67 As IDC providers with less attractive networks can charge lower prices and thus compensate customers for the lower value of their networks, network effects need not result in monopoly. In the extreme case, however, when network effects are very strong, the tipping of markets towards a single IDC provider may occur Multi-homing could mitigate network effects Many CAPs and ISPs follow a policy of using several IDCs run by different IDC providers in each metropolitan area ( IDC multi-homing ). 68 Multi-homing occurs on the one hand due to the interconnection needs of large CAPs and ISPs. For example, a large Tier 1 ISP may want to be present in several IDCs to interconnect with all Eyeball ISPs in a region. On the other hand many customers multi-home in order to increase redundancy in their interconnections, i.e. for risk diversification, and to ensure a level of competition between IDC providers. 69 For example, networks could multi-home, connecting to the same IX through different IDCs. Multi-homing can be less effective as regards Cross-Connects, however, when the alternative IDC offers a smaller and less attractive IDC network compared to the primary IDC where the customer chose to interconnect Asymmetries amplified by consolidation In IDC markets mergers can lead to a substantial increase in IDC network size of the merging firm. For example, the leading IDC provider could purchase all IDCs of a competing IDC provider in a metropolitan area. It could then connect the purchased IDCs with high performance fibre cables allowing the customers of the formerly competing IDC networks to cross-connect to each other and thus form a single larger IDC network See Economides (2008). According to one industry participants all big IDC customers multi-home in each metropolitan area. Information based on discussions with industry participants. As shortly discussed above, such cross-connects between different IDCs are no option for customers seeking very low latency interconnections, such as trading firms or video stream providers. NERA Economic Consulting 27

37 Aspects of the Economics of the IDC industry As described above, in markets with strong network effects even without mergers between IDC networks small differences between network sizes at some early point of market development may lead to extreme asymmetry or even monopoly. In markets with weaker network effects, such mergers could be decisive for the final market outcome. For example, by increasing its IDC network substantially above the network size of its next competitor a leading IDC provider could use a merger to gain critical competitive advantages, possibly leading to the same outcomes as in a market with strong network effects ( tipping ) Competitive advantages across geographic markets Due to the self-reinforcing tendency of network effects, large IDC providers have a competitive advantage over smaller IDC providers within a given relevant geographic market. Established IDC providers also have a competitive advantage in entering new geographic markets. In geographic markets where IDC networks do not yet exist or are relatively weak, it is important to shape the expectations of prospective customers regarding the IDC network size they will eventually be able to benefit from. The success of convincing customers that the relevant IDC network of a given IDC provider will be large and attractive will depend on the credibility of the IDC provider to attract the demand for interconnection of many prospective customers in the new geographic market. Established players in other geographic IDC markets will have more credibility to do so due to their existing activities and customer contacts than smaller players or new players in a market. Large IDC providers in other geographic markets may also have the ability and incentive to foreclose established IDC providers in a geographic market. 71 Due to their established business relationships with magnet customers, e.g. large, internationally active CAPs and Tier 1 ISPs, in other geographic markets, they can make bundled offers across IDC markets to those magnet customers. The bundled offers could be so attractive that the magnet customer decides to switch IDC provider in the new geographic market leaving IDC providers with a formerly attractive IDC network without that magnet customer. Case Study: Amsterdam metropolitan area According to industry participants, the most suitable measure of the importance of an IDC is the amount of traffic passing through it. Traffic is a good proxy for the IDC network s size everything else equal, a larger IDC network will result in more traffic and may also be indicative of the importance of individual IDC customers, as the most significant (or magnet) customers often generate high volumes of traffic. Traffic information is, however, not publicly available. Therefore precise shares of supply on the basis of actual traffic passing through IDCs are hard to come by. 71 See Carlton and Waldman (2002). NERA Economic Consulting 28

38 Aspects of the Economics of the IDC industry The Amsterdam Internet Exchange, AMS-IX, publishes information on IX traffic exchanged through each of the IDCs that host AMS-IX. 72 Table 1 displays these traffic data. Table 1 AMS-IX traffic volume by IDC Source: Interxion on the basis of AMS-IX website ( retrieved on 2 September On the basis of these IX traffic volumes by IDC, shares of IX traffic for the individual IDC providers can be calculated for the Amsterdam metropolitan area. These shares are shown in the following table. 72 The website of AMS-IX shows this traffic information graphically for each IDC. The graphical information was inspected by Interxion staff and provided to NERA. NERA Economic Consulting 29