White Paper: Bulk Data Migration in the Cloud. Migration in the Cloud

Transcription

1 Bulk Data Migration in the Cloud

2 Executive Summary Cloud storage has the potential to transform IT operations for organizations struggling to deliver a consistent level of service to a distributed user base and to keep up with the unrelenting growth of data. Cloud offers an affordable, highly scalable, resilient infrastructure that can be accessed from anywhere. However, cloud storage is still a nascent industry, and many organizations remain apprehensive about the unknown aspects of the technology. One of the most commonly cited points of apprehension is around cloud transfer speed. However, beyond this issue there lies a deeper anxiety about vendor lock-in. Cloud transfer speed is a legitimate concern, but it is possible for these performance issues to go undetected for years as data trickles in small daily increments. Vendor lock-in is a much more severe issue. If a customer wants to take their business elsewhere, they may be shocked to discover that their provider cannot support the migration. At Nasuni, engineers constantly stress test the capabilities of selected backend providers. Cloud transfer speeds is one of the key capabilities Nasuni tests, in order to determine how well cloud providers can support a bulk data migration. Nasuni is an enterprise storage company that uses cloud as a component of its services. As such, the company cares a great deal about the quality of the service offered by cloud storage providers (CSP). To that end, Nasuni has spent more than two years integrating with CSPs and developing a comprehensive battery of tests to ensure that its storage services are built on a firm foundation. Because Nasuni benefits from having multiple topnotch technology providers that compete to deliver the best services at the best price, it can act as a neutral, objective observer of the CSP industry, caring only which provider of commodity, back-end cloud storage provides the best quality and performance. In its December 2011 State of Cloud Storage Industry Benchmark Report, Nasuni examined all of the public CSPs, reporting on their performance, data stability and scalability. This report looks at the narrower but equally important issue of aggregate I/O performance within and across a selected set of providers to determine how long it would take to move a large amount of data in the cloud. In tests of cloud-to-cloud transfers, Nasuni looked at the top three CSPs from its December report Amazon S3, Microsoft Windows Azure and Rackspace and compared transfer rates among them. The results mirrored the results from the December CSP tests. Based on Nasuni s tests, it would take an estimated 40 hours to move a 12 TB volume from S3 to Azure, but only about 4 hours to move from Azure to S3. Rackspace would take an estimated 5 hours to move 12 TB volume to S3, and to move a 12 TB volume from S3 to Rackspace would take just under a week. Moving 12 TB between buckets within the Amazon S3 cloud would take about 4 hours. 2

3 Methodology So, how fast is fast enough when it comes to bulk migration of data in the cloud? The answer, like many answers related to storage performance, is that it depends. This kind of bulk movement depends on many factors: The performance of the CSP out of which one is moving data (CSP read performance); The performance of the CSP into which one is moving data (CSP write performance); The raw speed of the network between the CSPs and the machines one is using to move the data (CSPs cannot move data to each other directly); The load on the network between the CSPs and the migration machines at the time of the migration; The number of machines being used to do the migration; The specifications of the machines being used to do the migration (memory, CPU speed, disk speed, etc); and Quality and speed of the tools being used to do the migration. There are many reasons to do a migration: an organization merging accounts after an acquisition, one company giving data to another company, or a company deciding that another CSP could better support their requirements. Whatever the underlying motivation, they all result in one of two scenarios: moving data within a single provider, or moving data between providers. To evaluate providers migration capabilities, Nasuni set up a series of tests to examine these different cases: 1. Move data from one S3 account to another S3 account. 2. Move data from Microsoft Azure to Amazon S3. 3. Move data from Amazon S3 to Microsoft Azure. 4. Move data from Amazon S3 to Rackspace Cloud Files. 5. Move data from Rackspace Cloud Files to Amazon S3. Nasuni chose these three providers because they were the top three providers in the December CSP Report. And with S3 having the No. 1 storage cloud, Nasuni was primarily interested in discovering how long it would take to move data to and from S3, since S3 will almost certainly be one of the clouds that Nasuni will use for any given customer s data, unless the standings in the December CSP report were to dramatically change. To test these migrations, Nasuni needed a data set, a set of tools, and compute resources to run the tests. 3

4 Data Set Nasuni used a sample data set of roughly 12 TB, consisting of about 22 million files of mixed sizes for an average file size of about 550 KB. The files are all encrypted and compressed so moving them around poses no security threat. The data set lives in a bucket in an Amazon S3 account in the US Standard region. Ideally Nasuni would have used the entire data set for all these measurements, but since both time and money for testing were not unlimited, Nasuni chose to use the first 5% of the data set. That accounts for about one million files and about 200GB of data (the data set is not homogeneous). Tools Nasuni used the same core technology it created to evaluate CSPs in the State of Cloud Storage Industry Benchmark Report, adapted to measure data migration. As input, the tool takes a source CSP and a target CSP, and then selects a number of hosts (machines) to use to execute the data movement. When run, the tool first determines the list of files to copy from the source CSP and the list of files already present on the target CSP. It then identifies the difference files not on the target CSP and uses that list as the set of files to copy. The tool splits that list by the number of machines being used to move the data. (For example, if 10 machines were used to move 22 million files, each machine would be assigned 2.2 million files.) Items from the list are divided among the machines in a roundrobin fashion to ensure fairness and good resume-ability. On each machine, the tool uses 50 concurrent operations (threads) to move the set for that machine. So with 10 machines, Nasuni would have 500 concurrent operations moving data from the source CSP to the target CSP. During the test, the machines generally had a load average in the 15 to 20 range (when the targets could keep up), meaning that they were very busy. Data is moved between CSPs using an encrypted HTTPS connection, and the data is never stored on disk by the machines doing the movement. The data was encrypted at the source, so the tool never had visibility into the data. To measure how far the systems could go, Nasuni scaled up the number of hosts doing the migration from one machine to 40 machines. As the load increased, Nasuni received increasingly higher error rates from all the providers. Nasuni s tool has the ability to retry on errors, so it could run to completion despite the errors, but errors and the required retries have an impact on performance. Errors such as Server too busy from Azure, Internal Error from Amazon or the more frightening The resource could not be found error from Rackspace are not uncommon as one scales up the load on CSPs. All were eventually successful following appropriate retries. 4

5 Compute Resources To run the jobs, Nasuni selected Amazon EC2 as the cloud compute provider and, specifically, the EC2 m1.large machine type. Nasuni selected EC2 for reasons including: The data was already on Amazon S3 so using EC2 to access it reduced cost; Some of the tests were S3 to S3, so by using EC2, Nasuni had a possible best case test of moving everything within a single provider; and EC2 has a simple management console, and Nasuni has AMIs (machine images) on EC2 that are already compatible with its tools. It is certainly possible that the choice of a different compute cloud could have changed the results; however, the compute and storage clouds are separate enough that we did not believe that choosing an Azure compute cloud, for instance, would have much effect on the relative performance of each storage cloud to one another. While compute clouds can usually write faster to their associated storage clouds, the data transfer speed into the compute clouds is limited by the same factors as the data transfer speeds into the storage clouds. While Nasuni picked EC2 for the reasons above, the company s engineers were a bit surprised to find out that EC2 limits how many machines one may run by default, and that it is a relatively small number. Accounts are limited to 20 machines unless one contacts AWS to request more. To get by this artificial limit when testing with more than 20 machines, Nasuni just mixed machines from multiple accounts. Since these are generic machines in the cloud, machines can be mixed from multiple accounts or even multiple providers. In an ideal world, Nasuni would have repeated a number of combinations of these tests using compute resources from different providers, including Rackspace and Microsoft, but doing so for this study was not feasible due to cost and time considerations. 5

6 Case 1: Amazon S3 to Amazon S3 Migration While Amazon has some internal functions, like Copy Bucket to move data within Amazon S3 to the same account or to other accounts, and it s presumably faster than anything we could accomplish with external hosts, we used our tools to run this test so we could do the later comparisons. As one would expect, moving data around within Amazon scales well, although one can still see the per host operations per second dropping off as the workload scales up, which implies that there is a limit to how fast it can go. Since Nasuni was limited to only 40 machines for this test, engineers couldn t push Amazon S3 to its limit. Amazon S3 to Amazon S3 Migration Nasuni was pleased to see how performance remained relatively linear as load was added. While performance understandably degrades as one adds hosts, it does so predictably, which is exactly what one would want from a storage system. Nasuni estimates that it would take a minimum of four hours to move 12 TB of data from one Amazon S3 bucket to another. 6

7 Case 2: Microsoft Azure to Amazon S3 Migration In this next case, Nasuni examined how fast one can load data into Amazon S3 from the outside. As a general rule, writes are slower than reads on most storage systems. In addition, external bandwidth is much more limited than internal bandwidth. In the following chart, overall performance is lower than it was in the S3 to S3 test, and Nasuni appears to be approaching the maximum faster. Nevertheless, Nasuni was still unable to saturate the systems with the 40 hosts. Note that the limit Nasuni was approaching could have either been Amazon s write limit (bandwidth or technology), or just as easily Microsoft s read limit, again bandwidth or technology. Microsoft Azure to Amazon S3 Migration At their peak in this test, Amazon S3 was receiving over 270MB/s, meaning that Nasuni could have moved the entire 12TB test data set from Azure to S3 in about four hours. This test demonstrated that Amazon S3 had tremendous write performance and bandwidth into S3, and also that Microsoft Azure could provide the data fast enough to support the movement. 7

8 Now let s look at what happens when the transfer goes the other way. Case 3: Amazon S3 to Microsoft Azure Migration When the tables are turned, the results are strikingly different. At the lower host counts, Azure s performance and scalability were pretty impressive. They did, however, hit a wall at about 25 MB/s: Amazon S3 to Microsoft Azure Migration By the time two machines were executing the data migration, Nasuni had hit the limits of Microsoft Azure to accept data. It is difficult to determine from the outside whether the limits are due to Azure s incoming network, or a technology limitation. Usually at larger providers, bandwidth limits are symmetric one has the same upstream and downstream Internet connection speeds so the limitation may be in their architecture. Note that from the previous cases Nasuni saw with Amazon S3, Nasuni s engineers were starting to see a decline in performance at the really high host counts, so there is also a limit for Amazon S3. The difference is that testing could saturate Microsoft Azure with two hosts writing to them while Nasuni couldn t saturate Amazon S3 with 40 hosts. At its peak in this test, Microsoft Azure was receiving more than 30MB/s and engineers could have moved the entire 12TB test data set from S3 to Azure in about 40 hours. A week to move 12TB is not bad, but it is 10 times slower than moving into Amazon S3 or within S3. Nasuni also noticed that as the machine counts increased, error rates from Azure climbed. While one would expect that 1,000 operations per second should not challenge a public cloud, perhaps there are challenges at the account or container level in Azure s architecture. 8

9 Also worth noting was that performance on Microsoft Azure could vary significantly from run to run and appeared to be highly dependent on the time of day when the test was being run: Average Throughput Based on Time of Day All runs in the above chart use the same data set, the same number of machines (40), and all from the same Amazon S3 bucket to the same Microsoft Azure container (with the container emptied out before each run). As one can see, there is a high degree of variability in the results, with performance ranging from 30MB/s to 190MB/s. Nasuni generally saw the lower numbers during normal business hours. Unfortunately, the CSPs are not very forthcoming about why their performance would vary so greatly. Nasuni did not experience the same behavior with Amazon S3, and this measurement probably further indicates limitations in Azure s architecture or bandwidth, as other customers using the system appear to be affecting our results to a large degree. Nasuni estimates it would take a minimum of 40 hours to transfer 12TB of data from S3 to Azure. 9

10 Case 4: Amazon S3 to Rackspace Cloud Files Migration Running the same test to Rackspace as Nasuni did to Azure provided similar results to the S3 to Azure migration, but with lower overall performance: Amazon S3 to Rackspace Migration Like Azure, Rackspace has a limit that was easily hit with Nasuni s test. The surprising thing was that their limit is almost exactly 10MB/s, which is easily hit with only two machines. The near perfect number and the lack of variability in the results (unlike Azure) seems to imply that Rackspace has some kind of restrictions applied at an account, container or other level. As such, Nasuni estimates that it would take just under one week to transfer a 12TB volume to Rackspace from S3. Note that it is in the provider s interest to give end-users plenty of write performance. None of these providers charges end-users for data transfers into their service they all figure they ll make up the cost later with the storage, access, or data transfer-out fees. So Azure evidently makes whatever bandwidth is available accessible to users transferring in, while Rackspace sets a hard cap. Regardless of where the restriction is, one can see that Rackspace Cloud Files is in a different class than Azure. This poor transfer-in performance gave rise to concerns within Nasuni about all the other clouds that are springing up based on OpenStack. It is hard for Nasuni s engineers to imagine that these other clouds based on OpenStack would perform better than Rackspace s Cloud Files, since Rackspace is OpenStack s premier reference implementation. Of course some of these restrictions could be artificial and just settings applied at an account level, but these are default standard accounts from each of these providers. Rackspace could also scale better if the workload were spread out over multiple accounts or containers (as could Azure). But the CSPs do not provide guidance on where their bottlenecks are or how to avoid them. 10

11 Given this interesting result, the next thing to check is the read performance for Rackspace. Do they limit read performance too? Case 5: Rackspace Cloud Files to Amazon S3 Migration Similar to the test from Azure to Amazon, we moved the same data set from Rackspace to Amazon and measured the results: Amazon S3 to Rackspace Migration The chart above shows that Rackspace, like Microsoft, has read performance that appears to keep up with the Amazon S3 ingestion performance. Like the Microsoft case, as the performance appears to plateau, it is unclear whether testing is reaching Rackspace s read performance limits or Amazon S3 s write performance limits. It is interesting that the number is pretty close to the one from Azure to S3, which may imply that the limit is on Amazon s side. Nasuni estimates it would take a minimum of 5 hours to move 12TB of data from Rackspace to S3. 11

12 Data Deletion Regarding Rackspace, another by product of running these tests was having to delete data from the different providers. For each provider the test was simple: fetch (query) the list of files to delete from the test set, and then delete the list from the provider using 50 concurrent operations (threads) from a single EC2 host. The difference in performance in this area is astounding. Cleaning up data on Rackspace takes an order of magnitude longer than it does on other providers and about as much time as it takes to get it there in the first place. Nasuni was so shocked at how slow file deletion was on Rackspace, engineers set up a deletion test using Rackspace s compute cloud to see whether Amazon s compute cloud was causing the slowdown. The sluggish deletion speed was the same with either compute cloud. 12

13 Summary Unlike cloud computing, cloud storage has a lot of inertia that must be overcome to move it from one provider to another or even within the same provider. How much inertia can depend on which cloud storage providers are involved. The biggest difference among the three providers tested was their write speed. For example, in the chart below, Nasuni compares the estimated minimum time that it would take to move 12TB of data between the clouds. Estimated Minimum Hours to Transfer a 12 TB Volume It takes roughly the same amount of time to copy files to S3 from all three clouds. However, enormous differences arise when one attempts to copy files to Azure or Rackspace. As with Nasuni s Cloud Storage Report, Amazon once again ranks at the top of the list, a full 10 times better than the next best cloud storage provider. This speaks to Amazon s time in the market, architecture, operations, etc. Also not surprising from our previous report is that Azure is still in second place and Rackspace a more distant third place. For many cloud storage users, a 10MB/s ingestion rate (equivalent to a 80Mbps upstream Internet connection) is more than sufficient, and Rackspace s performance is nothing to be disappointed in. Nasuni s own corporate upstream bandwidth is limited at a paltry 10Mbps. But keep in mind that Nasuni offers customers multi-site global access to their data. So, for example, Nasuni has customers with 30 sites in different countries accessing the same data set. With a more reasonable upstream pipe of 30Mbps and 30 sites, one would quickly exceed the abilities of some cloud storage providers to take data. 13

14 There are all sorts of additional tests that would be interesting to run, given unlimited time and resources. How would Rackspace Cloud servers perform to do the migrations? How about using Azure compute services? Or what if we performed runs at different times of day to nail down when the CSPs are really busy and when they aren t? To return to the initial question: How long will it take to move an organization s data? The answer still may depend, but if one has 12TB of data with Nasuni, it can be moved securely without any service disruption in about four hours or less to S3, a couple days with S3 to Azure, and just under a week from S3 to Rackspace. While cloud storage has a lot of mass and is difficult to shift around quickly, that doesn t necessarily mean that the data is stuck in a single cloud storage provider. Nasuni has built technology that enables it to determine and select the best cloud storage provider for its customers. If times change, Nasuni can rapidly and transparently shift the data securely to new locations and providers without impacting customers. 14

15 About Nasuni Nasuni offers a powerful Storage Infrastructure as a Service: a uniform, complete storage solution delivered as a service, enabling distributed organizations to offload the management of the storage infrastructure (including backup, replication and offsite protection) and consume storage on-demand, without any change to the way users access information. With Nasuni, corporate data is uniformly available in all locations, enabling organizations to increase user productivity, streamline costs and improve business agility. Our team is made up of enterprise storage, security and networking industry veterans with a shared vision of transforming the way enterprise organizations view data storage. We believe that storage should be as easy to purchase, consume and manage as the electricity that keeps the lights on. Nasuni Corporation 2014, All Rights Reserved 313 Speen Street, Natick, MA WP_2.14