Backup, Recovery and Archive in the Clouds May 2, 2013

Transcription

1 Backup, Recovery and Archive in the Clouds May 2, 2013 Henry Newman CEO/CTO Instrumental, Inc. 1 of 17

2 Agenda Background of a recent study Issues to consider for backup/restore Issue to consider for archive Other considerations 2 of 17

3 Background of a recent study Review of requirements and offerings 3 of 17

4 Recent MHS.org Study Report location: s/instrumental_mhsreportfinal_public_v2.pdf Redacted version with no pricing information What MHS required Archive for state records, examples include Birth records Newspapers Oral histories Economic history Other high value collections 4 of 17

5 MHS requirements MHS has proprietary and copyrighted information for which it has contractual obligations to protect In addition, there are contractual requirements to return some data collections to owners upon completion of the collections For this reason, maintaining the integrity and availability of the data in the collections has significant legal ramifications 5 of 17

6 Issues Reliability Vendors do not tell you what they are doing Since no cloud vendors provide any data integrity guarantees, a requirement is that there be checksums generated before the data is moved to the cloud to ensure that data is correctly stored in the cloud and that the data retrieved from the cloud is identical to what was stored Some of the cloud storage providers have this capability while others do not Most cloud providers have similar standard reliability capabilities by having multiple data copies 6 of 17

7 Results Provider / Service Amazon S3 Amazon Glacier Google Cloud Storage Tessella Preservica VISI ReliaCloud SDSC Cloud Storage DuraSpace DuraCloud (S3) DuraSpace DuraCloud (S3 and Glacier) IBM SmartCloud FujiFilm Permivault FujiFilm Permivault Client Code 42 CrashPlan Pro 7 of 17 Data Integrity Reliability Scalability Limited checksums Average Almost unlimited Limited Multiple Almost checksums tape copies unlimited No Almost checksums Average unlimited Checksums Multiple and CRC cloud copies Same as S3 Limited Cannot checksums Average support MHS Automatic Average but Almost verification no tapes unlimited User run Single cloud checksums copy Same as S3 Retention and Portability Availability Data Ownership Preservation Functionality Total 3-Yr. Costs Not easy to move Average Similar to others None Medium Not easy to Lower since Similar to move on tape others None Low Not easy to None in Contract move contract concerns None Medium Multiple Multiple Similar to Developed providers cloud copies others for this? Somewhat Similar to Some limited Average others claimed High Claimed to Concerns Similar to be easy about disks others None Medium Multiple Similar to Some providers Same as S3 others claimed Medium User run Multiple Multiple Similar to Some checksums cloud copies Same as S3 providers Same as S3 others claimed Medium Limited None in Similar to checksums Average Unknown Unknown contract others None? Custom On-site and Almost Somewhat Similar to plans cloud unlimited limited On-site copy others Limited Low Custom Almost Somewhat Similar to plans Cloud only unlimited limited Average others Limited Low Limited More limited Somewhat Similar to checksums Average than S3 limited Average others None Low

8 Issues to consider for backup/restore Some things to think about 8 of 17

9 Background Backing things up is easy Hook up your system to the internet Put some backup product on the system Wait a few weeks or a few months Presto, you are backed up Getting them back to more difficult Same is true for local backups How many times have you had backup fail 9 of 17

10 Testing cloud backups A few things to consider: What is your recovery time objective? Backup is not about backup but about restore What is the impact of dedup? Do you have the CPU power and memory bandwidth to achieve your objectives? Is what you put in what you get back? How are you going to tests data integrity? Is what you need really an archive or a backup? Archives are different For example there is no versioning These are just the tip of the iceberg Local requirements must be considered 10 of 17

11 Time to move data 1 PB 10 PB 40 PB 100 PB Network Days to replicate or move OC OC OC OC OC OC of 17

12 Issue to consider for archive Archives are different but some things are the same 12 of 17

13 Testing cloud archives Is what you put in what you get out How is integrity checked end to end What about things like silent corruption? Shameless plug for tutorial I am giving on this topic at IEEE Mass Storage Conference next week tml Slides will be up in a few weeks What about the time to migration to new technology? Architecting for availability and meeting SLAs is difficult when migrating Becomes a bigger issue when namespaces are broken up into smaller chunks What is you need to bring it all back? 13 of 17

14 Other considerations Last but not least 14 of 17

15 What to ask It is good to have questions answered but answers in writing are a must It the vendor will not put it in writing and commit to it in a contact it does not count Reliability numbers are not given and there are no commitments Be prepared for that answer There are risks in clouds You will be accepting the risks not the providers and getting information is difficult to impossible 15 of 17

16 Final Thoughts 9 s matter Data Loss in Bytes 9s Data Reliability % 1 PiB 50 PiB 100 PiB 500 PiB 1 EiB 2 99% 11,258,999,068, ,949,953,421,312 1,125,899,906,842,620 5,629,499,534,213,120 11,529,215,046,068, % 1,125,899,906,843 56,294,995,342, ,589,990,684, ,949,953,421,312 1,152,921,504,606, % 112,589,990,684 5,629,499,534,213 11,258,999,068,425 56,294,995,342, ,292,150,460, % 11,258,999, ,949,953,419 1,125,899,906,838 5,629,499,534,188 11,529,215,046, % 1,125,899,907 56,294,995, ,589,990, ,949,953,438 1,152,921,504, % 112,589,991 5,629,499,531 11,258,999,063 56,294,995, ,292,150, % 11,258, ,949,956 1,125,899,913 5,629,499,563 11,529,215, % 1,125,900 56,294, ,589, ,949,938 1,152,921, % 112,590 5,629,500 11,259,000 56,295, ,292, % 11, ,950 1,125,900 5,629,500 11,529, % 1,126 56, , ,938 1,152,896 With even 12 9s you get data loss at 1 PiB of data It is in 1 file or 1 byte in 1,126 files making them all corrupted Thinking about things in 9s is important for both you and cloud providers Sadly they will not give you any information 16 of 17

17 Thank you! Thanks for listening 17 of 17