Storage & Data Management Practices for the Long Term. Raymond A. Clarke Enterprise Storage Consultant Data Management Group Sun Microsystems, Inc.

Transcription

1 Storage & Data Management Practices for the Long Term Raymond A. Clarke Enterprise Storage Consultant Data Management Group

2 Long Term Digital Information Retention & Preservation needs a Holistic Approach Strategy: Apply IT Service Management, Information Lifecycle Management & Intelligent Fixed Content Management Move away from unmanaged repositories and silos of disconnected information Use applications Shift towards ILM-based practices and ILM-capable repositories Focus on retention and preservation, not creating silos Page 2

3 Reasons to Care The Digital Data Paradigm Shift 2010 New Digital Data (¼ Created, ¾ Replicated) 2006 New Digital Data (Created, Captured, Replicated) 6X Growth Emerging Economies growing 30-40% faster than mature Economies 161 Exabytes 70% Created by Individuals 95% Unstructured 988 Exabytes Enterprises responsible for 85% of this new data (Security, privacy, reliability, compliance) Source: IDC, The Expanding Digital Universe, 2007 Page 3

4 Reasons to Care - Key Survey Data Points 70% of respondents say they are highly dissatisfied with their ability to read their retained information in 50 years Current practices are too manual and too prone to error COST! - Value of data will vary over time Collaboration is recognized as necessary in order to define information retention strategies Remember that IT doesn't own the information. RIM, Legal, Business(Libraries, Museums, Archives) units and IT all have a part to play in the decisions applied to business records and should be sitting down at the table together. (Source: Respondent) Source: 100 Yr Archive Requirements Survey 2007 N=276 Page 4

5 Reasons to Care - Impact of Long-Term Retention Real Drivers: Legal, Security, Compliance, Education, escience, Research, Perpetuity and Business Risk ediscovery, Compliance, Legal Hold, Customer Privacy, Business Loss, Fines, Theft, Damage Impact: More information is being kept long-term, and it must be accessible and discoverable Lack of confidence that information can be retained long-term Logical and physical migration do not scale cost-effectively Page 5

6 Ke y Concern Logical and physical migration do not scale cost-effectively Only operating standard today is to migrate information physically (to new media) every three to five years and logically (to new formats) before the applications and readers die and become obsolete (every 5-10 years) A ne ver e nd in g, cos tly c yc le of m igra tion Practitioners are struggling to keep up with migration requirements. Only 30% claimed to be doing physical migration correctly on disk & none on tape or optical. Only 20% claimed they were confident in their ability to logically migrate some of the data. Info rmat io n is a t ri sk lon g- te rm! 6 Page 6

7 Storage Subsystem Price Trends (OEM price/equiv. unless otherwise noted; no capacity compression or utilization factors) 1000 Enterprise Class disk subsystem - capacity Tape Drives Price (OEM/Integrator - $/GB) 100 average Dis k Sys tem Performance Disk Drives (IDC) low Enterprise Class disk subsystem - performance s average Tape Drives Tapes low Desktop Disk Drives (IDC) ME MS /P 10 AT Ta pe S Tape Media Only average 1 ys tem s Tape Drives A Di sk rob e Syst ems Optical Disk Subsystems (IDC) low Desktop Disk Drives "last generation" 0.1 Holo gr aphic GA year Page 7

8 Long-Term is Real! More information is being kept long-term than you think and it is at risk! Longest Retention Requirement Longest Requirement 68 % over 100 Years 83% over 50 Years Requirements vary by organization type, information type, and compliance rules/risk 4% >10 Yrs beyond Project Life of Company 2% 1% Life of Product > 5 Yr >10 Yr >20 Yr >25 Yr 4% 4% 2% 3% 13% >50 Yr 15% >100 Yr 53% Permanent 0% 10% 20% 30% 40% 50% 60% Percent of Responses Source: 100 Yr Archive Requirements Survey 2007, N=104 Page 8

9 Retention and Preservation Retention: To keep and control information objects for specific periods of time. As a policy, a retention period for an information object or class of objects specifies the period of time the information is to be retained along with its administrative, legal, fiscal, historical, business, security, or other disposition requirements. (Source: SNIA and Society of American Archivists) Preservation: The processes and operations involved in ensuring the technical and intellectual survival of authentic information objects through time. (Source: NARA ERA) Page 9

10 ILM*- Based Practices a Solution Collaborate Dr iver s Stakeholders: IT, RIM, Legal Business/Operations, Security, Curators, Archivists, Librarians, etc. Identify Assets & Resources Classify Information Set Requirements Implement Services Measure and Improve Organizational Response Implement ILM Practices (ITIL* & ITSM*) Applications XAM ILM Managers Meta-data Common Services ILM = Information Lifecycle Management ITIL = Information Technology Infrastructure Library ITSM = IT Service Management SMIS 10 Page 10

11 Sun's Big 8 Archive Attributes Storage Infrastructure Features Security Encryption/Identity Management Storage repositories and information are to be protected against inappropriate access, viewing, change, copying, or misuse Auditable controls on access, privacy, integrity, and authenticity to ensure that information is not compromised. Accessibility Snap Mirror/Replication/Search/Query Keep all data available, but not necessarily on high cost storage > Moves data across the tiers according to access patterns Integrity - Continuous Data Protection SAM-Q/ZFS On-going backup & file system validation Scale SAM-Q/ZFS File system I/O performance/capacity scales linearly with the hardware Page 11

12 Sun's Big 8 Archive Attributes Storage Infrastructure Features Long Life Backup/Migration When the records retention requirement is met, all copies of records including series, preservation and backup copies should be permanently deleted Define disposition methods and policies by class of information Permanent Deletion is a key method to reduce the storage load, the legal risk, and the cost Open Standards (Access and data format) - SAM-Q Files can be recovered with or without proprietary methods Cost, Eco Responsible and Data Effective Migration/Deduplication Moves data across the tiers according to access patterns and/or current Page 12

13 Collaborative Archive Attributes Sun's Big 8 Security - IT, RIM, Legal, Business/Operations, Security,Curators, Archivists, Librarians Accessibility - IT, Business/Operations, Curators, Archivists, Librarians Integrity - IT, Legal, Security,Curators, Archivists, Librarians Scale - IT, RIM, Business/Operations, Curators, Archivists, Librarians Long Life - IT, RIM, Legal, Business/Operations, Security,Curators, Archivists, Librarians Open Standards (Access and data format) - IT, Business/Operations Cost and Data Effective - IT, RIM, Legal, Business/Operations, Security,Curators, Archivists, Librarians Eco Responsible - IT, RIM, Legal, Business/Operations, Security,Curators, Archivists, Librarians Page 13

14 Commodity Evolution of Disk Architecture SW O/S Storage design removes custom components over time HW CPU SW O/S HW CPU Open Storage Std H/W & Open S/W Thumper, Honeycomb RAIN Storage Clustered std servers EMC Centera, XIV Nextra SW O/S HW Proprietary CPU SW O/S HW SW O/S HW CPU Source: Wikipedia CPU SLED Modular Storage Custom server Data General CLARiiON, NetApp FAS, Sun Enterprise RAID Redundant Array of Inexpensive Disk Sun ST9900V EMC Symmetrix Single Large Expensive Disk IBM Page 14

15 The Evolution of Data Storage: Application Application File System File System Storage Storage Application Application Application File System Object API Object API Storage File System Intelligence Storage File System Storage DAS SAN NAS OSD ISD Page 15

16 Sun's Architectural Solution Integrated Systems Powered by Storage Density High Throughput Rates Lower Cost/GB Server + Intelligence + Storage + Arrays Page 16

17 Server and Storage Plug-Ins... NFS FCP SAS iscsi IB CIFS VTL OSD CAS IB XAM Web IB DAV Powered by Replication Security/Indentity Amber Road / Fishworks Galaxy Servers Niagara Servers Mirror/Snap Search/Query Thumper Series SPARC Servers Deduplication CIS Migration Honeycomb StorageTek Array Family ZFS THE LAST WORD IN FILE SYSTEMS Encryption StorageTek Tape Family Backup Compliance Page 17

18 Example Helpful Practices Classify your information (into a few common buckets) Set retention periods and delete expired information Free up space, only store what is required Include your databases Control the number of copies for protection and operational recovery and their locations Set policies for audits and perform them Measure and improve Page 18

19 Global Archive Requirements are changing fundamental storage thinking... Transport Technology Foundation OpenStorage Solutions Solution Innovation Trusted Advisor Page 19

20 Thank You. The Network is the Computer. TM Raymond Clarke (212)