Effizientes Speichern von Cold-Data Dr. Dirk Gebh Storage Sales Consultant Oracle Deutschland
Program Agenda 1 2 3 4 5 Cold-Data OHSM Introduction Use Case Removing Cold Data from Primary Storage OHSM and Cloud OHSM new Features 3
Program Agenda 1 2 3 4 5 Cold-Data OHSM Introduction Use Case Removing Cold Data from Primary Storage OHSM and Cloud OHSM new Features 4
Cold-Data Infrequent access but significant efford Example: Svalbard Global Seed Vault forgotten vegetables => Log Term Preservation! 5
80% of Data Never Accessed After 90 Days 6
When Is Long Term Preservation Needed? It depends... 1. Delete trash data as soon as possible! Don t store, don t backup, don t archive That is most efficient for trash data Deduplication will not solve this problem! 2. Select worth data It can be hot or cold Put Right Data on Right Storage at Right Time 7
Long Time Preserving Data up to 80 Years required Example: Aircraft construction data Which storage medium and architecture will meet this requirement? 8
Bit Error Rate of Tape is 10,000 Times Lower Than Disk Seagate Constellation ES.2 StorageTek T10000D How long is data transferred before an uncorrectable error? 9
Bit Error Rate of Tape is 10,000 Times Lower Than Disk Seagate Constellation ES.2 StorageTek T10000D 9 days 165 years Equivalent to 40 RAID rebuilds per year per disk! Higher disk capacities = Longer rebuild times = Increasing data loss exposure 10
You can have all the tapes in the world, but that won't do you any good without hardware to read them. Jory Prum, LucasArts veteran http://www.polygon.com/2015/1/27/7921837/grim-fandango-remastered-interview-double-fine-disney-lucasfilm-sony 11
Don t forget the Cables, Interface and Drivers Old Device Cable Interface Driver Computer? Do Not Ignore It Do Validate it! Do Migrate it! Keep up with Technology Changes 12
All Data must be Accessable during Life Cycle! 13
Long Term Preservation Make sure data is unchanged Make sure data is valid Checksum protection? Re-archive if needed 14
Program Agenda 1 2 3 4 5 Cold-Data OHSM Introduction Use Case Removing Cold Data from Primary Storage OHSM and Cloud OHSM new Features 15
Oracle Hierarchical Storage Management Right Data Right Storage Right Time Lower Costs, Improve Efficiencies, Ensure Access 16
Archiving with Oracle HSM Lower Costs Backup and Archiving Tiered Storage Private Storage Clouds Object Cloud Storage Oracle Archive Cloud Improve Efficiencies Policy based movement Limitless scalability Horizontal scaling Automated media migration End to end data integrity validation (DIV) Fixity Ensure Access Open formats Product integrations High Availability Remote Archiving OKM support 17
Save 77% by Archiving Rarely Used Data Save 20x on Backup Software This savings alone pays for entire archive solution! $1,600,000 $363,000 Archive Software Archive Storage Backup Software Backup Disk Primary Disk Backup Only Backup and Archive 18
OpenStack Swift API Applications Standard POSIX File System API Storage Abstraction Layer (Oracle HSM) Manage Tiered Storage with Oracle HSM Primary Storage Archive Storage 19
Improve Efficiency with Policy-based Data Movement Types of Policies: Archiving, staging, releasing Number of copies Preferred media type Recycling tape volumes Media migration DIV Fixity Hierarchical Storage Management and Archive Management Policy Characteristics: Name pattern Owner Group Size Access age 20
How Oracle HSM Simplifies Management 1 Application writes file to StorageTek QFS file system 2 Archive copies created based on policy 3 Application requests file and the file is retrieved from Oracle HSM Application Disk Cache Archive Devices 21
Oracle HSM provides transparency for applications 1 2 Application only knows that it wrote a file And brought it back Application Disk Cache Archive Devices 22
Achieve Massive Scale with Oracle HSM Volume Capacity Throughput >1 billion files per file system Users with 7 billion+ files Scale to 1EB+ Multiple users with 40+PBs Achieve 10s of GB/s by scaling horizontally Ingest at 5+ GB/s 23
Increase Data Ingest Rates by 10X Tape I/O Horizontal Scalability Ingest >1 PB of data per day Use 10 servers to archive data instead of 1 Perform faster migration of data to new tape drive technology Enable the use of commodity servers 24
Enterprise Scalability: Horizontal Tape I/O License Cost Effective Scale Out Architecture Metadata Server (MDS) QFS Clients Metadata controlled by MDS MDS and QFS clients read/write to disk I/O for tape read/write comes from MDS and QFS clients No bottleneck I/O load spread evenly across all servers Primary Disk Tape Storage Tier Yellow = metadata / control path Black = disk data path Red = tape data path
Oracle HSM Automated Media Migration Leverage higher density formats with the efficiency of automatic migration Policy-based for automatic, behind-the-scenes migration Migrates multiple data streams without impacting production Seamless file system management 26
Oracle HSM Automates Data Integrity Validation Set policy-based DIV audits Verify integrity on each read, write, or any time Dedicate tape drives for DIV audits Self-heals Automatically Supports StorageTek T10000C/ T10000D and LTO 5/6/7 27
Oracle HSM Ensures Long Term Access with Fixity Support Validates file remains unchanged Verifies checksum on each read, write, or copy of a file Supports industry standard checksums: MD5, SHA-1, SHA-256, SHA-384, SHA-512 28
The Need for Open Formats Ensure Access Support movement of data between different systems Provide protection for data recall separate from the archive system Simplify migration from one storage medium to another 29
Oracle HSM supports Open Formats Tape Archive (TAR) Primary Use case: Long term storage Collects many files into one archive data set Standard established in 1988 and 2001 Linear Tape File System (LTFS) Primary Use case: File interchange Describes layout of data structures stored on tape Introduced in 2010 ISO standard established in 2016 30
Ensure Future Access with LTFS Format Support Import LTFS formatted tapes Export archive data onto LTFS formatted tapes LTFS Access Import Export FILE INDEX FILE LTFS Format Oracle HSM Archive FILE INDEX FILE LTFS Format 31
Oracle HSM Integrates with WebCenter Content to Ensure Access Throughout Data Lifecycle WebCenter Content Ingest Access Collaboration Search Reuse Storage Abstraction Tiered Storage Cost Effective Dynamic Policy Based Storage Virtualization Content Experts Storage Experts 32
Program Agenda 1 2 3 4 5 Cold-Data OHSM Introduction Use Case Removing Cold Data from Primary Storage OHSM and Cloud OHSM new Features 33
Data on Primary Storage Large installed NAS infrastructure Problem: 80% Cold Data! Slow NDMP Backup (full / incremental) all data: hot and cold Hot data Cold data
Removing Cold Data from existing Primary Storage Use Case Existing Infrastructure Oracle Partner Offering Oracle Offering
Program Agenda 1 2 3 4 5 Cold-Data OHSM Introduction Use Case Removing Cold Data from Primary Storage OHSM and Cloud OHSM new Features 36
Applications Standard POSIX File System API Automation Is Key Enabler Simplifying Administration & Improving Storage Efficiency Without automation everything stays on expensive primary disk With automation via policy 1. After 5 minutes create a copy to tape for Data Protection 2. After 30 days of inactivity make copies to tape and/or cloud Primary Disk 3. Make additional copies to tape 4. After 1 year remove from disk archive Capacity Disk Digital Tape Cloud 37
Private Cloud Infrastructure Solution OpenStack Swift Object Store Ingest & Access Object store with support for OpenStack Swift RESTful API Web interface for tape storage Swift Oracle Hierarchical Storage Manager Flash Disk Tape Make low cost tape easily accessible to web users Enable object storage Custom metadata, no namespace limitation, near infinite scalability Eliminate backup software Deliver a cold storage service directly to end users 38
Applications OpenStack Swift REST API Create On-Premise Storage Clouds with Oracle HSM and OpenStack Swift Primary Disk Tape Archive 39
Cloud Object Storage Storing on Disk Storage Nodes! OpenStack Swift architecture Proxy: authenticates Users and routes requests to the appropriate storage nodes Account: tracks the containers associated with an account Container: tracks the objects associated with a container Object: stores objects in the file system 3 copies to different storage nodes GET / PUT / DELETE 40
Cloud Object Storage with OHSM Backend External Swift Clients GET / PUT / DELETE single data copy up to 4 data copies handeled by OHSM system 41
Putting It All Together Store data on the optimal storage device in the optimal location Applications / Data Source Application Interfaces (NFS, CIFS, HTTP, FTP, WebDav, OpenStack Swift) (control path) Data path QFS Clients Oracle Cloud Software Appliance Archive Metadata Server NFS Gateway On Premise Primary Disk On Premise Tape Storage Remote Site / Private Cloud Cloud Tier 42
Oracle HSM Featured at Oracle OpenWorld! Announced: Oracle HSM integrates with the cloud for huge cost savings 43
Oracle HSM Support for Oracle Storage Cloud - Archive Service Coming Soon! Breakthrough cost for off-site data protection Google Nearline Disk Storage $0.011 $/GB/Mo. Oracle Storage Cloud Service Archive Service Oracle HSM Archive Storage Amazon Glacier $0.007-0.013* $/GB/Mo. * Region dependent $0.001 $/GB/Mo. $12,000/PB/Yr. Archive 44
Program Agenda 1 2 3 4 Cold-Data OHSM Introduction OHSM and Cloud OHSM new Features 45
Strong Investment in Oracle HSM Up to 5.2 5.3 6.0 6.1 - New! Archive Virtualization Automation Data integrity validation (DIV) WORM enhancements Stager enhancements Horizontal scaling Cloud Automated DIV Linear tape file system (LTFS) import/export Media migration Fixity Expanded LUN size Extended attribute metadata 46
New Advancements in Oracle HSM 6.1 Media Migration Accelerates migration Frees CPU cycles and personnel Fixity Enhancement Supports standard checksum algorithms Verifies file integrity throughout entire lifecycle Extended Attribute Metadata Improves file system performance Expanded LUN Size Supports hardware advancements Increases scalability 47
Solve Migration Challenges with Oracle HSM Automated Media Migration Migration Challenges Time: months or years to complete Cost: hardware, network, and personnel Complexity: manage multiple formats Oracle HSM Automated Media Migration 1. Policy-based migration 2. Eliminates data staging 3. Enables parallel migration 4. Reduces resource burden 5. Fastest path to single format 48
Improved Media Migration with Oracle HSM Migration Challenges Time: months or years to complete Cost: CPU hungry with little business value Complexity: manage multiple formats Oracle HSM Automated Media Migration TM 1. Memory-Assisted Copy 2. StorageTek Direct Copy 49
ctrl data data ctrl ctrl data Accelerate Migration with Oracle HSM Automated Media Migration TM Current migration method With Oracle HSM Memory-Assisted Copy TM With Oracle HSM and StorageTek Direct Copy T10000A/B/C/D LTO3/4/5/6 TS1130/40/50 Disk data ctrl HSM HSM T10000D Copy to disk and then to tape Slow, CPU-intensive migration Copy to memory and then to tape Fast migrate up to four streams Copy directly from tape to tape Fast/parallel migrate unlimited streams 50
scsi/status data scsi/status data Migrate All Data or Repack a Portion of Data Migrate Repack Expired Active Expired Active Expired Active Active Active Active Migrate entire contents of multiple cartridges to a single cartridge Repack only active source files sequentially to a single cartridge 51
scsi/status data Migrate All Data or Repack a Portion of Data Automated Media Migration Feature at Scale Migrate Repack Expired Active Expired Active Expired Active Expired Active Expired Active Expired Active Expired Active Expired Active Expired Active Active Active Active Active Active Active Active Active Active Migrate entire contents of multiple cartridges to a single cartridge Repack only active source files sequentially to a single cartridge 52
Fixity Enhancement Ingest file and checksum Verify checksum Verify checksum for each read /write /copy End-to-end fixity Standard checksums: MD5, SHA-1, SHA-256, SHA-384, SHA-512 Solaris library utilized for hardware acceleration Checksum stored in extended attribute 53
Dataflow of Setting and Using Fixity Ingest Data Write to disk cache Access file for archive Validate Access file for stage Validate 1 2 3 4 5 6 Run ssum or API Calculate the hash and store OR calculate and compare with given hash Update attributes Archive to tape Stage to disk cache 54
Improved Performance for Extended Attribute Metadata Metadata Tier Inodes / POSIX Metadata Metadata Tier Inodes / POSIX Metadata Extended Metadata Disk Cache Tier Extended Metadata Fast Access Data Disk Cache Tier Fast Access Data Archive Tiers Archive Data Archive Tiers Archive Data Oracle HSM 6.0 Oracle HSM 6.1 55
Max LUN Size Increased 8x Oracle HSM 6.1 Prior to 6.1 16 TB LUN 4 PB 128 TB LUN 32 PB 56
Oracle HSM 6.1 HW / SW Support Updates Solaris 11.3 LTO7 Tape Drives ACSLS 8.4 57
Put Right Data on Right Storage at Right Time Lower Costs Improve Efficiencies Ensure Access Cold - Data Archiving With Oracle HSM 58
59
Safe Harbor Statement The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle.