ORACLE AND NEARLINE STORAGE: STRATEGIES FOR INTEROPERABILITY

Transcription

1 ORACLE AND NEARLINE STORAGE: STRATEGIES FOR INTEROPERABILITY Ron Weiss, Oracle Corporation Paul Tsien, Oracle Corporation INTRODUCTION There is a class of storage subsystem, referred to as a Hierarchical Storage Manager (HSM), that is commonly used in today s large enterprise. An HSM is an integrated set of software and hardware, installed on a computer system, that provides enterprise-wide storage management services. HSMs can provide many different functions including automatic backup, disaster recovery, media management, archiving, migration, security, compression, high-speed file transfer, among others. One of the key features most HSMs also provide is the ability to store datafiles on less expensive offline or nearline storage devices, such as tape or WORM drives, while providing transparent disk-like, albeit delayed, access to the data. This can pose issues for Oracle database users since Oracle normally expects all of its datafiles to be online and immediately available. This paper describes several strategies that allow users to successfully use Oracle with HSM nearline storage technology. HIERARCHICAL STORAGE MANAGERS The idea of a Hierarchical Storage Management (HSM) system goes back over twenty years. Management Information Systems (MIS) organizations found they had a tremendous investment in storage subsystems and the expenses associated with managing them were growing extremely fast. This growth forced MIS to find ways to automate and reduce costs. Where the operating system (OS) and hardware offered numerous techniques to make effective use of CPU and memory resources, very little was available to manage disk and tape resources, and more importantly ensure that the data they held was secure. The idea of a storage hierarchy was developed and is illustrated in the following chart.

2 Speed and Cost/Byte Increase As You Move Up CPU Registers & Pipeline CPU Cache Real Memory Virtual Memory & Buffer Cache RAID Subsystems & Solid State Disk Conventional Disk Drives & JBOD Capacity Increases As You Move Down Tape Drives & Libraries CD-ROM/WORM/Optical Disk Drives & Juke Boxes Figure 1: HSM Concepts: Storage Hierarchy As you move up the storage hierarchy the speed of data access for applications and users improves, but costs increase and capacity decreases. When you move down the storage hierarchy, capacity and access times increase and costs decrease. In the upper portion of the storage hierarchy all management of the data is automatic. Once data is read into the computer, the OS and hardware include many facilities to automatically place data at the most cost-effective layer in storage hierarchy for processing. For example, only data being processed by the CPU is kept in registers or the CPU cache; while less frequently accessed data is in real memory and even less frequently accessed data resides in virtual storage. And of course, all of this is managed transparently by the system. For data in the lower half of the storage hierarchy, there is no automatic placement of data on the most cost-effective storage media without an HSM. All of this must be administered manually. HSM systems automatically move data to the media best suited to the performance needs and cost targets of the enterprise. The other issue that HSMs address is backup and disaster recovery. Are there adequate backups of the data resources of the enterprise? Where are they located? How frequently should backups be made? How is data recovered? How corporate data resources are disseminated beyond the large servers and mainframes in the corporate data center to the PCs, workstations, and workgroup servers can complicate backup and recovery. Lastly, it is more likely that a user or operator error will cause data loss than failed hardware or a natural disaster. All of these contingencies need to be insured against. Without an HSM, manual procedures must be established and executed to accomplish these tasks and ensure that corporate data resources are secured. HSM ARCHITECTURE While the implementation specifics are different from vendor to vendor, all HSM systems provide similar capabilities and features. Storage devices with similar characteristics are usually grouped into storage pools. An HSM catalog records metadata about the storage subsystem. For example, storage pool definitions, metadata about offline devices and media, and the location of backup copies of data is recorded in the catalog. The HSM software operates on and manages the data in the storage pools, based on user-defined storage management policies that are also stored in the catalog. The storage management policies define backup, archival, migration, and security rules the HSM enforces. These storage management policies can define things such as: How often backups should be made;

3 If files on high-speed disk have not been accessed in the last ninety days, they are to be migrated to the robotic tape library; Who is authorized to access and restore old versions of files, and how restoration is to be accomplished. Servers and Clients HSM Software HSM Server HSM Catalog Co-resident on an existing server or on a dedicated server Local or Network Connections Disk Farm Robotic Tape Library Offsite Storage CD-ROM Juke Box Figure 2: HSM Architecture Key to the architecture of HSM systems is their ability to provide two levels of transparency. Device transparency allows access to files via one interface (usually a virtual disk interface), regardless of the actual device on which the data is stored. Location transparency allows access to files without needing to know the actual location of stored data. Both device and location transparency are required to implement the dynamic storage hierarchy that is central to HSM. Transparency is implemented by a combination of media management and migration services, which are discussed in the following sections. HSM MEDIA MANAGEMENT Unlike disk-based files that are organized into filesystems, files stored on tertiary media, such as tape or CD-ROM, normally have no directory metadata structure into which they are organized. Without a metadata structure for offline/nearline media, management of this portion of the storage hierarchy is unwieldy and cumbersome. Most HSM systems use the HSM catalog as a repository for metadata about offline media files instead of a filesystem. File ownership, permissions/acl information, creation/modification/usage dates, file size, file location, and other information, can all be stored in the HSM catalog. This provides some necessary infrastructure to support easy access to offline/nearline media files. Storing metadata about offline/nearline media files in the HSM catalog enables automatic file location and retrieval for users. Many customers use robotic tape libraries or CD juke boxes that hold thousands of tapes and CDs. Automatically cataloging the metadata as files are created makes possible their automatic location, retrieval, and mounting at a later time. This has significant benefits for the user. It allows nearline media to be automatically migrated within the storage hierarchy and eliminates the need for manual procedures.

4 HSM MIGRATION SERVICES HSM provides the ability to automatically migrate files, based on user-defined criteria (storage management policies), from one class or pool of devices to another. Migration is the process of moving a file from one location to another while keeping the fact that the datafile has been moved transparent to any applications that might access it. This allows files that are no longer being frequently accessed, and are currently stored on expensive high-performance disk, to be automatically moved to cheaper and slower mass storage devices. It also enables the transparent and automatic retrieval of the file when accessed by an application or user at a later time. Migration transparency is usually implemented by replacing the original file locator with a link, stub file, or some other metadata construct that points to the new file location. The HSM catalog is also updated to record that the file has been migrated, where it was migrated from, and other information required to support transparency. Some HSM systems support partial migration as well as migration of whole files. When accessing a migrated file, some HSM systems can be configured to migrate portions of the file back to disk. These systems support the placement of blocks of a file that are frequently accessed on high-speed disk, and other infrequently accessed blocks of the file on other slower media. This is especially useful when large files are being accessed. Other HSMs have the ability to maintain a fixed number of blocks from the file on disk with the remainder of the file stored on tertiary storage. HSMs also have different strategies for satisfying application access to migrated data. Some HSM systems notify the application that the file is available only after the whole file has been migrated back to disk while others grant access when the first block has arrived on disk. This functionality will have a direct impact on system performance. ❶ Application accesses file /usr/disk5/appl/cad training_video329.avi ❷ HSM intercepts file request HSM Catalog ❸ HSM locates file from OS and catalog metadata ❺ HSM migrates (destages) data back to tape based on HSM policies ❹ HSM migrates (stages) data to disk Figure 3: HSM Transparent Migration The migration policies that control datafile placement are user defined and can take many forms. Policies can be based on datafile size, access frequency, file name prefix or suffix, age, among other variables. Policies may also identify datafiles that are never to be migrated. Migration services may take other forms as well. For example: Because of the volume of data, certain files are always stored on tape. When an application starts that needs to randomly access these files, they are migrated (staged) to disk automatically and transparently. When the application completes, the files are automatically migrated (destaged) from disk back to tape. Since this application needs to do random access updates to these files (this can not be done on most magnetic tape drives), and the amount of data involved is so large, the migration of the data to disk is warranted.

5 A CD juke box is used to store large amounts of data. In this case the user does not want to incur the overhead of file migration so after the CD is located and mounted (all provided automatically by HSM media management), the user accesses the data directly from CD. While CD data transfer and seek time is slower than hard disk, this performance differential does not justify migrating the data to disk first. Also, be aware that the HSM-managed storage might be attached to a remote server. In this case, data access will be throttled by the available intervening network bandwidth in addition to datafile staging. In summary, HSM media management and migration services provide transparent access by applications to files regardless of file location and device, and each HSM vendor implements these facilities in different ways. HSM VENDORS AND PRODUCTS Following is a list of vendors that offer HSM products. It identifies many of the common commercial products, but there may be others not listed. Vendor Product For More Information CA/Cheyenne ARCserve HP OpenView OmniStorage EMC Data Manager IBM ADSTAR Distributed Storage Manager (ADSM) IBM Data Facility Storage Management Subsystem (DFSMS) IBM & Oak Ridge National Laboratory High Performance Storage System (HPSS) Legato Networker Seagate Backup Exec Silicon Graphics Data Migration Facility (DMF) UniTree Software UniTree Central File Manager (UCFM) Veritas Veritas HSM ORACLE AND HSM INTEROPERABILITY In this section strategies for successful use of an Oracle database with nearline media and an HSM system will be discussed. While these strategies cover many of the interoperability issues of HSMs and Oracle, you must review the appropriate HSM and Oracle documentation for a more detailed understanding of the techniques described. ORACLE DATAFILES AND HSM HSM systems can introduce file access delays, and the impact on applications, such as Oracle, needs to be understood and planned for. Oracle expects that all needed datafiles are on a local disk and are immediately available and accessible. Files that are under HSM migration control may or may not be immediately available. A file that resides in a CD juke box needs to be mounted, and possibly migrated to disk, before Oracle can access it. It can take a significant amount of time to mount, or migrate, the file and this can have a very serious impact on overall Oracle performance if not monitored and controlled. The Oracle database server uses several different datafile types. Only some datafiles are appropriate for placing under HSM migration control. DATAFILES NOT TO PUT UNDER HSM CONTROL None of the Oracle system files should be placed under HSM control. If they are, database performance will be erratic and unpredictable. Oracle files that should NOT be placed under HSM migration control include:

6 All Oracle System Files, including: Control Files Parameter Files Oracle Binaries Datafiles of the System Tablespace Online Redo Log Files Trace Files Alert Files Datafiles of Tablespaces containing Rollback Segments Datafiles of Tablespaces containing Temporary Segments Users should either specify to the HSM that these datafiles are not subject to migration control or they should be placed outside of the HSM s control. DATAFILE CANDIDATES FOR HSM CONTROL In general, historical data that is infrequently accessed is the best candidate for management by an HSM and placement on nearline media. This assumes the nuances of the Oracle-to-HSM interaction are well understood and managed. Three types of files can be considered candidates for placing under HSM migration control They are: Archived Redo Log Files Backups of Oracle Datafiles Datafiles of User or Application Tablespaces Different strategies for using these file types on nearline media or an HSM will be discussed in the remainder of this paper. These strategies are: Read-Only Tablespaces Online and Offline Tablespaces Table Partitioning LOB and BFILE Management ARCHIVED REDO LOG FILES UNDER HSM CONTROL Archived Redo Logs make an ideal group of Oracle files to place under HSM control. This can be established by specifying LOG_ARCHIVE_DEST point to a device under HSM control. In conjunction with this, a number of HSM storage management polices can be defined to manage these files. For example: Retain archived logs on disk for two weeks. Migrate those logs over two weeks old to tape and retain the tapes for one year. Each night create two backups of each new archive log. One is retained onsite and the other is sent to an offsite storage location. Of course, the particular storage management polices used will depend on database activity, enterprise recovery requirements, and the ability of the chosen HSM system to implement these policies. BACKUPS OF ORACLE DATAFILES UNDER HSM CONTROL Backup of the database files is essential to the recoverability and availability of the database. An HSM can be used as the repository of database backups, thus automating many of the administrative tasks associated with backups. It can be used as a

7 large backup data store with the appropriate storage management policies to control backup aging, rotation, and offsite archival. USER AND APPLICATION TABLESPACES UNDER HSM CONTROL As previously discussed, it is not appropriate for Oracle system datafiles to be under HSM migration control. But it may be appropriate, and provide significant benefits to users, for some user or application tablespace datafiles to be under HSM control. While any user tablespace is a candidate, before you decide to use an HSM to control any database files, you should carefully examine and understand the usage pattern of the files in question. It may save many gigabytes of expensive disk storage to store files on nearline media, but the overhead and resulting performance may make this impractical. For example, while placing a table, or a portion of a table, on nearline media may be practical, also placing indexes for this table on nearline media may be problematic. The tradeoffs must be understood and evaluated before allowing files to be stored on nearline media or placed under HSM migration control. Fine tuning the storage management migration policies associated with tablespace datafiles is critical to a successful implementation. Again, focusing your investigation on infrequently accessed historical data should yield the best candidates. PRE-STAGING TABLESPACE DATAFILES At database open and checkpoint, Oracle accesses the headers of all datafiles under database control to verify data integrity and ensure recoverability. For files stored wholly on standard disk, this poses no problem. For files under HSM control that reside on nearline media, this can significantly lengthen the time these operations take and cause very poor database performance since all these files will need to be mounted or staged. On some HSMs this problem can be mitigated by using the HSM s ability to keep the blocks of the datafile that contain the Oracle headers on standard disk while keeping the remainder of the file on nearline media. This addresses the issue of Oracle open and checkpoint processing. If other portions of the nearline datafiles are subsequently accessed by an application, I/O time can be increased since this data may need to be mounted and staged to access the remainder of the datafile. Be aware that all HSM systems do not support this capability. READ-ONLY TABLESPACES Oracle supports read-only tablespaces in addition to the more typical read-write tablespaces. Oracle never updates the files, or file headers, of a read-only tablespace, and therefore these files can reside on read-only media such as CD-ROM. Therefore, read-only tablespace datafiles might be good candidates to store on nearline media under HSM control. Things to consider when evaluating this include: The database still needs read access to read-only tablespace file headers at open and checkpoint This needs to be considered and addressed as described in the previous section. When using Oracle Version or higher, there is an additional option that is described in the next section. Special procedures need to be followed to change a tablespace from read-write to read-only mode that are described in the Oracle documentation. Under Oracle8i, a tablespace may be made read-only online. Prior to Oracle8i the database must first be quiesced. Of course, this technique only applies to static portions of the database. Since read-only tablespaces cannot be modified, they do not need repeated backup. Even though regularly scheduled backups of read-only tablespaces are not required, at least one backup of the datafiles for this tablespace should be available. An HSM can ensure this backup exists and is available when required. When read-only tablespaces are placed on write-once media such as WORM or CD-ROM, the issue of Oracle upgrades needs to understood. When migrating from Oracle release to Oracle release, file headers, and potentially other on-disk structures, can, and do, change. If your files are on CD, the headers can not be rewritten. Since requiring Oracle users to rewrite potentially thousands of CDs is impractical when upgrading from one Oracle release to another, the database server provides dynamic and transparent access to these read-only tablespaces. Nothing is required to access these read-only tablespaces from old releases of Oracle but as a precaution customers taking this approach should test this as part of their migration testing and validation before upgrading Oracle. This is fairly simple to do. Where read-only tablespaces on CD exist, the customer should: 1. Create a small test DB with one or more of these CD-based read-only tablespaces.

8 2. Upgrade the test database to the new release. 3. Test access to the CD-based tablespaces. READ-ONLY DELAYED OPEN OPTION In Oracle a new init.ora parameter, READ_ONLY_OPEN_DELAYED, was introduced that modifies the database open and checkpoint datafile header access behavior. This feature was added to make it easier to use nearline media with Oracle. When this parameter is enabled, datafiles of read-only tablespaces are no longer accessed at database open or checkpoint. Instead, a read-only tablespace will only be accessed by Oracle when SQL is issued against it. So only the requester of the datafile is impacted by the HSM stage and mount activity. There are several side effects of using this technique, including: A missing or bad read-only datafile will not be detected at database open and will only be discovered when there is an attempt to access it. Some V$ table values will be different to reflect the new delayed open status of read-only tables. Note also that RECOVER DATABASE and ALTER DATABASE OPEN RESETLOGS will continue to access all readonly datafiles regardless of the READ_ONLY_OPEN_DELAYED parameter. For a complete discussion of these issues refer to the Oracle documentation. ONLINE AND OFFLINE TABLESPACES Another technique that can be used to manage datafiles that reside on nearline storage is to make use of the database s ability to place a tablespace in an offline state. A tablespace is normally online so that the data contained within it is available to database users. However, the user can take a tablespace offline by using the ALTER TABLESPACE command. When a tablespace goes offline, Oracle does not permit any SQL statements to access or reference objects contained in the tablespace. The online/offline state of a tablespace is recorded in the data dictionary, and hence, is remembered across instance startup and shutdown. If a tablespace that resides on nearline media is marked offline, access to it is bypassed at database open and checkpoint. When an application that requires this data needs to be run, the tablespace can be brought online. At this point the datafiles can be automatically mounted or staged by the HSM (assuming that the proper HSM policies have been defined) and be made available to the application. When the application processing is complete, the tablespace can again be placed in an offline state and the datafiles can be unmounted or destaged by the HSM. An advantage of this technique is that it can be used for readwrite tablespaces. The disadvantage is that the user, or application code, must manually initiate the online and offline state changes of the tablespace. Additional notes about this technique: You must use the ALTER TABLESPACE command to take datafiles offline. You must not use the ALTER DATABASE command. When taking a tablespace offline you must use the OFFLINE NORMAL clause. You must not use any of the other OFFLINE sub-options, such as TEMPORARY, IMMEDIATE, or FOR RECOVER. There are no checks made by the database to ensure that a tablespace is not being accessed when taking it offline. A tablespace will go offline without error even if users are accessing it. After the tablespace has been taken offline any access to the tablespace will cause errors. Thus, you should determine if a tablespace is being accessed before taking it offline. TABLE PARTITIONING Partitioning is a powerful new feature of Oracle8 that allows the rows in a table to be stored in different tablespaces (and datafiles) based on the value of a key. This makes managing large tables and indexes much easier and improves performance of query and maintenance operations. Partitioning allows DBAs to have much more control in managing tables and indexes, since all maintenance operations (including HSM access) can be directed at individual partitions, rather than being restricted to operations on the entire table or index. Partitioning improves manageability, availability, and performance in VLDBs and data warehouses.

9 If a table is partitioned, and the access frequency of the data is predictable, then you might be able to place some of the table partitions under HSM control. Again, the most common application of this would be for infrequently accessed historical data. For example, let s assume that: A sales table is partitioned by quarter -- i.e., each partition holds a quarter s worth of sales activity. Records of sales that occurred within the current quarter are constantly being accessed. Records of sales that occurred in the previous two quarters are accessed, on average, once a month. Records of sales that occurred over three quarters ago are rarely accessed. Sales Table Columns Account Number Product ID Sales Quantity Sales Date 01-JAN-97 to 31-MAR-97 SALES Table Q1 Partition Key 01-APR-97 to 30-JUN JUL-97 to 30-SEP-97 Q2 Q3 01-OCT-97 to 31-DEC-97 Q4 Partition Key Range Partition Name Table Partitions Stored On Disk and HSM Media Figure 4: Table Partitioning HSM storage management policies can be constructed to support this and similar scenarios by placing infrequently accessed partitions on HSM managed tertiary storage. New in Oracle8i are two features that can be employed to allow Oracle to better interoperate with nearline media and HSM. The partitioning key of a partitioned table is now fully updateable in Oracle8i. There is a new clause which may be specified when creating or altering attributes of a partitioned table. When ROW MOVEMENT is enabled, an update that changes a partition key in such a way that a row no longer belongs in its current partition will proceed, resulting in the updated row automatically being migrated to the appropriate partition. When ROW MOVEMENT is disabled, such updates are disallowed, resulting in an error being returned to the user. This functionality can be used to construct applications that automatically initiate the migration of rows in the table between regular disk and nearline media. Being able to identify and manage a subset of the data in a table is a level of control never provided before. Infrequently accessed rows, or rows containing large objects, can be segregated from other

10 rows in the table such that rows are placed on the most cost effective storage device.. Obviously, the movement of rows between partitions must be monitored and controlled to prevent excessive migration overhead. Indexed-organized-tables (IOT) may now be partitioned under Oracle8i. Online moves and reorganizations of IOT are also supported. This could be used to move an IOT, or IOT partition, between regular disk and nearline media without quiecsing the database. LOB AND BFILE MANAGEMENT Today s enterprise wants to build new, more sophisticated applications that incorporate unstructured non-scalar data such as graphic images, video and audio clips, x-rays, and text files into the database. Oracle supports this through the LOB (Large Object) column datatype. Since these objects can be very large (e.g., a single LOB object can be up to 4GB in size) costeffective storage management under an HSM can be very beneficial. There are two kinds of LOBs: Internal LOBs are stored within the database, are read-write objects, and fully participate in the transactional model of the database server. They can be further characterized depending on where the LOB data is stored. Inline LOB data is stored in a table if it is less than approximately 4,000 bytes in size. Out-of-line LOB data is stored in a tablespace and is accessed via a pointer in the table (LOB locator). The outof-line LOB data can be in the same, or different, tablespace as the table that points to it. External LOBs are stored outside the database, are read-only objects, and do not participate in transactions. An external LOB, or BFILE, is an operating system file that is accessed via a pointer in the table (BFILE locator). Internal LOBs Out-of-Line LOBs Tablespace 1 T ab External LOBs BFILEs Tablespace T ab l e Datafiles of Tablespace 1 l e Datafiles of the Tablespace L O B Tablespace 2 LOBs Stored In OS Datafiles Datafiles of Tablespace 2 Figure 5: Large Object (LOB) Storage

11 In the context of an HSM, out-of-line LOBs and BFILEs are of the most interest. When the application design dictates that the LOB data be incorporated into the database, out-of-line LOBs can be used to separate LOB data from other table data. This is done by placing the LOB data and table in a separate tablespaces. Oracle8i supports partitioned tables that contain LOBs. For each partition, LOBs may be placed in a separate tablespace from the other table data. Rows containing frequently and infrequently accessed LOBs can also be managed through the automatic row migration capabilities provided in Oracle8i. Partitioning described previously. When using BFILEs, the data resides in external operating system files, and therefore can easily be placed under HSM control. BFILEs have the advantage of allowing you to tune the HSM storage policy for frequently and infrequently accessed BFILEs. The disadvantage is that BFILEs are read-only objects and do not participate in transactions. Note that it is the user s responsibility to close any opened BFILEs after normal or abnormal termination of the program accessing them. If this is not done, the Oracle server considers these files open, and if the number of unclosed files exceeds the SESSION_MAX_OPEN_FILES value, you will not be able to open any more files in the session. POTENTIAL PITFALLS As described in the previous sections, there are a number of design issues that need to be planned for when attempting to have Oracle use nearline media or an HSM with. Others that should be considered include: The performance of database applications can be effected if database files are stored on nearline media. It will take more time to access files that need to be mounted or staged than it will to access files that do not require this additional processing. I/O and seek times for files on tertiary storage can also take longer. Database applications can be effected by the number of parallel accesses the HSM supports. If a CD juke box includes four CD drives and a database application is attempting to access datafiles on five different CDs, the HSM will not be able to satisfy this request. The degree of parallelism supported by the HSM hardware and software can impact application functionality and performance. HSM media or hardware recovery can cause issues for database applications accessing read-write files under HSM control. If a file has been partially staged to disk, and a disk or media failure occurs, both portions of the file must be treated as a unit even though the file is spread across two HSM devices. The HSM must report these type failures to the user so that appropriate database recovery can be made. Oracle datafile handling can be an issue for HSM systems. When Oracle opens a datafile it remains open for the life of that Oracle instance -- i.e., for performance reasons Oracle does not normally close datafiles. If opening a file causes a datafile residing on CD to be mounted, the CD can remain mounted for the life of that Oracle instance. When Oracle opens datafiles that are migrated from tertiary storage, staging disk space is consumed. This staging disk space can remain allocated for the life of that Oracle instance. The HSM makes resources available and Oracle can keep them allocated longer than other applications that access the HSM. If the HSM can destage or unmount open files these problems can be avoided, but many HSMs do not support this. If the HSM cannot destage an open file, all CD drives may become busy or all disk staging space may become exhausted causing application failures. One potential way to avoid this situation is to provide an adequate amount of hardware (e.g., staging disk space or CD drives) to satisfy all database requirements. Another approach is to take a staged tablespace offline. Be aware that taking a tablespace offline may not close the datafiles across the Oracle instance in Oracle releases prior to Oracle8i. If other sessions had accessed the tablespace prior to the offline command being issued, the datafiles will remain open. The user needs to ensure that all server processes that accessed the tablespace during the life of the Oracle instance have ended to be certain that the datafiles are closed. In Oracle8i taking a tablespace offline does close the datafiles. Again, the ideal solution is for the HSM to be able to operate on open files. Network-connected HSM systems can pose issues similar to those of NFS-mounted datafiles. Except under very special circumstances, Oracle does not support NFS-mounted read-write disks for database files because in some situations file writes do not actually occur even though the database has received acknowledgment that they have. When using a networked HSM system you should verify that the HSM guarantees synchronous writes to disk for any read-write tablespace placed under its control.

12 Oracle Parallel Server configurations can add an additional layer of complexity that needs to be evaluated. Multinode Oracle configurations are inherently complex, requiring more planning to successfully implement and integrate with an HSM. In Oracle Parallel Server configurations the HSM must be accessible to all nodes, as normal non-hsm datafile storage is. Also, in most cases, read-write access to the HSM cannot be provided because the HSM staging disks can only be made accessible to one node in an Oracle Parallel Server cluster. Be sure to review these issues before you attempt to put an Oracle/HSM system into production. SUMMARY AND CONCLUSION Customers have found that there are large costs and manageability issues associated with manually administering their storage subsystem. Some have chosen to use HSM systems and nearline media to address these concerns. Since much of the data that enterprises use is in Oracle databases, there is interest in how these two data management systems can be made to interoperate. As discussed in this paper, there are strategies that can enable the use of Oracle and HSM systems. We have also seen that there are times when it may not be possible to achieve the desired degree of interoperability. Care must be taken to understand the data and its access patterns and the capabilities of both Oracle and the HSM software. Some will find these techniques useful and in other cases it will be best to minimize Oracle and HSM interaction. Each situation and set of circumstances must be evaluated individually in the context of business objectives and the capability of the available technology.