Automatic Storage Management (ASM) Best Practices OUGN Marie-Louise Lindstrøm Service Delivery Engineer Oracle Support ACS

Transcription

1 Automatic Storage Management (ASM) Best Practices OUGN Marie-Louise Lindstrøm Service Delivery Engineer Oracle Support ACS

2 Agenda Concepts, Architecture New Features in Database Release 11g Tips and Techniques Examples, Demo

3 ASM Architecture - Overview 1. Automatic Storage Management (ASM) instance Oracle instance that manages the diskgroup metadata 2. ASM Disk Groups Logical grouping of disks Determines file mirroring options 3. ASM Disks LUNs presented to ASM 4. ASM Files Files that are stored in ASM disk groups are called ASM files, this includes database files

4 ASM Architecture - Overview ASM provides filesystem and volume manager capabilities built into the Oracle Database kernel Provides a simple storage management interface across all server and storage platforms Discover and associate (adding) disks under ASM management A disk can be partition of a physical spindle, the entire spindle, or a LUN in a RAID group set Logical volumes can be presented to ASM as disks, but not a good practice, since an extra layer is added to IO code path, and thus some level of overhead

5 Key Benefits of ASM I/O is spread evenly across all available disk drives to prevent hot spots and maximize performance ASM eliminates the need for over provisioning and maximizes storage resource utilization facilitating database consolidation Performs automatic online redistribution after the incremental addition or removal of storage capacity Maintains redundant copies of data to provide high availability, or leverage 3rd party RAID functionality Capable of leveraging 3rd party multi-pathing technologies

6 Key Benefits of ASM (cont.) An Oracle Database can contain ASM and non-asm files. Any new files can be created as ASM files whilst existing files can also be migrated to ASM RMAN commands enable non-asm managed files to be relocated to an ASM disk group Enterprise Manager can be used to manage ASM disk and file management activities ASM reduces Oracle Database cost and complexity without compromising performance or availability

7 Software Stack With and Without ASM APPL1 APPL2 APPL3 APPL1 APPL2 APPL3 DB1 DB2 File System DB1 DB2 Logical Volume Manager ASM OPERATING SYSTEM OPERATING SYSTEM

8 Disk Group Dynamic Rebalancing Automatic online rebalancing whenever storage configuration changes Only move data proportional to storage added No need for manual I/O tuning Online migration to new storage

9 Mirroring And Striping Mirroring : Redundant copies of data over multiple disks Primary data extents Secondary data extents Striping : Spreading data over multiple disks Unstriped Disks Striped Disks

10 ASM Instance Oracle instance to manage the metadata for disk groups All metadata modifications are done by the ASM instance Database instances connect to ASM instance to create, delete, resize, open, or close files Database instances read/write directly to the disks One ASM instance is required per node The ASM instance comprises shared memory structure (SGA) and background processes The ASM instance contains no physical files other than the password file and the parameter file Instance alert log file and trace files in the admin directories (release 10g), or diag directory (release 11g)

11 ASM Instance (cont.) Works with the RAC architecture to support clustered storage environments In RAC environments, there will be one ASM instance per clustered node, and the ASM instances communicate with each other on a peer-to-peer basis using the interconnect Generally named +ASM, in a RAC environment +ASM1, +ASM2, etc

12 ASM Instance (cont.) instance_type=asm asm_diskstring asm_diskgroups asm_power_limit sga_target or memory_target ; used in ASM release 11.1 db_cache_size=64m ; The buffer cache area is used to cache metadata blocks shared_pool=12m ; Used for standard memory usage (control structures, etc.) to manage the instance large_pool=128m ; Used to store extent maps processes Background processes RBAL This process performs global opens of all the disks in the disk groups, manages rebalancing operations ASMB Periodic messages are exchanged to update statistics and provide a heartbeat mechanism O00x A group of slave processes establish connections to the ASM instance

13 ASM Diskgroups Highest-level data structure in ASM Comparable to a LVM s volume group SQL> create diskgroup <name> disk <path>, <path> ; When the diskgroup has been created, metadata information, which includes creation date, diskgroup name, and redundancy type, is stored in the SGA and on each disk (in disk header) within the diskgroup A disk and a database file can only be part of one diskgroup A database can store its files in multiple diskgroups Recommended to have two diskgroups: Database area Flash recovery area

14 ASM Disks Unit of persistent storage for a disk group Accessed through normal OS interfaces Oracle must have read/write access Must be accessible to all nodes of a cluster May have different names on different nodes Cannot be a local filesystem file Must not include partition table Unit that fails if using redundancy Disks in a disk group are the same percent full Best practice is to have disks of same capacity (size, throughput) belonging to the same diskgroup ASM must scale to support at least 10,000 disks in a disk group

15 Allocation Unit ASM Disk 1 MB ASM disks are divided into allocation units Unit of allocating storage to a file 1 MB in size ; release 10g 1, 2, 4, 8, 16, 32, 64 MB ; release 11g Small enough to not be hot Large enough for efficient sequential access Many allocation units in a single file

16 Redundancy and Failure Groups External redundancy No mirroring, no failure groups Normal redundancy ; default Files are two-way mirrored, requiring at least two failure groups High redundancy Using three-way mirroring, requiring at least three failure groups Redundancy level set at creation time A failure group, a subset of a diskgroup, is a collection of disks that can become unavailable due to a failure of one of its associated components; e.g., controllers or entire arrays Disks in two separate failure groups (for the same diskgroup) must not share a common failure component The primary extent and its mirror copy are always stored in separate failure groups

17 Redundancy Levels External redundancy data exists only once Normal redundancy two-way mirroring one additional extent copy at least two failure groups High redundancy three-way mirroring two additional copies at least three failure groups

18 ASM Files Datafiles, control files, spfiles, and redo log files, incl archived log files, can be stored in ASM diskgroups Rman backups ASM file names are in the format of +diskgroup_name/database_name/file_type/tag_name.file_n umber.incarnation ASM Alias name format: instead of a file and incarnation number, a user-friendly string name is used For best practices, implement the Oracle Managed File (OMF) feature Log files, trace files and executables are stored outside ASM diskgroups

19 ASM Rebalance and Redistribution ASM allocates space in units called allocation units (AU) A database file is broken up into file extents (coarse and fine distribution) Changing the storage configuration, for example adding a disk to a diskgroup, will trigger a rebalance operation The objective of the rebalance operation is to provide an even distribution of file extents and space usage across all disks in the diskgroup Rebalancing involves physical movement of file extents If removing or adding several disks, it is best to add or remove drives all at once, this will reduce the number rebalance operations that are needed for storage changes

20 ASM Views V$ASM_DISK V$ASM_DISKGROUP V$ASM_DISK_STAT V$ASM_DISKGROUP_STAT V$ASM_OPERATION V$ASM_CLIENT V$ASM_FILE V$ASM_ALIAS V$ASM_TEMPLATE

21 ASMCMD Utility Command line interface providing access to the files and directories within the ASM diskgroups $ export ORACLE_SID=+ASM $ asmcmd ASMCMD> lsdg State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Name MOUNTED NORMAL N DATA1/ MOUNTED EXTERN N FRA1/ ASMCMD> cd data1 ASMCMD> ls X1020/ X1110/ ASMCMD> cd X1110/ ASMCMD> ls CONTROLFILE/ DATAFILE/ ONLINELOG/ PARAMETERFILE/ TEMPFILE/ spfilex1110.ora ASMCMD> ls DATAFILE/ SYSAUX SYSTEM UNDOTBS USERS ASMCMD>

22 Migrating to ASM ASM migration using Data Guard physical standby Use this method if your requirement is to minimize downtime during the migration. It is possible to reduce total downtime to just seconds by using the best practices described in the white paper ASM migration using Rman A simpler approach, but one that can result in downtime measured in minutes to hours, depending on the method used for migration ASM migration using DBMS_FILE_TRANSFER

23 Copying Files To and From ASM Diskgroups Across ASM diskgroups Between the OS file system and ASM diskgroups 1. Using Rman 2. Using the DBMS_FILE_TRANSFER package 3. Through ftp to the XML DB Virtual Folder 4. Using the ASMCMD command cp in release 11g

24 ASM New Features in Release 11g ASM Fast Mirror Resync ASM Preferred Mirror Read ASM Rolling Upgrades and Patches ASM Scalability and Performance Enhancements Multiple Allocation Unit Size SYSASM Privilege ASMCMD Extensions To use the new diskgroup features, the disk group compatibility attribute must be set to 11.1

25 Without ASM Fast Mirror Resync 1 ASM redundancy used 2 Disk access failure Oracle Database 10g and 11g 4 Disk added back: 3 Disk automatically dropped: Extents rebalanced All dropped extents re-created

26 With ASM Fast Mirror Resync 1 ASM redundancy used 2 Disk access failure Oracle Database 11g 4 Disk again accessible: Auto resync modified extents 3 Failure time < DISK_REPAIR_TIME> Access other extents

27 ASM Fast Mirror Resync Define DISK_REPAIR_TIME ALTER DISKGROUP dgroupa SET ATTRIBUTE 'DISK_REPAIR_TIME'='3H'; ALTER DISKGROUP dgroupa OFFLINE DISKS IN FAILGROUP contrl2 DROP AFTER 5H; Check time left before ASM drops offlined disk SELECT REPAIR_TIMER FROM V$ASM_DISK; Online disk after completing maintenance ALTER DISKGROUP dgroupa ONLINE DISKS IN FAILGROUP contrl2 POWER 2 WAIT;

28 ASM Preferred Mirror Read Allow local mirror read operations Eliminate network latencies in extended clusters Site A Site B P Extended Cluster S ASM_PREFERRED_READ_ FAILURE_GROUPS=DATA.SITEA ASM_PREFERRED_READ_ FAILURE_GROUPS=DATA.SITEB SELECT preferred_read FROM v$asm_disk; SELECT * FROM v$asm_disk_iostat;

29 ASM Rolling Upgrades and Patches Maximizes database availability in a cluster How it works Place cluster in Rolling Migration mode Bring down ASM on a cluster node Upgrade or patch software Restart ASM Stop Rolling Migration mode after upgrading all nodes

30 ASM Scalability and Performance Enhancements Variable size extents Grows automatically with file size Benefits Increase ASM file size Reduce memory utilization in SGA 100% automatic 63 disk groups 10,000 ASM disks 4 petabyte per ASM disk 40 exabyte of storage 1 million files per disk group Maximum file size: External redundancy: 140 PB Normal redundancy: 42 PB High redundancy: 15 PB

31 SYSASM Privilege SYSASM role to manage ASM instances separates duty of DBAs and storage administrators SQL> CONNECT / AS SYSASM SQL> CREATE USER ossysasmusername IDENTIFIED by passwd; SQL> GRANT SYSASM TO ossysasmusername; SQL> CONNECT ossysasmusername / passwd AS SYSASM; SQL> DROP USER ossysasmusername; SYSDBA will be deprecated for ASM management: Oracle Database 11g Release 1 behaves as in 10g In future releases SYSDBA privileges restricted in ASM instances

32 Multiple Allocation Unit Size Allocation Unit (AU) selected at disk group creation time and may be 1,2,4,8,16,32,64 MB Set Oracle MAX_IO_SIZE = AU size Default MAX_IO_SIZE is 1MB MAX_IO_SIZE determines default AU size Striping coarse stripe size = 1 AU fine stripe size = 128KB Higher performance for large sequential I/O (DW) Better leverage of Hardware RAID read-ahead

33 ASMCMD Extensions User created directories Templates Disk group compatibility Disk group name Disk names and failure groups md_backup md_restore repair Repair Bad Blocks $ asmcmd help lsdsk cp Copy files List ASM disks

34 Tips and Techniques Best Practices Should ASM be installed in a separate Oracle_Home? Yes, it is recommended to install and run ASM in a separate Oracle_Home. Use the latest RDBMS version, currently version For release 10g, use version plus patch for defect Where do I run the database listener from, the ASM Oracle_Home or the Database Oracle_Home? It is recommended to run the listener from the ASM Oracle_Home, or a separate Oracle_Home. This is particularly important for RAC installations since the listener is a node-level resource.

35 Tips and Techniques Best Practices (cont.) How to separate ASM administrators from DBAs? Choose a different OSDBA group for the ASM installation (asmdba) than for the Database installation (dba). The Database instance connects to ASM using OSDBA group of the ASM instance, which means that the software owner for each Database instance connecting to ASM must be a member of ASM's OSDBA group. In 11g, ASM administrator has to be member of a separate SYSASM group to separate ASM administrators and DBAs.

36 Tips and Techniques Best Practices (cont.) Can the Database instance and the ASM instance run different versions? Yes, the ASM instance can be at a higher version or a lower version than its client databases. There are two components of compatibility: 1. Software compatibility The compatible instance parameter 2. Diskgroup compatibility attributes: compatible.asm compatible.rdbms

37 Tips and Techniques Best Practices (cont.) What instance parameters need to be configured for ASM instances? The default parameter settings work perfectly for ASM release 11g. The only parameters needed for ASM release 11g: instance_type asm_diskstring asm_diskgroups Does the FILESYSTEMIO_OPTIONS parameter need to be configured in the Database instance when using ASM? No, the Database instance does I/O directly to the raw disk devices, whereas the FILESYSTEMIO_OPTIONS parameter is only for file systems

38 Tips and Techniques Best Practices (cont.) How many diskgroups should I have? The general recommendation is two diskgroups: 1. Database area 2. Flash recovery area What redundancy level should be used? Use normal redundancy level, or high redundancy level, if the underlying storage system does not provide mirroring. You can rely on external redundancy level if the underlying storage system provides mirroring and you determine that failgroups are not needed.

39 Tips and Techniques Best Practices (cont.) What size would be a good allocation unit (AU) for disk groups? In 10g, the allocation unit size is, by default, set to 1 MB. In 11g, it is recommended to use 4 MB. What is the best LUN size for ASM? There is no generic best size, in most cases the storage team will dictate based on their standardized LUN size. Communicate the ASM Best Practices and application characteristics to the storage team: Need equally sized / performance LUNs Do not use LUNs larger than 2 TB (minimum of 4 MB) The capacity requirement The workload characteristic (random r/w, sequential r/w) & any response time SLA

40 Tips and Techniques Best Practices (cont.) Would it be better to use BIGFILE tablespaces, or default (SMALLFILE) tablespaces for ASM? The use of BIGFILE tablespaces has no bearing on ASM, or vice versa. In fact, most database object related decisions are transparent to ASM. How to backup the ASM instance? All the metadata the ASM instance needs to startup is on on-disk structures (in the disk headers). There are no datafiles or control files that need to be backed up. The only file you can consider to back up is the instance parameter file. In 11g, use the ASMCMD md_backup to take a copy of the metadata.

41 Tips and Techniques Best Practices (cont.) When should I use RMAN and when should I use ASMCMD copy? RMAN is the recommended and most complete and flexible method to backup and transport database files in ASM. ASMCMD cp is good for copying single files Supports all Oracle file types Can be used to instantiate a Data Guard environment Does not update the controlfile Does not create OMF files

42 Tips and Techniques Best Practices (cont.) Is it possible to unplug an ASM disk group from one platform and plug into a server on another platform, for example from Solaris to Linux)? No, cross-platform disk group migration is not supported. To move datafiles between endian-ness platforms, you need to use XTTS, Datapump or Streams. How does ASM work with multipathing software? Oracle does not certify or qualify multi-pathing tools. ASM does, however, leverage multi-pathing tools. Use multi-pathing if the underlying storage supports it. Multipathing software is at a layer lower than ASM, and thus transparent to ASM. You may need to adjust the ASM_DISKSTRING parameter to specify only the path to the multipathing pseudo devices.

43 Tips and Techniques Best Practices (cont.) AIX specific information: Do not assign a PVID to ASM disks Assigning a Physical Volume ID (PVID) to an existing ASM disk will destroy the ASM disk header rendering the ASM disk unusable. How to repair damaged disk headers? If you encounter an error situation caused by corrupt disk headers leading to invalid ASM metadata, open a service request with Oracle Support to get technical assistance from the ASM team. Depending on the extent of the damage, it may be possible to manually repair the disk headers. However, if internal structures cannot be patched, the only solution will be to recreate the diskgroup and restore data from backup. Make sure you always have a good backup of your data. Also, store backup copies outside ASM diskgroups.

44 Tips and Techniques Best Practices (cont.) Should ASMLib be used on Linux installations? Yes, this is the recommendation. The ASMLib interface is not required to run ASM, however, it provides easier management of disk discovery and provisioning. Set the ASM_DISKSTRING parameter to ORCL:.

45 References Oracle Database Storage Administrator's Guide 11g Release 1 (11.1) B pdf ASM Collateral and Content Best Practices for Minimal Downtime Migration to ASM Oracle 10g Release 2 a_wp_10gr2_asmmigrationwithdg.pdf

46 References (cont.) ASM and Multipathing Generic Best Practices and Information Matrix March asm%20and%20multipathing%20best%20practices%20info%20matrix % pdf MyOracleSupport Note: ASM Technical Best Practices MyOracleSupport Note: How to perform Rolling Upgrade/Downgrade in 11g ASM

47