ASM ON LINUX. May Author: Esteban Bernal. Automatic Storage Management Industry's first and only database file system

Transcription

1 ASM ON LINUX. May 2008 Automatic Storage Management Industry's first and only database file system Author: Esteban Bernal

2 ASM ON LINUX Configuring Automatic Storage Management On Linux. There are 3 storage options to setup ASM on Linux: 1) Raw devices (10.1 and Onwards) 2) (10.1 and Onwards). 3) Block devices (10.2 and Onwards). Author: Esteban Bernal (Storage Team) 2

3 ASM ON LINUX Creating Disks Partitions for ASM In order to use a disk (e.g. SAN) in Automatic Storage Management, the disk must have a partition table. Oracle recommends creating exactly one partition for each disk containing the entire disk. You can use any physical disk for Automatic Storage Management, as long as it is partitioned. Use IDE, SCSI, or RAID devices. Author: Esteban Bernal (Storage Team) 3

4 ASM ON LINUX Creating Disks Partitions for ASM Identify the device name for the disks that you want to use. # /sbin/fdisk -l Depending on the type of disk, the device name can vary. Oracle recommends that you create a single whole-disk partition on each disk that you want to use. Use either fdisk or parted to create a single whole-disk partition on the disk devices that you want to use. Author: Esteban Bernal (Storage Team) 4

5 ASM ON LINUX Creating Disks Partitions for ASM Disk Type Device Name Format Description IDE disk /dev/hdxn /dev/hda1 SCSI disk /dev/sdxn /dev/sdb1 RAID disk Multipath disk (e.g. emcpower) /dev/rd/cxdypz /dev/ida/cxdypz /dev/emcpowerxn /dev/dm-xx /dev/ida ida/c0d1 /dev/emcpowera1 Author: Esteban Bernal (Storage Team) 5

6 ASM ON LINUX 1. Raw Devices. 1) Identify the existing partitions: Author: Esteban Bernal (Storage Team) 6

7 ASM ON LINUX 1. Raw Devices. 2) Open the /etc/sysconfig/rawdevices and add a raw device for each partition that you created and you want to bound: Author: Esteban Bernal (Storage Team) 7

8 ASM ON LINUX 1. Raw Devices. 3) To bind the partitions to the raw devices, enter the following command: Author: Esteban Bernal (Storage Team) 8

9 ASM ON LINUX 1. Raw Devices. 4) For the raw device that you created for ASM, set the owner, group, and permissions on the device file: Author: Esteban Bernal (Storage Team) 9

10 ASM ON LINUX 1. Raw Devices. 5) Verify the raw disk through ASM: Author: Esteban Bernal (Storage Team) 10

11 ASM ON LINUX 2. Block Devices 1) Identify the existing partitions: Author: Esteban Bernal (Storage Team) 11

12 ASM ON LINUX 2. Block Devices 2) Use /sbin/partprobe diskpath on each node to update the kernel partition table : Author: Esteban Bernal (Storage Team) 12

13 ASM ON LINUX 2. Block Devices 3) For the block device that you created for ASM, set the owner, group, and permissions on the device file: Author: Esteban Bernal (Storage Team) 13

14 ASM ON LINUX 2. Block Devices 4) Update asm_diskstring parameter with the block devices directory: Author: Esteban Bernal (Storage Team) 14

15 ASM ON LINUX 2. Block Devices 5) Create a diskgroup using block devices: Author: Esteban Bernal (Storage Team) 15

16 3. is a supported library for the ASM feature. The objective of is to provide a more streamlined and efficient mechanism for identifying and accessing block devices used by ASM diskgroups. This API serves as alternative to the standard operating system interface. An is a storage-management interface between Oracle Kernel and disk storage. The kernel driver is released under GNU General Public License (GPL), and Oracle Corporation freely delivers an for Linux Platforms. This library is provided to enable ASM I/O to Linux disks without the limitations of the standard UNIX I/O API. oss.oracle.com/ INTRODUCTION Author: Esteban Bernal (Storage Team) 16

17 BENEFITS The main functions are grouped into three collections of functions: Device discovery functions must be implemented in any. Discover strings usually contains a prefix identifying which this discover string is intended for. For the Linux provided by Oracle, the prefix is ORCL:*. Note Device Persistence and Oracle Linux ASMLib I/O processing functions extend the operating system interface and provide an optimized asynchronous interface for scheduling I/O operations and managing I/O operation completion events. These functions are implemented as a device driver within the operating system kernel. The performance and reliability functions use the I/O processing control structures for passing metadata between the Oracle database and the back-end storage devices. They enable additional intelligence on the part of back-end storage. Author: Esteban Bernal (Storage Team) 17

18 Oracle Linux ASMLib Installation You can download the Oracle software from OTN. There are three packages for each platform. The two essential packages are the oracleasmlib package, which provides the actual ASM library, and the oracleasm support package, which provides the utilities to configure and enable the ASM driver. The remaining package provides the kernel driver for the ASMLib (loaded into the OS kernel at system boot). Example: oracleasm-support i386.rpm oracleasmlib i386.rpm oracleasm el i686.rpm Author: Esteban Bernal (Storage Team) 18

19 Oracle Linux ASMLib Installation After the software is installed, you need to make the ASM driver available by executing the /etc/init.d/oracleasm configure command. This operation creates the /dev/oracleasm mount point used by the to communicate with ASM driver. When using RAC, instalaltion and configuration must be completed on all nodes of the cluster. # /etc/init.d/oracleasm configure Author: Esteban Bernal (Storage Team) 19

20 Oracle Linux ASMLib Installation In order to place a disk under ASM management, it must first marked to prevent inadvertent use of incorrect disk by ASM. This is accomplished by using the /etc/init.d/oracleasm createdisk command: # /etc/init.d/oracleasm createdisk With RAC, this operation needs to performed only on one node because it is a shared-disk architecture. However, the other nodes in the cluster need to ensure the disk is seen and valid. Therefore, the other nodes in cluster need to execute the /etc/init.d/oracleasm scandisks command. # /etc/init.d/oracleasm scandisks Author: Esteban Bernal (Storage Team) 20

21 Oracle Linux ASMLib Installation After the disk are marked, the ASM initialization parameter can be set to appropriate values. ASM_DISKSTRING='ORCL:*' On 10gR2 and onwards if the disks are not discovered using the string ORCL:*, the alternative option is using /dev/oracleasm/disks. This could be set in parameter asm_diskstring or using this path in the DDL statement when creating a diskgroup or adding a disk. This is only a temporary workaround until you fix the discovery issue and it does not pretend to be final solution since with this action you are bypassing the API. Author: Esteban Bernal (Storage Team) 21

22 Oracle Linux ASMLib Installation Checklist. I. Install the packages (on each node for RAC). Install oracleasm-support, oracleasmlib and kernel-related packages. II. Configure (on each node for RAC). Load ASM driver and mount ASM driver file system. Use the oracleasm script with the configure option III. Make disks available to by marking disks using oracleasm createdisk (only on one node for RAC). IV. Make sure that disks are visible using the oracleasm scandisks (on all the nodes for RAC) V. Use the appropriate discovery strings for this. Author: Esteban Bernal (Storage Team) 22

23 Oracle Linux ASMLib Installation Oracle Linux Installation: 1. Enter the following command to determine the kernel version and architecture of the system: # uname -rm 2. Download the required ASM library driver packages from OTN Web site. You must download the following three packages, were version is the version of the ASM library driver, arch is the system architecture and kernel is the kernel version you are using: oracleasm-support-version.arch.rpm oracleasm-kernel-version.arch.rpm oracleasmlib-version.arch.rpm Author: Esteban Bernal (Storage Team) 23

24 Configuration Author: Esteban Bernal (Storage Team) 24

25 Oracle Linux ASMLib Installation 3. Install the proper packages for your platform. For example, if you are using Red Hat Enterprise Linux AS 3.0 enterprise kernel, enter a command similar to the following: # rpm -i oracleasm-support i386.rpm \ oracleasm e-enterprise i386.rpm \ oracleasmlib i386.rpm 4. Enter the following command to run the oracleasm initialization script with the configure option: # /etc/init.d/oracleasm configure Author: Esteban Bernal (Storage Team) 25

26 Oracle Linux ASMLib Installation You will be prompted for the following: The UID of the driver owner. This will be the UID for the Oracle user. The GID of the driver group. This will be the GID for the dba group. Whether the driver should be loaded at startup. The corrects answer is YES. Author: Esteban Bernal (Storage Team) 26

27 Oracle Linux ASMLib Installation The script then completes the following tasks: Creates the /etc/sysconfig/oracleasm configuration file. Creates the /dev/oracleasm mount point. Loads the oracleasm kernel module. Mounts the ASM library driver file system. 5. On RAC environment, you will need to repeat this procedure on all cluster nodes. Author: Esteban Bernal (Storage Team) 27

28 ASMLib Installation for Enterprise Linux via ULN The script then completes the following tasks: 1) Installing via ULN # up2date -i oracleasm-support oracleasmlib oracleasm-`uname -r` This will install the support tools, the library, and the kernel driver for the currently running kernel. 2) Upgrading ASMLib via ULN # up2date -i kernel-smp el oracleasm elsmp Author: Esteban Bernal (Storage Team) 28

29 ASM Library Disk Creation To configure the disk devices that you want to use in an ASM diskgroup, complete the following steps: 1. If necessary, install the shared disks that you intend to use for the diskgroup and restart the system. 2. To identify the device name for the disks that you want to use, enter the following command: #/sbin/fdisk -l Author: Esteban Bernal (Storage Team) 29

30 ASM Library Disk Creation Author: Esteban Bernal (Storage Team) 30

31 ASM Library Disk Creation 3. Using fdisk, create a single whole-disk partition on the device that you want to use. 4. Mark a disk as an ASM disk: # /etc/init.d/oracleasm createdisk DISK1 /dev/hda7 In this example, DISK1 is the tag or name that you want to assign to the disk. 5. On RAC, to make the disk available on other cluster nodes, enter the following command as root on each node: # /etc/init.d/oracleasm scandisks This command identifies all the shared disks attached to the node that are marked as ASM disks. Author: Esteban Bernal (Storage Team) 31

32 ASM Library Disk Configuration Important oracleasm options: configure: Use this option to reconfigure the ASM library driver, if necessary. enable / disable: Use the disable and enable options to charge the behavior of the ASM library driver when the system starts. The enable option causes the ASM library driver load when the system starts. start / stop / restart: Use the start, stop and restart options to load or unload the ASM library driver without restarting the system. createdisk: Use this option to mark a disk for use with the ASM library and name it. deletedisk: Use the deletedisk option to unmark a named disk device. Do not use this command to unmark disks that are being used by an ASM diskgroup. You must drop the disk from the ASM diskgroup before you unmark it. Author: Esteban Bernal (Storage Team) 32

33 ASM Library Disk Configuration*** querydisk: Use this option to determinate whether a disk device or disk name is being used by the ASM library driver. listdisks: Use this option to list the disk names of marked ASM library driver disks. scandisks: Use the scandisks option to enable cluster nodes to identify which shared disks have been marked as ASM library driver disks on another node. force-renamedisk: Used to rename the disk header name, to create a pseudo disk /dev/oracleasm/disks or migrating raw to. After you have prepared your disks, set the ASM_DISKSTRING initialization parameter to an appropriated value. The oracleasm script marks disks with an ASM header label. You can set the ASM_DISKSTRING parameter to the value ORCL:*. This setting enables ASM to scan and qualify all disks with that header label. Author: Esteban Bernal (Storage Team) 33

34 Configuration (Example Installation) Author: Esteban Bernal (Storage Team) 34

35 Configuration Author: Esteban Bernal (Storage Team) 35

36 Configuration Author: Esteban Bernal (Storage Team) 36

37 Configuration Author: Esteban Bernal (Storage Team) 37

38 Configuration Author: Esteban Bernal (Storage Team) 38

39 Configuration Author: Esteban Bernal (Storage Team) 39

40 Upgrade i. Shutdown the ASM instance(s). ii. iii. iv. Execute /etc/init.d/oracleasm stop. Execute rpm -Uvh oracleasm-*.rpm *.rpm. Execute /etc/init.d/oracleasm start. v. Startup the ASM instance(s). Author: Esteban Bernal (Storage Team) 40

41 ASMLib Additional Information. Migrating Raw Devices to ASMLib Configuring on Multipath Disks Device Persistence and Oracle Linux ASMLib Source Code (Oracle ASMLib 2.0 Kernel Driver Source (GPL) ) Author: Esteban Bernal (Storage Team) 41

42 Migrating Raw Devices to ASMLib 1) Disk should not be mapped thru raw devices, ASM cannot see the disk twice. # raw /dev/raw/raw # raw /dev/raw/raw ) Remove it from /etc/sysconfig/rawdevices 3) Set: asm_diskstring='orcl:* instead of: asm_diskstring= /dev/raw/* 4) Shutdown the ASM instance (Very important!). 5) Label the disk thru the oracleasm command: # /etc/init.d/oracleasm force-renamedisk /dev/hda18 DISK1 # /etc/init.d/oracleasm force-renamedisk /dev/hda22 DISK2 Author: Esteban Bernal (Storage Team) 42

43 Configuring ASMLib on Multipath Disks Edit the /etc/sysconfig/oracleasm file to define the scanorder: # ORACLEASM_ENABELED: 'true' means to load the driver on boot. ORACLEASM_ENABLED=true # ORACLEASM_UID: Default user owning the /dev/oracleasm mount point. ORACLEASM_UID=oracle # ORACLEASM_GID: Default group owning the /dev/oracleasm mount point. ORACLEASM_GID=dba # ORACLEASM_SCANBOOT: 'true' means fix disk perms on boot ORACLEASM_SCANBOOT=true # ORACLEASM_CLEARBOOT: 'true' means clean old disk perms on boot ORACLEASM_CLEARBOOT=true # ORACLEASM_SCANORDER: Matching patterns to order disk scanning ORACLEASM_SCANORDER="emcpower sd" #emcpowera emcpowerb, scsi #Or ORACLEASM_SCANORDER="emcpower hda" #emcpowera emcpowerb, ATA # ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan ORACLEASM_SCANEXCLUDE="sd raw" Author: Esteban Bernal (Storage Team) 43

44 Device Persistence guaranteed thru 1) Devices name has changed during reboot (e.g. /dev/sdg to /dev/sdh) will discover the disk with no problems. 2) Oracle (ASM) does not need OS permissions to access the real disks (e.g. /dev/sda1). 3) The disks must belong to root (Security reasons). 4) Using guarantees device persistence for ASM and the device still belongs to Oracle. Author: Esteban Bernal (Storage Team) 44

45 Source Code (Oracle ASMLib 2.0 Kernel Driver Source) Access the source code repository for this project in one of the following ways: Browse source code on-line to view this project's directory structure and files. Download source code archives, if any are available, to copy read-only source files for this project on to your local system. Subscribe to any of the project mailing lists. Author: Esteban Bernal (Storage Team) 45

46 Troubleshooting # df -ha # lsmod # cat /proc/filesystems # /etc/init.d/oracleasm configure $> cat /etc/*release $> uname -a # rpm -qa grep oracleasm $> ls -l /dev/oracleasm/disks SQL> alter system set asm_diskstring = ORCL:* scope=both; $> /usr/sbin/oracleasm-discover $> kfod asm_diskstring='orcl:*' disks=all $> kfed read /dev/oracleasm/disks/vol1 Author: Esteban Bernal (Storage Team) 46

47 Troubleshooting Author: Esteban Bernal (Storage Team) 47

48 Troubleshooting Author: Esteban Bernal (Storage Team) 48

49 Troubleshooting Author: Esteban Bernal (Storage Team) 49

50 Troubleshooting Author: Esteban Bernal (Storage Team) 50

51 Troubleshooting Author: Esteban Bernal (Storage Team) 51

52 Troubleshooting Author: Esteban Bernal (Storage Team) 52

53 Troubleshooting Author: Esteban Bernal (Storage Team) 53

54 Troubleshooting Author: Esteban Bernal (Storage Team) 54

55 Troubleshooting Author: Esteban Bernal (Storage Team) 55

56 References =)> =)> =)> Note Installing & Configuring Oracle ASMLib =)> Note Configuring Oracle ASMLib on Multipath Disks =)> Note Device Persistence and Oracle Linux ASMLib =)> Note Migrating Raw Devices to ASMLib on Linux =)> Note Troubleshooting on Linux =)> Note How to know Which Disks Are Handled by ASMLib Kernel Driver? Author: Esteban Bernal (Storage Team) 56

57 References =)> Note How To Install API via the Unbreakable Linux Network =)> =)> Author: Esteban Bernal (Storage Team) 57