IBM Power Systems 57D7/57B4 Adapter Write Cache 1 of 15
Contents 1 Feature Description 3 2 Software and Hardware Requirements 4 2.0.1 Supported Systems...................................... 4 2.0.2 Supported Storage Adapter Backplanes.......................... 4 2.0.3 Supported Linux Operating System Releases & Configurations............. 4 2.0.4 Minimum SAS RAID Adapter Firmware Level...................... 5 2.0.5 Minimum Linux iprutils Package Level........................... 5 3 Installation Instructions 6 3.1 Linux Software Installation Instructions.............................. 6 3.1.1 YUM Repository....................................... 6 3.1.1.1 Red Hat...................................... 6 3.1.1.2 SUSE........................................ 6 3.1.2 Manual Installation..................................... 7 3.2 Adapter Firmware Installation Instructions............................ 8 4 Feature Enablement 12 4.1 Curses Interface............................................ 12 4.2 Command Line Interface....................................... 14 5 Support 15 2 of 15
Chapter 1 Feature Description This document describes how to enable and use the new volatile adapter write cache feature for the integrated SAS controllers in the #EL3T and #EL3V SAS backplanes on IBM Power 8247-21L and 8247-22L systems, the integrated SAS controllers in the #EJ0N and #EJ0S SAS backplanes on the IBM Power 8247-42L, and the PCIe3 x8 SAS RAID adapter low profile Quad Port 6Gb (#EJ0J and #EJ0M). Once the required adapter firmware and Linux software prerequisites are installed, this feature is enabled by default, but can be manually disabled on a per RAID array basis. This feature enables a volatile write cache in the SAS RAID adapter into which data is written. This data is then later flushed to the underlying disks. This allows the RAID adapter to combine multiple host writes into fewer writes to the disks. It also enables the RAID adapter to issue stripe writes to RAID 5 or 6 arrays, when a full parity stripe of data is available in the cache, which can eliminate read-xor write overhead that may occur when a write cache is not available. This can provide a noticeable performance advantage for some write workloads. The adapter firmware which implements this volatile write caching feature includes support for a SYNCHRONIZE CACHE mechanism. This allows journaled filesystems in the Linux operating system to flush any previously written data prior to updating the filesystem journal. This ensures the filesystem data stays coherent with respect to the journal in the case of an unexpected power loss of the system. For use cases which access the RAID array block devices directly and bypass any filesystems in Linux, it may be desireable to disable this feature, particularly if the software has not been written to issue SYNCHRONIZE CACHE commands. Instructions on how to disable this feature are described later in this document. 3 of 15
Chapter 2 Software and Hardware Requirements The required hardware, firmware, and software levels for this feature are described below. 2.0.1 Supported Systems IBM Power 8247-21L IBM Power 8247-22L IBM Power 8247-42L 2.0.2 Supported Storage Adapter Backplanes #EL3T: PCIe3 x8 SAS RAID internal adapter 6Gb #EL3V: PCIe3 x8 SAS RAID internal adapter 6Gb (Split Backplane) #EJ0N: PCIe3 x8 SAS RAID internal adapter 6Gb (4U) #EJ0S: PCIe3 x8 SAS RAID internal adapter 6Gb (4U) (Split Backplane) #EJ0J: PCIe3 x8 SAS RAID adapter Quad Port 6Gb 1 #EJ0M: PCIe3 x8 SAS RAID adapter low profile Quad Port 6Gb 1 2.0.3 Supported Linux Operating System Releases & Configurations Bare Metal Systems (indicated by #EC16): Ubuntu Server 16.04, or later PowerKVM 3.1 Systems (indicated by #EC40) or PowerKVM 2.1 (#EC20) Not supported Other environments, such as PowerVM (not PowerKVM, not Bare Metal) Red Hat Enterprise Linux 6.5, or later Red Hat Enterprise Linux 7.2, little endian, or later Red Hat Enterprise Linux 7.2, big endian, or later 1 Single adapter configurations only, feature not supported for dual adapter, HA configurations 4 of 15
SUSE Linux Enterprise Server 11, Service Pack 4, or later SUSE Linux Enterprise Server 12, Service Pack 1, or later Ubuntu Server 16.04, or later 2.0.4 Minimum SAS RAID Adapter Firmware Level 14511900 2.0.5 Minimum Linux iprutils Package Level 2.4.11 5 of 15
Chapter 3 Installation Instructions 3.1 Linux Software Installation Instructions 3.1.1 YUM Repository Installing via the IBM Power Tools YUM repository is the preferred method for obtaining the required 2.4.11 version of the iprutils package. Use this method if your system has connectivity to the external network and is able to download files direct from the internet. If you are using Ubuntu 16.04, the requisite version of iprutils should be able to be installed using apt-get, like any other package that is part of the Ubuntu distribution. In this case, you can skip ahead to section 3.2 on updating the adapter firmware. First, configure the IBM Power Tools YUM repository. Go to the following page for instructions on how to download the configuration RPM file and how to configure the repository: http://www14.software.ibm. com/webapp/set2/sas/f/lopdiags/yum.html. Once the repository is configured, upgrade the iprutils package to the 2.4.11 version from the newly configured repository. During this process, you will be prompted to accept the GPG key for the newly added YUM repository. Please allow this new GPG key to be accepted, otherwise the package installation will be aborted. 3.1.1.1 Red Hat # yum update iprutils 3.1.1.2 SUSE # zypper update iprutils The currently installed version of iprutils can be verified with the following command. # iprconfig --version iprconfig: 2.4.11.1 6 of 15
Ensure the iprutils daemons are enabled to start on the next boot. Please note, that the iprinit daemon must be enabled and running for this write cache feature to get enabled at boot time. # chkconfig iprinit on # chkconfig iprupdate on # chkconfig iprdump on 3.1.2 Manual Installation Use this method if either the YUM repository method fails or your system is unable to download files directly from the internet. Download the iprutils package directly from one of the following locations, depending on your OS level, and transfer to the server. RHEL 6.5, or later http://public.dhe.ibm.com/software/server/power/linux/yum/oss/rhel/6/ppc64/iprutils-2.4.11-1. 41.1.ppc64.rpm RHEL 7.2 Little Endian http://public.dhe.ibm.com/software/server/power/linux/yum/oss/rhel/7/ppc64le/iprutils-2.4. 11-1.1.ppc64le.rpm RHEL 7.2 Big Endian http://public.dhe.ibm.com/software/server/power/linux/yum/oss/rhel/7/ppc64/iprutils-2.4.11-1. 41.1.ppc64.rpm SLES 11 SP4 http://public.dhe.ibm.com/software/server/power/linux/yum/oss/sles/11/ppc64/iprutils-2.4.11-1. 41.1.ppc64.rpm SLES 12 SP1 http://public.dhe.ibm.com/software/server/power/linux/yum/oss/sles/12/ppc64le/iprutils-2.4. 11-1.41.1.ppc64le.rpm Next, remove the old iprutils package and then install the 2.4.11 version by running the following commands on the server to which you transferred the iprutils rpm package. # rpm -e iprutils # rpm -ivh iprutils-2.4.11-1.41.1.ppc64.rpm The currently installed version of iprutils can be verified with the following command. # iprconfig --version iprconfig: 2.4.11.1 7 of 15
Ensure the iprutils daemons are enabled to start on the next boot. Please note, that the iprinit daemon must be enabled and running for this write cache feature to get enabled at boot time. # chkconfig iprinit on # chkconfig iprupdate on # chkconfig iprdump on 3.2 Adapter Firmware Installation Instructions The required 14511900 version of SAS adapter firmware to enable this feature can be downloaded from IBM Fix Central at the following URL: http://www.ibm.com/support/fixcentral/main/quickorder?parent= Power%2BIO%2BFirmware&product=ibm/io/featurecode/EJ0T&release=all&platform=all&function=all&source= fc. The pci.1014034a.51-14511900-1.linux.noarch.rpm file should be downloaded. Once downloaded, install to your Linux system with the rpm command. # rpm -ivh pci.1014034a.51-14511900-1.linux.noarch.rpm To begin the process of installing the new adapter firmware, first login as root on the target server, then run iprconfig. You should see a screen similar to this: IBM Power RAID Configuration Utility Select one of the following: 1. Display hardware status 2. Work with disk arrays 3. Work with disk unit recovery 4. Work with configuration options 5. Work with microcode updates 6. Devices Statistics 7. Analyze log Selection: e=exit Select option 5. Work with microcode updates, and press Enter. 8 of 15
Select option 2. Download latest microcode to all devices, and press Enter. Work with microcode updates Select one of the following: 1. Download microcode 2. Download latest microcode to all devices Selection: e=exit q=cancel You should then see a screen similar to the following screen, showing one or more adapters to be updated with the pci.1014034a.51.14511900 firmware image. Confirm Microcode Download The following devices are going to be updated. ATTENTION: System performance may be affected during the microcode download process. Download may take some time. Press Enter to continue. q=cancel to return and change your choice. Name Version Date Image File ------ --------- --------- ----------------------------- sg6 14511900 05/12/16 /lib/firmware/pci.1014034a.51.14511900 sg7 14511900 05/12/16 /lib/firmware/pci.1014034a.51.14511900 q=cancel Press Enter and follow the prompts. Once the operation is completed, return to the main menu by pressing e if necessary. 9 of 15
The current version of the adapter firmware can be confirmed by selecting option 1. status. This should present a screen similar to this: Display hardware Display Hardware Status Type option, press Enter. 1=Display hardware resource information details OPT Name PCI/SCSI Location Description Status --- ------ ------------------------- ---------------------------- ------------ 0003:70:00.0/0: PCI-E SAS RAID Adapter Operational sda 0003:70:00.0/0:2:0:0 RAID 0 Array Optimized 0003:70:00.0/0:0:0:0 RAID 0 Array Member Active sdb 0003:70:00.0/0:2:1:0 RAID 10 Array Optimized 0003:70:00.0/0:0:1:0 RAID 10 Array Member Active 0003:70:00.0/0:0:2:0 RAID 10 Array Member Active sdc 0003:70:00.0/0:2:2:0 RAID 5 Array Optimized 0003:70:00.0/0:0:3:0 RAID 5 Array Member Active 0003:70:00.0/0:0:4:0 RAID 5 Array Member Active 0003:70:00.0/0:0:5:0 RAID 5 Array Member Active More... e=exit q=cancel r=refresh t=toggle f=pagedn b=pageup Using the arrow keys, place the cursor next to the PCI-E SAS RAID Adapter, type 1, and press Enter. 10 of 15
This should display a screen similar to the screen below. IOA Hardware Resource Information Details Manufacturer.............. : IBM Machine Type and Model......... : 57D7001SISIOA Firmware Version............ : 14511900 Serial Number.............. : 00423088 Part Number............... : 0000000E2845 Plant of Manufacturer.......... : 00UE Write Cache Size............ : 256 MB DRAM Size................ : 512 MB Resource Name.............. : /dev/sg12 Physical location PCI Address............... : 0003:70:00.0 Resource Path.............. : FE SCSI Host Number............ : 0 Platform Location............ : U78CB.001.WZS00E2-P1-C14 Cache Protection............ : Synchronize Cache Press Enter to Continue e=exit q=cancel The Firmware Version field should show 14511900. You should now see a Cache Protection field showing a value of Synchronize Cache. Note, you may need to page down using f in order to see this value. 11 of 15
Chapter 4 Feature Enablement 4.1 Curses Interface By default, the write cache should now be set to Write Back, indicating the adapter volatile write cache is enabled. This can be confirmed by selecting option 4. Work with configuration options from the main menu, then pressing Enter. At this point you should see a screen like the following. IBM Power RAID Configuration Utility Select one of the following: 1. Work with SCSI bus configuration 2. Work with driver configuration 3. Work with disk configuration 4. Work with adapter configuration Selection: e=exit q=cancel Next, select option 3. Work with disk configuration from the menu and press Enter. 12 of 15
You should now see a screen similar to the following. Type option, press Enter. 1=Change Disk Configuration Change Disk Configuration OPT Name PCI/SCSI Location Description Status --- ------ ------------------------- ---------------------------- ------------- sda 0003:70:00.0/0:2:0:0 RAID 0 Array Optimized 0003:70:00.0/0:0:0:0 RAID 0 Array Member Active sdb 0003:70:00.0/0:2:1:0 RAID 10 Array Optimized 0003:70:00.0/0:0:1:0 RAID 10 Array Member Active 0003:70:00.0/0:0:2:0 RAID 10 Array Member Active sdc 0003:70:00.0/0:2:2:0 RAID 5 Array Optimized 0003:70:00.0/0:0:3:0 RAID 5 Array Member Active 0003:70:00.0/0:0:4:0 RAID 5 Array Member Active 0003:70:00.0/0:0:5:0 RAID 5 Array Member Active e=exit q=cancel r=refresh t=toggle Here, you will see all the physical disks as well as the logical RAID array devices as well. In this example, we will choose the first RAID array, sda. Using the arrow keys, navigate to the desired RAID array, select the device by entering 1, then press Enter. Change Configuration of Disk Current Disk configurations are shown. To change setting hit c for options menu. Highlight desired option then hit Enter. c=change Setting Queue Depth............... : 4 Device Write Cache Policy........ : Write Back Or leave blank and press Enter to cancel e=exit q=cancel Here we can see the Device Write Cache Policy is set to Write Back, as expected. If it is desired to disable the volatile adapter write cache for a specific RAID array, on this same screen, using the arrow keys, select the Device Write Cache Policy field, then type c to display the possible options. Then use the arrow keys to select the desired caching mode and press Enter. Press Enter again to confirm the change 13 of 15
4.2 Command Line Interface IBM Power Systems 57D7/57B4 Adapter Write Cache Alternatively, the iprconfig command line interface can be used to display and change the current caching mode. To list the current hardware RAID arrays current configured on the system use the following command: # iprconfig -c show-arrays Name PCI/SCSI Location Description Status ------ ------------------------- ------------------------- ----------------- sda 0003:70:00.0/0:2:0:0 RAID 0 Array Optimized sdb 0003:70:00.0/0:2:1:0 RAID 10 Array Optimized sdc 0003:70:00.0/0:2:2:0 RAID 5 Array Optimized To display the current caching mode for the first RAID array, we do the following: # iprconfig -c query-write-cache-policy sda write back Here we can see the write cache policy is set to write back, indicating the volatile adapter write cache is enabled for this array. To change this setting, you can do the following. # iprconfig -c set-write-cache-policy sda writethrough # iprconfig -c query-write-cache-policy sda write through Now let s change it back. # iprconfig -c set-write-cache-policy sda writeback # iprconfig -c query-write-cache-policy sda write back 14 of 15
Chapter 5 Support Problems this feature should be reported using the normal IBM Support process by opening a hardware support ticket. 15 of 15