1 Revisions. Storage Layout, DB, and OS performance tuning guideline for SAP - V4.4. IBM System Storage layout & performance guideline for SAP

Transcription

1 1 Revisions Storage Layout, DB, and OS performance tuning guideline for SAP - V4.4 Location of this document: This document has been optimized for screen reading. Press CTRL l on Acrobat Reader V1 First version V2 Separated from sizing guideline V2.1 Added References, precise XIV recommendations V2.2 Changed documentation classification V3 Added DS8700, several updates V3.1 Added Storwize V V3.2 Updated Section file system layout V3.3 Updated V7000 & SVC section. Changed file system layout section V3.4 Added XIV Gen3, DS V4 Added VMware, changed layout V4.1 Modified VMware layout, V7000 setup V4.2 Removed DS3/4/5000 systems, add new Storwize & FlashSystem V4.3 Added settings for DB2 & FlashSystem Document Owner: Markus Fehling Walldorf, Germany, April 2015 Page Copyright IBM Corporation, 2015

2 2 Table of Contents: 1 REVISIONS 1 2 TABLE OF CONTENTS: 2 3 ABSTRACT 2 4 AUDIENCE 2 5 INTRODUCTION TO DATA BASE AND STORAGE LAYOUT 3 6 DATA DISTRIBUTION MATRIX 6 7 RECOMMENDED CONFIGURATIONS 7 8 LAYOUT RECOMMENDATIONS 11 9 SUMMARY CONTACT INFORMATION DOCUMENT REFERENCES TRADEMARKS AND DISCLAIMER 16 3 Abstract This white paper provides guides how to layout IBM storage systems for SAP applications; this paper coves best practices on how to setup data base and file system layout to be used by SAP applications. This document includes recommendation for IBM XIV System, IBM System Storage SVC and IBM Storwize family, DS8000, FlashSystem, IBM DB2, IBM AIX, LINUX, and Microsoft Windows. 4 Audience This document has been written for technical communities both within IBM and externally including storage and database specialists, systems and solution architects. This document aims to provide high level guidance and assumes that the reader already has experience with storage systems, and simply needs additional knowledge about SAP specific performance and storage sizing requirements. This document covers the OPEN environment, like UNIX and Windows. Page Copyright IBM Corporation, 2015

3 5 Introduction to Data Base and Storage layout 5.1 DB2 data distribution This document describes how to store SAP data on the storage system, how to design a layout for a data base, how to define file systems, volumes, LUNs, groups, RAID arrays etc. depending on the type of data base, the OS, and storage system. In most cases SAP data bases can be classified as an OLTP workload, but a SAP Business Warehouse generates an OLAP workload (see Sizing Guideline). In order to achieve the highest performance (with a given storage system) we recommend spreading all data equally across all available resources (disks, host adapters, device adapters etc) which means spreading all SAP applications, like ERP, HR, SCM, CRM,... across the same resources No dedicated disks for dedicated applications - always Share Everything - this technique avoids bottlenecks in case an application experiences a peak workload. However, data safety and data integrity may be considered more important than performance, so typically DATA and LOG files will be stored on different, physically separated disks - if allowed by the storage system. Perform Data Distribution on 3 different levels: 1. Data Base 2. Logical Volume Manager of the Operating System 3. Storage System or Storage Virtualization Layer Figure 1: DB2 data distribution Figure 1 shows the data distribution technology for DB2: The data are stored in tables; one table is physically written to one or more files, called containers. The table is striped across the containers, the STIPESIZE is typically 2 times the block size. For SAP the block size is 16KB; therefore the stripe size is 32 KB. LOG data are written into separate files. Consider all three layers to configure systems appropriately. Page Copyright IBM Corporation, 2015

4 Logical Volume Manager data distribution: 2. Maximum distribution: LVM locates the first block of a LV on the first LUN, the second block on the second LUN, and so on. Also called Round-Robin 3. Preselected LUNs: Creates a LV on a subset of LUNs of the VG Optionally selects a striping mode. Depending on the storage system used for SAP, create Logical Volume using maximum distribution or linear options; DO NOT use Logical Volume striping and File System striping. On AIX this setup allows the redistribution of the file system if additional hdisks (LUNs) are added to the Volume Group AIX command reorgvg. Figure 2: LVM data distribution Figure 2 shows the data distribution performed by LVM, e.g. AIX in principle this method applies to any LVM. LUNs (called hdisks with AIX) are grouped together into a Volume Group. Out of the volume group create one or more Logical Volumes, and create a file system within each LV. Several options are available to allocate storage (space) for the file system: 1. Linear: Fills up the first LUN first, then takes space from the next LUN until full, and so on. Page Copyright IBM Corporation, 2015

5 5.2 Storage System data distribution Within (IBM) Storage System data distribution or striping is implemented in several ways. Striping across a Storage Pool in place update: Several RAID arrays are grouped into a Storage Pool, then out of the Storage Pool one or more Logical Volumes (LUNs) are created. These LVs are striped across all disks within the Storage Pool. If data are overwritten then the physical location of the block will persist; this is called an in place update. This is the behavior for DS8000, V7000, and SVC. If data are updated (overwritten) by the server then this data are always written to a new, physical location; this mechanism is called redirect on write. This is the behavior of XIV. LUN 1 LUN 2 Extents Figure 4: Volume (LUN) definition on IBM XIV RANK Logical Volume 1 Logical Volume 2 in-placement update never overwrite technology C O N T E N T C O N T E N T Extent Pool C O N T E N T N E C O N T E N T Figure 3: Logical Volume (LUN) definition on DS8000 Striping across a Storage Pool redirect on write: Several arrays or disks are grouped into a Storage Pool, and then out of this Storage Pool one or more Logical Volumes (LUNs) are created. These LVs are striped across all disks within the Storage Pool.. Figure 5: In-place update vs. redirect on write For further details please read the document: IBM Storage Virtualization Guide for SAP landscapes, Page Copyright IBM Corporation, 2015

6 6 Data distribution Matrix The table b elow shows which data distribution technique to focus on, depending on the DB, OS and Storage System combination. AIX / LINUX / UNIX Storage System DB2 *) Oracle MS Server MS SQL VMware with LINUX or MS SQL FlashSystem 840/900 A A E F DS8000 B & C B & C F & C G & F or G XIV B B E F SVC / Storwize / V9000 B & D B & D E & D D & F or G Table 1: Data Distribution Matrix Data distribution method DB OS / LVM Storage System SVC Legend: *) DB2 stripes always (across containers) A) Set LVM physical partition allocation, small block size B) Set LVM physical partition allocation, large block size C) Use DS8000 Extent Pool striping D) Use SVC VDISK striping E) Use MS Dynamic disks without Striped Volumes F) Use VMware VMFS G) Use VMware vrdm Page Copyright IBM Corporation, 2015

7 7 Recommended configurations All numbers provided are ballpark numbers to give an order of magnitude. E.g. if stated 20 to 40 it could turn out 15 or 60 for a specific configuration, but it is unlikely the number will turn out to be 2 or Storage System Always use the same type for any configuration for the entire environment being used, like (if applicable): Disk Type (use just one type, either FC, SAS, ) Capacity (600GB, 900, 1,2 TB, ) Speed (15,000/10,000, rpm) RAID (6+1+1, 7+1, RAID 10, ) LUN size (storage logical-volume) Separate DATA and LOG on different physical disks, not because of performance improvements (there will be seldom one), but because of data safety in case a RAID array fails. Separating DATA and LOG need not to be considered if one (or more) of these items has been implemented: RAID 6 LVM mirror SVC stretched cluster with VDisk mirroring FlashSystem Because XIV uses a grid architecture in contrast to DS RAID controller architecture it is neither possible nor desirable to separate DATA from LOG Easy Tier The DS8700, DS8800, V7000 Easy Tier function should not be combined with SVC Easy Tier. Use Easy Tier for DATA only SVC Create one MD Group (MDG) for DATA, and one MDG for LOG, if sufficient number of storage systems are available. Match the extent size according to the stripe size of the storage system, or a multiplier of it (2,4,8,...) With DS8000 it should match the DS8000 extent size, 1GB. Include as many SAP applications as possible into these two MDG or alternatively create additional MDG sets However minimize the number of MDG if possible. Create multiple VDisks for DATA and LOG; the size is not critical, and can fall in the range between 100GB and 1TB (ball park number). Distribute VDisks equally across all SVC nodes. Use SVC Easy Tier, not the Easy Tier function of the used storage system. Use the SSD of the used storage systems, the SSD within SVC must be configured as RAID 1 (too high capacity penalty), and the total usable capacity is usually too small, SAPmagic. There are no best practices documented with respect to performance in relation to how many VDISKs per IO Group are used, but use at least 2 (one VDISK for each node within IO group). Typically one OS needs a handful LUNs in order to create multiple IO queues: Create around 4 to 8 VDISKs (hdisk) for each SAP instance; on very large DB servers (serving 50,000 SAPS or Page Copyright IBM Corporation, 2015

8 more), more VDISKs could be required ( ); for a small Dev/Test system one or two are sufficient DS8000 Use RAID 5 or RAID 6. Share everything approach: o Create one Extent Pool for each controller (Server 0 & Server1) o Include multiple RANKs within Extent Pools. o Create striped LVs (using the rotate extents mechanism) within Extent Pools, all the same size within one Extent Pool. o DATA and LOG share the same storage Storwize family Use RAID 5 or RAID 6, with a RAID size of 7+1 / / 10+2 / Raid 10 for high workload use SAPmagic Create separate MDG for DATA and LOG Create multiple VDisks for DATA, one VDisk for LOG; the size is not critical, and can be in the range between 100GB and 1TB Create 4 to 8 VDISKs (LUNs) for each SAP instance, or just one or two for a Dev/Test system XIV (Gen 3) XIV uses a unique storage virtualization design making administration much easier: Always use the latest version of the Host connection kit then in principle just one LUN needs to be created to gain maximum IO performance with the MP device driver, however: For DB DATA: Use only a small number of large LUNs, typically 4 to 8 LUNs. Please keep in mind that XIV always uses thin provisioning, so we recommend creating larger LUNs than needed if the OS volume manager fully supports thin provisioned volumes, e.g. AIX, HP UX, SUN Solaris, or Veritas. For DB LOG: Match the LVs size according to the LOG space requirement for the DB, just one LUN. FlashSystems, no special setup for DB or LOG required. 7.2 OS definitions AIX / LINUX / UNIX Create three Volume Groups for SAP in addition to root VG: BIN, DATA, LOG; this is a requirement for IBM FlashCopy Manager and SAP ACC / SAP LVM. o Assign Storage Logical Volumes (LUNs) accordingly (AIX hdisks) o Do not use file-system or logical-volume striping! o Depending on recommendations in section: 6 Data distribution Matrix set the LVM inter-physical volume allocation flag to max : This flag enables the distribution of LVM blocks (physical partitions) across all LUNs (Disk) equally, for max performance, E.g. AIX: mklv... -e x For an explanation on setting the volume allocation Page Copyright IBM Corporation, 2015

9 policy refer to the Operating System guide, section Volume Manager (e.g. AIX LVM admin guide). o Match block size with Extent size ( MB) Create several file systems for DATA to enable parallel file system checks, especially if FlashCopy is used. For AIX use JFS2 with inline log. For LOGs create just one File system JFS2 with inline log. For Oracle create a separate file system for archive LOGs. The file-system block-size should match the DB block-size, with JFS2 a max block size of 4KB can be specified (512B to 4KB) Pay attention to the way the DB writes to the LOG file: The block-size of the LOG files may be different than the DATA block-size. If the block sizes are different, create file-systems within the OS with a block size for the LOG writes, e.g. Oracle uses 512 Bytes block size for LOGs. (For further details please refer to the Data Base administration guide) For DATA: Use Direct (DIO) & Concurrent (CIO) IO options for file system mounts (DIO: Bypassing File System and LVM stack) (CIO: File System does allow concurrent read & write) For further explanation of DIO and CIO please refer to the Operating System guides, section Volume Manager (e.g. AIX LVM admin guide) DB2: Use DIO & CIO for LOG as well. Oracle: Only use CIO for LOG and archive LOG. NFS recommendations for NAS: o Use mount option DIO & CIO o Enable tuning of IP buffer: no -o rfc1323=1 nfso -o nfs_rfc1323=1 o Use buffer size: rsize=32768,wsize=32768 MS Windows Server for MS SQL Format the file system using NTFS and an allocation unit of 64KB, equal to the sector size. Use the quick format for storage systems providing Thin Provisioning (XIV, SVC, Storwize) Configure the storage subsystem segment size at 64KB, equal to the MS SQL Server extent size. Configure the system to make sure that segment size = sector size = cluster size, also known as allocation unit. Sector-align logical drives to 64 KB, to equal the segment size of the hardware storage subsystem. For SVC, DS8000, V7000, XIV: Use Dynamic Disks, use Standard Volumes. For more information please refer to the White Paper: Best Practices for Installing and Configuring SQL Server, Section: Document References Page Copyright IBM Corporation, 2015

10 VMware VMware does not have a LVM, and hence VMware is not able to stripe VMFS (VMware File System) across multiple LUNs; VMware is only able to extend the size of a VMFS to a next storage volume (LUN), and filling up the first LUN, then the second, the next, and so on. Therefore the data must be striped by the storage system or the data base. In principle the same recommendations as for LINUX or MS server apply. o EXTENTSIZE With LVM or Storage based striping the EXTENTSIZE cannot be mapped anymore to the real physical RAID stripe size of the Storage system, and should just have a large number (32 to 256) o PREFETCHSIZE = Automatic o NUM_IOSERVERS = Automatic o DB2_PARALLEL_IO = * o Block size is set to 16KB o Ext. size is set to 2 32KB For many, small SAP instances we recommend using VMDK with VMFS. For single, larger production SAP instances raw device mapping (RDM) can be used instead. See section Layout recommendations, VMware. 7.3 DB definitions Do not use database storage performance management tools, as these capabilities are counterproductive to the virtualization features of the Storage system, especially with SVC, XIV, or V7000. DB2 o Use DMS AUTOMATIC STORAGE This is the recommended setting by DB2 for SAP. This implies striping using containers always. o See applicable SAP Notes for DB2 o Place log files on LOG file system / Volume Group Settings for FlashSystem: OVERHEAD = 0.3 TRANSFERRATE = 0.1 Oracle Page Copyright IBM Corporation, 2015 Set the DB_BLOCK_SIZE according to Oracle s administration guide, for SAP typically 8, 16, or 32 KB. Match the File system block size accordingly (if allowed by the file system). Set DISK_ASYNC_IO Unlike DB2, Oracle does not stripe its data (tables) across the used storage (files or LUNs). For more information on how to tune Oracle with respect to storage refer to the White Paper: Oracle Architecture and Tuning on AIX

11 8 Layout recommendations 8.2 Logical Volume Manager (LVM) setup This section illustrates the layout technique described in section File System layout recommendation for DB2 The following figure 7 describes how to layout the file-systems for SAP on AIX/ LINUX/ UNIX using DB2. The colored boxes indicate a file system mount point and the underlying Volume Group, and white boxes indicate a directory. Figure 7: UNIX LVM Figure 6: File System layout for DB2 on AIX / LINUX Page Copyright IBM Corporation, 2015

12 8.3 MS Windows Setup for XIV/V7000 Use this setup for storage system able to stripe data: DS8000, V7000, XIV, SVC IBM System Storage layout & performance guideline for SAP Figure 8: MS Server setup for XIV/V7000/SVC/DS VMware setup for higher performance This figure shows a volume mapping for high performance production workload. For a single SAP instance, just use one volume for LOG; depending on the workload use one to many (4) volumes for DATA. On XIV it is sufficient to just use one volume. Depending on the used storage system it could be reasonable to create different storage pools / RAID groups for EXE, DATA, and LOG volumes. Figure 9: VMware layout for higher performance 8.5 VMware setup for standard workload This figure shows a volume mapping based on VMFS, most convenient to manage in respect to VMware functionality. This applies to production or QA systems, for QA system a secondary environment should be installed, with again 3 different VMFS and an additional storage pool. Depending on the workload and used storage system it could be reasonable to create a dedicate storage pool / RAID group for LOG (and DATA). Page Copyright IBM Corporation, 2015

13 8.6 VMware setup for DEV/Test systems Usually the performance requirements of Dev or Test systems are low, therefore the volume mapping can be simplified like this: Figure 10: VMware layout for use with VMFS, prod environment Figure 11: VMware layout for Dev / Test systems Page Copyright IBM Corporation, 2015

14 8.7 DS8000 setup 8.9 XIV setup XIV does not need a storage layout, XIV places its volumes automatically; just define the LUNs, that s all! 8.10 Data Model from Data Base through SVC to RAID definition Figure 12: Storage Layout for DS Storwize V7000 / SVC Figure 14: Data Model, from Database to Storage System Figure 13: Storage Layout for V7000 / SVC Page Copyright IBM Corporation, 2015

15 9 Summary The major challenge during data base layout creation is evenly distributing database tables (data) across the available physical disks. As mentioned at the beginning of the document, the data distribution can be done on three levels: DB, LVM and Storage. In addition an even distribution of the workload must be achieved for all HBA as well as the storage (RAID) controllers preferred path, LUN to controller association. 11 Document References IBM Redbooks sg245432: AIX Logical Volume Manager from A to Z: Introduction and Concepts sg247146: sg247521: sg247653: DS8000 Performance Monitoring and Tuning SAN Volume Controller Best Practices and Performance Guidelines DB2 Deployment Guide Also over striping needs to be avoided, e.g. if the PP block size of the LVM is the same as the SVC extent size then this may be counterproductive in that the first data written for any file system could be placed on the same RAID array and hence the same physical disks. 10 Contact information If you have any (sizing or configuration) questions regarding SAP please send an to: ISICC Info Service mailto:isicc@de.ibm.com You can also visit this page, which provides additional information on SW & HW infrastructures, sales kits, presentation, more sizing guidelines, Wiki, Customer references, FAQ, Industry specific solutions and more, IBM Whitepapers WP101306: Logical volume layout for Microsoft SQL Server 2008 databases and IBM Storage Systems WP101601: Tuning SAP with DB2 on IBM AIX WP100883: Oracle Architecture and Tuning on AIX WP101452: Best Practices Guide for SAP Applications on an IBM XIV Storage System WP101136: Logical Configuration Concepts for IBM System Storage DS8000 Series in Large Data Centers WP101362: SQL Server 2005 with IBM System Storage DS8000, DS5000, and DS4000 Best Practices Guide WP101491: SQL Server 2008 with IBM System Storage DS8000, DS5000, and DS4000 Best Practices Guide Page Copyright IBM Corporation, 2015

16 Microsoft: Physical Database Storage Design 12 Trademarks and Disclaimer Copyright IBM Corporation All rights Reserved. References in this document to IBM products or services do not imply that IBM intends to make them available in every country. IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. SET and the SET Logo are trademarks owned by SET Secure Electronic Transaction LLC. DIA Data Integrity Assurance, Storewiz, and the Storwize logo are trademarks or registered trademarks of Storwize, Inc., an IBM Company. Other company, product, or service names may be trademarks or service marks of others. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Information concerning non-ibm products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-ibm list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-ibm products. Questions on the capability of non-ibm products should be addressed to the supplier of those products. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller for the full text of the specific Statement of Direction. Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here. Photographs shown are of engineering prototypes. Changes may be incorporated in production models. Any references in this information to non-ibm websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk. Page Copyright IBM Corporation, 2015