SAP HANA on IBM Power Servers and IBM System Storage

Transcription

1 SAP HANA on IBM Power Servers and IBM System Storage Supplemental IBM Installation Guide to the SAP HANA Master, Server Installation and Administration Guides IBM Deutschland Research & Development, GmbH Created on 3 rd February 2015 Version 0.5 Last modified on December 3rd, 2015 Version 1.3

2 2 Note Before using this information and the product it supports, be sure to read the general information under Disclaimer and Special Notices and Copyrights and Trademarks. Edition Notice and Version Information Copyright IBM Corporation All Rights Reserved. US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. All trademarks or registered trademarks mentioned herein are the property of their respective holders. IBM Corporation Systems and Technology Group Route 100 Somers, NY Produced in the United States of America Edition Notice: This is version 1. 3 of this document. Focus: SAP HANA scale-up and initial scale-out solutions Target: SAP HANA on Power Deployments in 2015 Draft: Please look for regular updates!

3 3 Preface SAP HANA on POWER is a new solution allowing customers to run HANA-based analytics and business applications on an agile IBM Power based infrastructure. Existing IT assets - servers, storage, as well as skills and operations - can easily be (re-)used, instead of enforcing additional investment into dedicated SAP HANA only appliances. About This Document This document is intended for architects and specialists performing an SAP HANA on POWER deployment. It describes the implementation specifics for hardware, networking, and software components of the SAP HANA on POWER solution stack to complete the existing SAP owned SAP HANA documentation for IBM Power specifics. Furthermore it provides additional chapters referencing the related SAP HANA documentation as an entry point as the handling of SAP HANA itself is identical to Intel deployments. The installation guide does not replace existing SAP HANA literature. It provides guidance on how to meet SAP HANA operational requirements using IBM Power Systems, IBM System Storage, IBM PowerVM, and Linux Operating System. Feel free to provide feedback and change requests for this document via to isicc@de.ibm.com.

4 4 Table of Contents Preface... 3 About This Document... 3 Figures... 7 Tables... 7 Introduction... 8 How to use this Guide... 9 Used Sample Configurations... 9 Sample Configuration 1: SAP HANA Single-Host System (scale-up) Sample Configuration 2: SAP HANA Single-host System with System Replication (Disaster Recovery or Hot Standby) Sample Configuration 3: SAP HANA Single-host System with Auto Host Failover (Cold Standby) Prerequisites Verify Installed Hardware VIOS, SAN and Ethernet Connectivity Setup Setup Considerations Related Planning Sheets from the Planning Guide Zones for Ethernet and Fibre Channel VIOS configuration Native attached connectivity Ethernet Tuning Additional recommended documentation Tools to consider Sample Implementation Enable the VIOS for Jumbo frames and large_send/receive Verify the current VIO level Verify if large send/receive and Jumbo Frames are enabled Operations to change the Ethernet Adapter Values Server Setup Setup Considerations Related Planning Sheets from the Planning Guide Additional recommended documentation Tools to consider Sample Implementation... 22

5 5 HMC based partition setup PowerVC based partition setup SAN Storage Preparation Setup Considerations Related Planning Sheets from the Planning Guide SAP HANA Features Requiring special storage layouts Additional recommended documentation Tools to consider Sample implementation internal Server Disks Sample Implementation SAN Verify the prerequisites Create storage Luns Result IBM Spectrum Scale Infrastructure Preparation (SAN and Elastic Storage Server) Setup Considerations OS Preparation Setup Considerations Related Planning Sheets from the Planning Guide Additional recommended documentation Tools to consider Sample Implementation Network and Kernel setup for large_send and large_receive Change LPAR MTU setting to Ensure and verify uniform topology XFS File System and Multipath Setup Setup Considerations Related Planning Sheets from the Planning Guide Additional recommended documentation Tools to consider When using fcclient for SAP HANA auto host failover Tuning for Multipathing SAP HANA Configuration Check Tool (HWCCT) Prepare Execution... 35

6 6 Validation Landscape Test (June 2015) Tools, Links and References SAP HANA installation Setup Considerations Related Planning Sheets from the Planning Guide Additional recommended documentation Tools to consider Sample Implementation Migration Path Introduction / Scenario Migrate SAP Business Suite and SAP NetWeaver BW on traditional databases to SAP HANA database Traditional database to SAP HANA database SAP HANA database to SAP HANA database Backup Additional recommended documentation Operating System Backup Linux native method rd party software method HANA File System Backup HANA database Backup and Recovery HANA Snapshots Recovery HANA Backint Integration High Availability and Disaster Recovery SAP HANA System Replication Setup Considerations Additional recommended documentation Tools to consider Auto-Host-Failover on shared disk using XFS Configure fcclientlvm Verify fcclientlvm setup Security... 47

7 7 Support Process and Services Related Documents Disclaimer and Special Notices Copyrights and Trademarks Figures Figure 1 Implementation path overview... 8 Figure 2 Sample configuration: SAP HANA single host setup Figure 3Sample configuration: SAP HANA scale-up with SAP HANA System Replication Figure 4Sample Configuration: Cold failover setup based on local disk with fcclient Figure 5 Sample Configuration: Cold failover setup based on GPFS Figure 6 Schematic overview of Ethernet and SAN connectivity (without SAP HANA System Replication) Figure 7 Difference Dual VIOS Setup vscsi and the desired NPIV setup Figure 8 Sample Installation Server Figure 9 Sample Implementation - PowerVC based LPAR setup Figure 10 Crate Lun (part 1) Figure 11 Create LUN (part2) Figure 12 Example: find multipath settings for an IBM Storage Subsystem (here IBM FlashSystems) Figure 13 SAP HANA migration methods for BW and Business Suite (example shows BS) Figure 14 Schematic Network overview for SAP HANA System Replication deployments Figure 15 From Scale-up to SAP HANA Auto host failover Tables Table 1: VIO and Linux Settings for 10Gb Ethernet Table 2 SAP HANA database migration options... 40

8 8 Introduction SAP HANA comprises several functional modules. These are deployed on top of a hardware and software stack, which should be planned according to the IBM SAP HANA Planning Guide. SAP HANA on POWER meets the customer requirements for a high degree of configurations and administration flexibility. The hardware assembly is a two folded process of a SAP sizing exercise and an IBM mapping effort outlined in the IBM SAP HANA Planning Guide. SAP HANA systems can be deployed exploiting PowerVM capabilities. This can lower the entry barrier for SAP HANA significantly, since test systems can be easily installed on a partition using available resources or Capacity Upgrade on Demand (CoD) capacity there is no need to purchase dedicated hardware for SAP HANA on POWER. From a technology perspective, IBM Power Systems are very well suited to provide excellent SAP HANA database performance and reliability. The port of the SAP HANA code to POWER architecture benefits from excellent memory bandwidth, SIMD parallelization, and Simultaneous Multi-Threading (SMT) on POWER. Complete the IBM Planning Guide excercise to define the requirements for the hardware and software stack Setup the IBM Power Server(s) (virtualization, partitions,...) Setup 10Gb Ethernet and SAN (multipathing) Layout the LUNs on the storage system(s) and their host attachments for low latency and fast read/write for multipath XFS file systems yes XFS file system Not covered no no Setup Ethernet and SAN (both IBM Spectrum Scale I/O optimized + SAP related Ethernet) ESS filer used yes Setup Ethernet and/or Infiniband (both IBM Spectrum Scale I/O optimized + SAP related Ethernet) Layout the LUNs on the storage system(s) and their host attachments for low latency and fast read/write for IBM Spectrum Scale file systems Create boot device OS System deployment SAP HANA filesystem setup Verify SAP KPI-compliance of the deployed system SAP Appserver/ Landscape HANA System deployment Backup/Restore High Availability Disaster Recovery Migration Figure 1 Implementation path overview

9 9 How to use this Guide This guide is a supplemental to the SAP HANA Documentation. The core SAP documents which have to be used together with this guide are: SAP HANA Master Guide SAP HANA Server Installation Guide SAP HANA Administration Guide The flow and structure is aligned to the SAP HANA on IBM Power Systems and IBM System Storage Planning Guide. It also uses the planning sheets as a reference to easier interlink the planning and deployment process. The tables can be downloaded from New features will be directly reflected in the following SAP Notes for SAP HANA on Power: SAP Note HANA on POWER planning and installation specifics - central note SAP Note SAP HANA on Power: Central Release Note Each chapter will contain following information: Setup Considerations This chapter references the related planning sheets from the IBM Planning Guide for SAP HANA and will give some background information. In the subchapters Additional recommended documentation and Tools to consider blank lines have been inserted to document your findings to reuse this document as handover between teams. Sample Implementation This chapter outlines the steps taken to achieve the setup of a sample installation. Used Sample Configurations The SAP HANA on POWER architecture provides the flexibility to operate SAP HANA on IBM Power Servers on all storage solutions which can fulfill the SAP HWCCT KPIs. SAP maintains a list of TDI certified storage solutions which can fulfill those KPIs if setup correctly. This does not erase the need to verify the Storage attachment as described in the chapter SAP HANA Configuration Check Tool (HWCCT). Due to the broad range of valid option this guide is written based on the following IBM Power Server sample configurations. Even for the chosen options there is a variety of alternatives possible in addition to the described ones. If not explicitly excluded in SAP Note (attachment) all alternatives that are possible with the Appliance solutions are allowed configurations. Some pretested configurations can be found in SAP HANA in Data Centers. To make new innovative ideas reusable by others please send an to:probst@de.ibm.com. The Appserver installation steps are not covered.

10 10 Sample Configuration 1: SAP HANA Single-Host System (scale-up) Figure 2 shows a basic SAP HANA deployment within a single site (green border). V V Figure 2 Sample configuration: SAP HANA single host setup In red, the SAP HANA database processes are protected by the SAP HANA Auto-Restart feature. It automatically restarts single processes in case of failures. The server (blue) is based on PowerVM, running a dedicated donating partition for SAP HANA. A dual VIOS setup is used for redundancy of Ethernet and SAN connectivity. Dual VIOS is chosen to leverage Power features such as Life Partition Mobility (LPM), transparent maintenance of underlying structures and consolidation (additional partitions with a different workload are running on the same server to gain optimal redundancy with an optimized number of connectivity network and fibre-channel ports). The used file system is XFS. Sample Configuration 2: SAP HANA Single-host System with System Replication (Disaster Recovery or Hot Standby) SAP HANA System Replication provides business resiliency based on a shared nothing concept using multiple sites/servers (green boxes). This sample uses XFS file systems. The Ethernet and fibre-channel cards are direct attached to the partition (without VIOS). To automate the failover, cluster solutions can be deployed. This is mandatory when using this setup as a HotStandby solution. System Replication Figure 3Sample configuration: SAP HANA scale-up with SAP HANA System Replication Sample Configuration 3: SAP HANA Single-host System with Auto Host Failover (Cold Standby)

11 11 This deployment is using SAP HANA Auto Host Failover for a cold standby high availability solution. For single-host HANA nodes the failover node can be stretched to the same extent the storage can be stretched. Auto Host failover The two most common options are: fcclient fcclient 1) Using SAP HANA Auto Host failover in a SAN with local file systems requires a SAP HANA component called Storage Connector coming in two versions: fcclient and fcclientlvm. The Storage Connector is Figure 4Sample Configuration: Cold failover setup based on local disk with fcclient responsible to mount the SAP HANA file system on the standby node to resume operations based on the latest data and log content (Figure 4). The client reconnect is done based on adding both nodes IP addresses to the connectivity list. The alternative would have been to exploit the SAP HANA failover hooks to move the service IP address or replacing the SAP HANA Auto Host failover feature by a cluster solution such as SA MP. The preferred IBM storage technology for stretching this setup is the SVC stretched cluster. 2) GPFS can be used to make the data and log of the SAP HANA instance accessible from both hosts. Concurrent writes are avoided by SAP HANA software. Data Protection has to be put in place using GPFS features and SAP HANA backups. Before selecting this option verify SAP Note for GPFS Support. Auto Host failover GPFS Figure 5 Sample Configuration: Cold failover setup based on GPFS Prerequisites Before starting with the Implementation Guide ensure to have well planned, ordered, and setup the required hardware as described in the Planning Guide.

12 12 Assemble the right resources to the HANA installation project. This requires: A storage specialist for: o I/O performance and disk layout with knowledge of the chosen file system option o Flash Copy, Volume Mirroring (sync/async), and Storage Virtualization o Host attachment of LUNs A SAP HANA and SAP Basis specialist for installing HANA, migration, application server connectivity, backup, HANA System Replication and SAP HANA Auto Host failover o Required Skill can be determined based on the SAP HANA Class: E_HANAINS131 A Power Server specialist for: o Setup HMC and IBM Power Servers o Prepare partitions and optional VIOS servers A Linux specialist for Linux on POWER for: o OS setup and installation accordingly to the OS vendor and SAP requirements o File system setup and tuning Optional skills are required for: o IBM Tivoli System Automation for Multi Platforms or other cluster solutions to automate HA and DR capabilities of HANA o Security o Monitoring tools o IBM Spectrum Scale 1 SAN or Elastic Storage Server (ESS) planning and tuning. o The guide assumes the hardware is setup and configured to the extent that partitions can be created and storage volumes can be attached to the server host. Carefully check if all items of the solution are supported within each other. For instance, SAP HANA TDI compatibility matrix for storage, and storage vendor compatibility matrix against Linux versions. Decide on network configuration for each used VLAN: Hostname IP Netmask Gateway Domain name At least, two DNS servers NTP server Confirm whether or not the Power System is using HMC and plan accordingly. If using HMC, confirm it is running at least HMC V8R8 for POWER8 servers. SAP setup parameters: SAP HANA SID SAP HANA Instance number Password for SAP HANA 1 IBM Spectrum Scale is the new branding for GPFS. Verify SAP Note for GPFS Support of SAP HANA

13 13 Verify Installed Hardware The tables referred to are from the Planning sheets which are completed as part of the Planning process. Download of sources: Server Use Table 3. SAP HANA Server Planning Sheet to verify the server in regards to type, virtualization and resources. Verify the Table 3 really reflects the needs of the partition design as outlined in Table 2. SAP HANA Partition Resource Planning Sheet, Table 7. VIOS Server Resource Planning Sheet, Table 8.Ethernet Connectivity Planning Sheet, Table 9.Fibre Channel Planning Sheet. Put shared resources into the context of VIOS if installed. Storage The Storage backend is shared inside the data center. Ensure the chosen Storage systems have reserved resources for the Server s workload. Table 4 summarizes the space requirement per Storage. Ensure data protection methods are in place and appropriate backup devices are installed and connected. Cabling and Zoning Ensure path redundancy is in place for all paths. Verify with the SAP HANA guides if the setup meets the requirements of the chosen SAP HANA installation type, Table 11.5 (e.g. SAP HANA System replication, Scale-up/out, ). Put the Installation type into the context of additionally planned Software, Table 13. Eco System Software for SAP HANA(e.g. SA MP cluster Manager, TSM Backups, ).Verify if the requirements of Table 8. Ethernet Connectivity Planning Sheet and Table 9. Fibre Channel Planning Sheet are in place. VIOS, SAN and Ethernet Connectivity Setup The design implemented must result in SAP specified file system and in case of scale-out Ethernet performance characteristics. To achieve the requirements, the following chapter describes a set of technologies and settings. The related SAP documentation and tool to verify the I/O characteristics is described in chapter SAP HANA Configuration Check Tool (HWCCT). Following the IBM Planning Guide will result in a valid hardware configuration design. SAP HANA on Power does not follow the Intel appliance model. SAP has established a TDI process for Power allowing for the desired flexibility in hardware configurations. Note: SAN based IBM Spectrum Scale (formerly GPFS) as well as Ethernet/Infiniband attached Elastic Storage subsystems are not documented and not pretested as of today. The requirements for those will be partially different to XFS based deployments. The same is true for internal disk deployments. Setup Considerations Ensure the deployment can also meet requirements resulting from additional partitions (Table 3.8).

14 14 Related Planning Sheets from the Planning Guide Table 2. SAP HANA Partition Resource Planning Sheet Table 3 SAP HANA Server Planning Sheet Table 5. Minimum Single-Host SAP HANA I/O Adapters Table 6. Minimum Multi-Host SAP HANA I/O Adapters per Server Table 7. VIOS Server Resource Planning Sheet Table 8. Ethernet Connectivity Planning Sheet Table 9. Fibre-channel Planning Sheet Zones for Ethernet and Fibre Channel The architecture is built on the invisible invincible principle. This principle is based on utilizing IBM s virtualization technologies to be able to survive single outages in the infrastructure without any interruption to the business application and to perform zero downtime maintenance. To obtain this from the OS level: redundancy, dual VIOS and Storage virtualization is mandatory. Some documentation is listed below in Additional recommended documentation. The partitions are setup using the dm-multipath driver for fibre channel attached storage. Ensure the zoning will result in the appropriate amount of paths (see Planning Sheets). Typical deployments will be in a high availability setup. In order to have a vital heart beating infrastructure please refer to your cluster vendor s guides. Also SAP introduces requirements. The first requirement is based on the overall SAP landscape aspects. This involves the SAP shared file systems (HANA-shared, SAP global and transport) typically served based on central high available NFS servers or an IBM Spectrum Scale/Elastic Storage Server (GPFS) or a V7000U.The second requirement comes from the security needs. And the third is based on the SAP HANA installation option. A SAP HANA System replication requires Ethernet connectivity between the two peers, SAP HANA scale-out with Auto Host Failover requires a special fibre channel zoning for shared disks and a strong HANA-shared file system across the nodes. Those considerations are well described in the SAP HANA documentation. Examples are shown in Figure 6 and Figure 14.

15 15 Frontend Single Node SAP HANA Server Dedicated[-donating] Partition for HANA SAP, IBM, 3rd party management and monitoring tools ETH0 ETH1 FC0 FC1 FC2 FC3 Dedicated Partition VIOS Dedicated Partition VIOS SAP application server 1 GB 10 GB HBA HBA 10 GB 1 GB Admin FC Switch FC Switch Business Application Business Application SAN Storage Figure 6 Schematic overview of Ethernet and SAN connectivity (without SAP HANA System Replication) VIOS configuration Exploiting dual VIOS has three key advantages: The physical resources for Ethernet and fibre-channel can be lowered if more than a single LPAR is installed. Redundancy of adapters at a lower price tag. LPM (Live Partition Mobility 2 ) and rolling maintenance. The dual Virtual I/O Server setup must be designed to provide continuous access to the SAN and network. Hence each of the VIOS must have physical dedicated fibre-channel and Ethernet cards. They have to be configured for SEA-failover or a future technology providing at minimum the same redundancy. NPIV capable HBA cards have to be used for the standardized setup. The clients are configured using dm-multipath driver. 2 Using LPM for a SAP HANA database under load will likely not successful due to the large memory footprint along with ongoing fast changes. But there are use cases where LPM is beneficial.

16 16 SAN SAN Physical Physical FC FC Adapters Adapters Physical Physical FC FC Adapters Adapters Physical Physical FC FC Adapters Adapters Physical Physical FC FC Adapters Adapters Device Driver Multi-pathing Device Driver Multi-pathing LUN hdisk SCSI LUN hdisk dm-multipath VSCSI server Adapter VSCSI client VSCSI client VSCSI server Adapter VIOS 1 Linux Partition VIOS 2 Virtual FC server Adapter Virtual FC client Virtual FC client Virtual FC server Adapter VIOS 1 Linux Partition VIOS 2 NOT RECOMMENDED VSCSI SETUP Figure 7 Difference Dual VIOS Setup vscsi and the desired NPIV setup RECOMMENDED NPIV SETUP Native attached connectivity There are reasons to setup a Server in the full partition mode. If so please ensure the following: Physical redundancy has to be ensured in sense of HBA cards and ports for Ethernet and fibre channel. Use Jumbo frames for Ethernet (MTU 9000) to utilize the 10Gb Ethernet ports For Ethernet bonding is recommended to protect the partition from single adapter failures. Ethernet Tuning The SUSE SLES 11 SP3 cannot support large_send and large_receive required to utilize the full 10 Gb ports from a single LPAR nor the VIOS provided vlan bandwidth when sending large amounts of small packets. This can become in scale-out deployments a bottleneck but is uncritical for scale-up deployments. There are tuning opportunities which can be evaluated depending on the individual workload today. Jumbo frames (MTU 9000) enabled in OS, VIOS and Infrastructure improves I/O ~3 times from an OS perspective. This setting helps VIOS and native attached Ethernet. Ensure when the OS does not support large_send and large_receive that the VIOS has this disabled too. SUSE SLES 11 SP4 has the large_send/large_receive (min kernel-ppc64-base ). When moving to SP4 it is recommended to enable this feature in all partitions on this server and VIOS.In this case the VIO must have level or, if lower, ifix IV72825 installed. Please verify if the components used (adapters, OS and VIOS) do support new technology at the time the deployment is done (SR-IOV, ). Note: Before starting to tune for fast Ethernet ensure all OS, VIOS and switches can support this configuration. If one does not support the tuning this will add additional overhead and a reduction of throughput. Additional recommended documentation IBM System Storage Architecture and Configuration Guide for SAP HANA TDI

17 17 Tools to consider Use HWCCT Network Test facility for scale-out SAP HANA installations: Chapter SAP HANA Configuration Check Tool (HWCCT). Virtual I/O Server Performance Advisor: In order to ensure the vitality of the Virtual I/O Server the Performance advisor can be used. The Virtual I/O Server Performance advisor is an application that runs within the customer's Virtual I/O Server for a user specified amount of time (hours), which polls and collects key performance metrics before analyzing results and providing a health check report and proposes changes to the environment or areas to investigate further. The goal of the Virtual I/O Server Performance advisor is not to provide another monitoring tool, but instead have an expert system view performance metrics already available to the customer and make assessments and recommendations based on the expertise and experience available within the IBM systems performance group. For more information on Virtual I/O Server Performance Advisor, see, SG Sample Implementation Download the latest Redbook for the chosen adapters and VIOS. This implementation does not differ from a normal deployment for Linux partitions on Power. Figure 6 provides the schematic view of the sample installation. If assistance is desired please see chapter Support Process and Services. Enable the VIOS for Jumbo frames and large_send/receive If the OS, VIOS and the network infrastructure including the switches support Jumbo frames and large_send and large_receive the following steps must be performed. Note: 1) large_send/receive is supported by SUSE starting with SLES11 SP4. 2) NIM Servers often cannot be run in Jumboframe enabled environments Recommended Network settings for VIO and Linux versions: Table 1: VIO and Linux Settings for 10Gb Ethernet VIO Version Linux Version Setting for VIO Setting in Linux OS Large_receive tso x SLES11 SP3 no off (not available) SLES11 SP4 no off x and SLES11 SP3 yes off (not available) IV72825 SLES11SP4 yes on x SLES11 SP3 yes off (not available) SLES11 SP4 yes on

18 18 Verify the current VIO level In order to use large send and receive make sure you have at least VIOS level installed. As padmin check with $ ioslevel If the result is lower than verify if IV72825 is installed: $ oem_setup_env # emgr -lv3 grep IV72825 EFIX LABEL: IV72825s5a LABEL: IV72825s5a IV72825 : CALCULATE IP HEADER CHECKSUM FOR LARGE RECEIVE PACKETS. If IV72825 is not installed, and ioslevel returns a value less then jumbo frames and large send can be enabled. Do not enable large receive in this case as you will observe an unstable network connection. Verify if large send/receive and Jumbo Frames are enabled Use the following commands to verify if the underlying hardware adapter has jumbo frames enabled as user padmin: $ oem_setup_env # lsattr -El ent4 jumbo_frames yes Request jumbo frames True jumbo_size 9014 Requested jumbo frame size True large_receive yes Request Rx TCP segment aggregation True large_send yes Request Tx TCP segment offload True If the retrieved values do not match the values above you have to perform the following operations starting with step 1: Remove shared Ethernet Adapter If the hardware parameters above match your settings check the values of the currently running interfaces (en6 is the shared ethernet adapter in this example): # netstat in Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll en link#2 0.2.c9.5a.c0.f en If the values above match your settings the VIO is setup correctly. Otherwise continue with Step 3

19 19 Operations to change the Ethernet Adapter Values Step 1: Remove shared Ethernet Adapter Select in the HMC the server whose LPARs and VIOS should be enabled. Then go to Configuration Virtual Resources Virtual Network Management on the HMC. After selecting the appropriate button with the name Remove SEA is displayed. Clicking on this button opens a window with instructions and commands for all VIOS partitions involved. If the parameters for the hardware interface have to be changed it can be accomplished by taking down the interface described in Step 1. If the parameters already match continue with Step 2: Step 2: Setup Hardware Adapter: Please note that the operations in this step bay cause a disruption of the network connectivity of the VIO and associated LPARs: Inside the VIOS server the following instructions have to be executed (example is ent4): $ chdev -dev ent8 -attr lldpsvc=no ent8 changed $ chdev -dev en8 -attr state=detach en8 changed $ rmdev -dev ent8 ent8 deleted $ rmdev -dev en8 en8 deleted $ rmdev -dev et8 et8 deleted $ chdev -dev en4 -attr state=up restoreroute en4 changed $ oem_setup_env # ifconfig ent4 down # ifconfig ent4 detach # chdev -dev ent4 -attr jumbo_frames=yes Verify the settings from padmin with $lsdev -dev ent4 attr grep jumbo jumbo_frames yes Request jumbo frames True jumbo_size 9014 Requested jumbo frame size True Now recreate the shared Ethernet adapter with the web interface. Continue with Step 3

20 20 Step 3: Verify MTU of SEA Verify the MTU of the SEA, in this case en8. $ oem_setup_env # netstat -in Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll en link#2 0.2.c9.5a.c0.f en If the MTU is 9000 verify the settings with: $ oem_setup_env # ifconfig en8 If LARGE_SEND is set you are finished, otherwise continue with step 4. Step 4: Change MTU and set large send/receive on SEA Login as padmin on VIO with a console, SEA in this example is en8 $ oem_setup_env # ifconfig en8 down # ifconfig en8 detach # rmdev -l ent8 ent8 Defined # chdev -l ent8 -a jumbo_frames=yes ent8 changed The following step sets large send. Look up in Error! Reference source not found. which setting you can apply. Using a wrong setting may cause a massive network performance degradation or even to loss of networking functionality. # chdev -l ent8 -a large_receive=yes ent8 changed # lsattr -El en8 # mkdev -l ent8 ent8 Available # chdev -l en8 -a state=up en8 changed # mkdev -l inet0 inet0 Available # netstat -in Repeat step 4 on the second VIO.

21 21 Server Setup This chapter describes a manual deployment of a dedicated partition for SAP HANA. Setup Considerations Related Planning Sheets from the Planning Guide Table 2 SAP HANA Partition Resource Planning Sheet Table 3 SAP HANA Server Resource Planning Sheet 1) Production systems must run on POWER8 processor based servers or upcoming new models. in dedicated or dedicated donating partitions 2) None Production systems can run Functional on POWER7+ processor based servers with a potential performance impact. in a shared pool accepting performance degradation. With a relaxed core to memory ratio ( 3) There are no special Power Systems firmware requirements introduced by SAP HANA, however it is recommended to update if a HYPER defect exists on the current installed version. 4) Memory virtualization features such as Advanced Memory Sharing (AMS), Advanced Memory Expansion (AME), or Memory Deduplication are not supported on SAP HANA partitions. 5) In the SAP HANA, version for IBM Power Servers Architecture Initial Shipment Stack CoD planning is possible for a subset of possible architectures. The initial GA code requires a cold operation (restart) to enable HANA to pick up changes in CPU or memory resources and adapting the *.ini files. Additional recommended documentation SAP Applications on PowerVM ( Tools to consider PowerVC

22 22 Sample Implementation E t h HANA Dedicated/- donating LPAR 1 LPAR 2 Shared CPU Pool VIOS 1 dedicated VIOS 2 dedicated E t h F C F C E t h E t h F C F C V V PowerVM CPU CPU CPU CPU CPU CPU CPU MEM MEM MEM MEM MEM MEM MEM Figure 8 Sample Installation Server For the I/O configuration see chapter Verify Installed Hardware HMC based partition setup In the IBM Knowledge Center you can find a good documentation how to setup Partitions using the IBM Hardware Management Console (HMC) ( 01.ibm.com/support/knowledgecenter/ L/p8hat/p8hat_createlpar.htm?lang=en). Please verify that the following configuration parameters are set accordingly to the values from the planning sheets out of the Planning Guide. Verify if the IBM Power Server can hold the HANA Partition and provides sufficient space for the overall workload planned for that server. (See Table 2 and Table 3 from the Planning Guide) In the sample installation the HMC partition wizard was used to create the HANA partition. This step has to be repeated as documented in Table 2.1 Ensure to choose a dedicated partition

23 23 Use the SAP HANA CPU count from Table 2.3 as Minimum processors value Use the SAP HANA Memory value from Table 2.2 as Minimum Memory value. Ensure to NOT activate AME. For the I/O configuration verify the number of minimum paths as outlined in Table 2.4 and 2.5 can be met. PowerVC based partition setup In a preconfigured PowerVC log in and press Images button on the left hand side. Select an image to deploy and press the Deploy button. Enter the desired hostname and IP and create the partition. Then press Deploy on the bottom.

24 24 Figure 9 Sample Implementation - PowerVC based LPAR setup SAN Storage Preparation The SAN storage preparation tasks discussed in this chapter comprises of: Understand the SAP HANA I/O performance requirements reflected in the SAP Documentation and the SAP HWCCT (Hardware Configuration Check Tool) Tool. Creating the required set of volumes and host attachments in a suitable manner. The planning and deployment of additional SLA requirements for data redundancy, backup as well as HA and DR capabilities special to SAP HANA and the SAP Landscape are not covered as this is a very customer specific architecture. Note: When planning for an IBM Spectrum Scale solution on SAN the storage configuration has to be adapted to the I/O patterns of GPFS, which are different from the here provided implementation recommendations. Special caution is required in SVC based environments. Ensure the chosen GPFS allows for all operations (e.g. LPM,..). Please contact your GPFS Specialist before create arrays and ranks. Setup Considerations The Storage and attachment selection criteria are: a) Driven by the load time a SAP HANA requires to start or to restart after a failure. b) The I/O performance requirements for SAP HANA data and log files c) In the context of shared resources

25 25 The overall Storage volume planning is an additive process starting from the SAP HANA Quicksizer output to a SLA orientated solution including Backup, HA, data Protection,. Finally the volumes and attachments have to also respect the SAP HANA installation type chosen as a SAP HANA System Replication configuration has different needs than a SAP HANA scale-out with DR objectives covered by Storage replication or third party tools. Related Planning Sheets from the Planning Guide Table 4. Storage Planning Sheet Table 5. Minimum Single-Host SAP HANA I/O adapters for production (floor configuration) Table 6. Minimum Multi-Host SAP HANA I/O Adapters per Server (floor configuration) Table 9. Fibre Channel Planning Sheet Table 10. SAP HANA File System SAP HANA Features Requiring special storage layouts SAP HANA built-in HA and DR capabilities require to connect disks accordingly to chosen HA functionality (find more in chapter Backup and

26 26 High Availability and Disaster Recovery ). Additional recommended documentation SAP HANA Storage Requirements IBM System Storage Architecture and Configuration Guide for SAP HANA TDI Tools to consider SAPmagic - Storage Sizing Tool for SAP landscapes Hardware Configuration Check Tool - Central Note (SAP Note ) Sample implementation internal Server Disks On the HMC drives attached through VIOS: To view the drives log into Linux: lsblk f lists th edisks

27 27 Sample Implementation SAN The prerequisite for this sample is a storage setup ready to create the luns and host attachments. Verify the prerequisites Use Table 9. Fibre Channel Planning Sheet to identify the zones and ensure that the cabling and zoning is sufficient to result in sufficient paths in the operating system as documented in Table2.5. Match the result with Table 4.8. If Table 10 indicates the use of GPFS ensure the Storage formatting and storage type allows HANA to perform sufficient I/O operations. Use Table 4 to verify the Selected Storage Subsystems. Ensure the configuration is not below the minimum requirements as outlined in Table 5 and Table 6) The list of Host required attachments can be found in Table 2.6 Create storage Luns Step 1: Verify based on Table 4.7 if the storage subsystems have sufficient space before starting. Also include additional workload considerations as documented in Table 3.8. This can be done in SVC Systems by looking at the bottom of the screen: You can drill down deeper to verify for the over commitment level when using thin provisioning.

28 28 Step 2: The SAP HANA volume for data has to be as planned in Table As SLA we want to keep 3 copies without any additional data protection mechanisms. In the multipath setup in the OS we have 4 active paths (Table 2.5). To have a performant setup 4 LUNs with 128GB each are created and attached to the host. 2 Open the Volume view (1), press Create Volume (2), Select a Present (we used Generic as a sample) and finally (3) select the pool and double click. 1 This will bring you to the next screen Figure 11. Specify the LUNs as shown and press Create and Map to Host. Repeat this for all required LUNs 3 Figure 10 Crate Lun (part 1) Figure 11 Create LUN (part2)

29 29 Result The sample installation will combine the SAP executables and the root file system on a single disk. SAP HANA Data and Log filesystem are separated strictly from each other The HANA shared filesystem will be served from a central, high available NFS Server later. No Backup strategies are implemented in this sample. IBM Spectrum Scale Infrastructure Preparation (SAN and Elastic Storage Server) Note: Under Investigation. Setup Considerations Using a cluster solution: alternative heart beating can be performed based on the given connectivity and disk. It has to be ensured that different protocols are used. Typically this is threefold using fibre channel (sfwcom), Ethernet and disk. A solution for the Boot disk has to be put into place. GPFS is very sensitive in regards to tuning from the disk up to the operating system. Ensure the right setup and regular monitoring if the system still meets the performance expectation. Ensure the GPFS version supports dual VIOS and Life Partition Mobility (LPM). IBM spectrum Scale (GPFS) support matrix lists the selected OS. OS Preparation The SAP Note provides the official SUSE Implementation and tuning guide for SAP HANA. Setup Considerations It is strongly recommended to enable the landscape for Jumbo frames and large_send/receive for production systems, if the operating system supports them. Now is also the best time to ensure that the LPAR has a valid numa layout (uniform topology) to get out best performance from the SAP HANA database.

30 30 Related Planning Sheets from the Planning Guide Table 11. SAP Software Planning Sheet Additional recommended documentation SAP Note SAP HANA Guidelines for SLES Operating System Tools to consider PowerVC Sample Implementation In our sample implementation PowerVC instantly created the OS along with the creation of the LPAR based on a snapshot. (See chapter PowerVC based partition setup ) Network and Kernel setup for large_send and large_receive Setting up the VIOS for large_send/receive and Jumbo frames ensure the same settings are applied to all partitions served by these VIOS. It is strongly recommended to enable all Power Servers communicating to these for these settings as well. This includes also appropriate switches. In order to use large send and large receive the following prerequisites have to be matched: All VIOs associated to the LPAR have to be enabled for large send/receive SLES 11 SP4 Kernel version greater or equal kernel-ppc64-base has to be installed Large send/receive has to be enabled with ethtool -K eth0 tso on (can be verified with ethtool -k eth0) Change LPAR MTU setting to 9000 # cd /etc/sysconfig/network # vi ifcfg-eth0 TYPE=Ethernet DEVICE='eth0' BOOTPROTO='static' BROADCAST='' ETHTOOL_OPTIONS='-K eth0 tso on' IPADDR=' ' MTU='9000' HWADDR='FA:82:2E:BA:8B:20' NETMASK=' ' NETWORK='' REMOTE_IPADDR='' NM_CONTROLLED='no' STARTMODE='auto' USERCONTROL='no' ONBOOT=yes _nm_name='static-eth0'

31 31 Restart the deamon # /etc/init.d/network restart Shutting down network interfaces: eth0 done Shutting down service network..... done Hint: you may set mandatory devices in /etc/sysconfig/network/config Setting up network interfaces: eth0 eth0 IP address: /23 done Setting up service network done Verify the setting # ifconfig -a eth0 Link encap:ethernet HWaddr FA:82:2E:BA:8B:20 inet addr: Bcast: Mask: inet6 addr: fe80::f882:2eff:feba:8b20/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1 RX packets: errors:0 dropped:0 overruns:0 frame:0 TX packets:4228 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes: (40.7 Mb) TX bytes: (454.2 Kb) Interrupt:32 [ ] Ensure and verify uniform topology Log into the LPAR and ensure the command numactl hardware has a symmetric output. In special ensure every NUMA Node with a core hold memory as well. The following LPAR has a bad numa layout. NUMA node0 has cores but no memory what can cause crashes of the SAP HANA Database. NUMA node 6 holds all the memory but no cores what degrades the performance of SAP HANA. lsh30021:~ # numactl hardware available: 2 nodes (0,6) node 0 cpus: node 0 size: 0 MB node 0 free: 0 MB node 6 cpus: node 6 size: MB node 6 free: MB node distances: node 0 6 0: : 40 10

32 32 Analyzing and fixing memory placement with the dynamic platform optimizer (DPO): In the newer versions of the firmware, the hypervisor team provided methods to analyze and fix memory placement issues on the ssh shell of the HMC. This works transparent for the applications and can be executed while every LPAR is up and running as long as there is some memory free on the machine and if this feature is enabled. To list the servers: lssyscfg -r sys -F name To analyze the current situation you can run the command lsmemopt: lsmemopt -m <Power Server Name> -r lpar -o currscore lpar_name=<name of LPAR>,lpar_id=1,curr_lpar_score=100 lpar_name=<name of LPAR>,lpar_id=2,curr_lpar_score=100 lpar_name=<name of LPAR>,lpar_id=3,curr_lpar_score=none lpar_name<name of LPAR>,lpar_id=4,curr_lpar_score=100 lpar_name=<name of LPAR>,lpar_id=5,curr_lpar_score=none lpar_name=<name of optimal LPAR>,lpar_id=6,curr_lpar_score=100 lpar_name=<name of improvable LPAR>,lpar_id=8,curr_lpar_score=74 lpar_name=<name of LPAR>,lpar_id=12,curr_lpar_score=32 lpar_name=<name of LPAR>,lpar_id=31,curr_lpar_score=none The tool does not show the exact placement of memory and cores. Instead it does a rating, where 100 is the best and 0 is the worst. As you see it rated my LPAR <name of optimal LPAR> with a 100, which means the placement can not be improved according to the rules of the hypervisor. The example shows that LPAR <name of improvable LPAR> has not the perfect rating. Running the command above with the option -o calcscore would tell to what degree DPO can optimize the LPAR based on the current situation. To use DPO to optimize the LPARS use the command optmem: optmem -m <Power Server Name> -o start -t affinity -p <name(s) of improvable LPAR(s)> All other LPAR's were candidates to be changed as well, in order to achieve best placement for the mentioned LPAR's. If a LPAR should not be touched, it can be specified with another option ( see man pages for optmem ). Since the command is running in the background you can check the optimization status with: lsmemopt -m <Power Server Name>

33 33 Analyzing and fixing memory placement manually: In case the DPO does not optimize the memory placement to the required extend, or the feature is not enabled, there are also some options to fix the issues manually. When the hypervisor creates an LPAR it assigns the Cores ( or Entitlement ) and Memory Dimms to the LPAR and this assignment is fixed, even when the LPAR is later restarted. So the idea behind the solution is to remove the fixed assignment of resources and then hope, the hypervisor is choosing a better placement, when the LPAR is activated later again. This obviously will work only, if you changed the profile of the LPAR to fit better to the available resources, or if other LPARs occupying needed resource have been changed. To get the memory assignment of LPAR's run the command lshwres: hscroot@is37ssp:~> lshwres -m <Power Server Name> -r mem --level lpar -F lpar_name,lpar_id,curr_mem <LPAR name>,12, <LPAR name>,31,0 <LPAR name>,6, <LPAR name>,5,0 <LPAR name to be fixed>,4, <LPAR name>,3,0 <LPAR name>,2,4096 <LPAR name>,1,4096 <LPAR name>,8, To remove the fixed memory assignment the command chhwres can be used on the HMC: chhwres -m <Power Server Name> -r mem -o r --id 4 -q So the above command would remove the memory assignment of LPAR <LPAR name to be fixed> (ID = 4 ). The option -q is the quantity to remove and this is the curr_mem from the previous list. The strategy to fix the memory placement is then basically the following: 1) Shut down every LPAR which needs to be changed 2) Get the correct memory settings with the command lshwres (see above) 3) Remove the memory assignment with chhwres 4) Start the LPAR's in the correct priority order and hope the hypervisor is doing now a better placement 5) Check the new placement again with the resource dump.

34 34 XFS File System and Multipath Setup The SAP Note provides the official SUSE Implementation and tuning guide for SAP HANA. Setup Considerations The following subchapters might bring up additional requirements to review depending on the chosen architecture 3 : Using Linux the pvid is not guaranteed to be unique. When creating the volumes always use the WWN/UUID. This is mandatory for Auto-Host-Failover and cluster setups. Never partition physical volumes, in order to be able to implement Auto-Host-Failover and cluster setups now or later. Partitioning will also impact the performance of XFS. For Auto-Host failover do not add the file systems into /etc/fstab to avoid concurrent mounts. The mounting is done by fcclientlvm (SAP) or the cluster manager. Apply SAP Note if applicable. Related Planning Sheets from the Planning Guide Table 10. SAP HANA File System Additional recommended documentation Tools to consider SAP HWCCT Filesystem Test When using fcclient for SAP HANA auto host failover The tool fcclientlvm comes together with SAP HANA to handle the unmount and mount of the SAP HANA data and log file systems if a HANA worker changes its node due to a failure. This scenario is also valid for a scale-up solution having one master and one standby. To make this handling robust certain prerequisites have to be fulfilled for HANA data and log volumes. Carefully read the SAP HANA documentation for these setups. Tuning for Multipathing Depending on the chosen storage subsystem the configured policy and features the service time and/or I/O protection can be positively or negatively affected. 3 There might be additional considerations than listed here.

35 35 To optimize performance and protect the I/Os from a path loss please apply following settings which are partially different from those documented in the IBM System Storage Architecture and Configuration Guide for SAP HANA TDI documentation: Step 1: Read in the IBM Knowledge Center the recommended multipath settings for the chosen storage subsystem: Step 2: Cross check the IBM Redbooks if there is a newer tunableset for host attachments. Figure 12 Example: find multipath settings for an IBM Storage Subsystem (here IBM FlashSystems) Step 3: Follow the OS Implementation and tuning guide attached to SAP Note Following SUSE documents provide additional options to tune a file system s I/O: tml (preferably do NOT use completely fair queuing cfq ). SAP HANA Configuration Check Tool (HWCCT) Before starting an installation of SAP HANA, the HWCCT tool, which is part of the SAP HANA installation images, has to be configured and run to verify the setup. For scale-up deployments please run the FileSystem and the Landscape test. For scale-out run all tests included. Only the so called LandscapeTest in HWCCT has a Power specific output. Hence, this chapter will focus only on the landscape related functional module. All remaining functionality is generic and described in the following SAP document: Hardware Configuration Check Tool - Central Note. Prepare Follow the SAP Note Hardware Configuration Check Tool - Central Note to install the tool. Ensure to download the ppc64 build. For single node setups the tool can be placed into any directory. For scale-out please follow the instructions given by SAP.

36 36 Execution # source envprofile.sh #./hwval -Vf <my-landscape>.json Validation For the initial release of SAP HANA, version for IBM Power Servers Architecture the landscape test and the file system test results of HWCCT have to be verified against the following rules. Later releases might have different outputs. Landscape Test (June 2015) ================================================================================ EVALUATED SYSTEM SETUP ON lsh35068 : ================================================================================ validatedistribution : SUCCESS <Validates the OS Distribution to be at least at the minimum SUSE SLES 11 SP3 for SAP HANA SPS9 code line what is the initial shipment stack of the SAP HANA, version for IBM Power Servers Architecture product.> validatelinuxkernelversion : SUCCESS <Validates the general SAP HANA and OS kernel prerequisites if validatedistribution returns SUCCESS.> validateruntime : SUCCESS <Validates if important packages are installed if validatedistribution returns SUCCESS. A SUCCESS of this test cannot guarantee that the latest changes in prerequisites and ptf are applied. A Manual check against SAP Note must be performed.> validatecpu : SUCCESS <If the Power Server runs at least a POWER 7+ processor generation the following Information will be displayed if at least 2 CPUs are assigned: "POWER 7+ processor architecture systems can only be used for none-production systems.. Otherwise this test will return a WARNING Not enough cores as this system is likely to be even unusable for a sandbox system If a POWER8 and later processor generation is used the following checks are applied: If the CPU count of this partition is between the floor and ceiling configuration SUCCESS If the CPU count is below the minimum for production systems this system must not be operated as production WARNING: none production system only.

37 37 If the CPU count is above the ceiling as documented in SAP Note WARNING: ensure to have SAP Note applied. First verify if a new ceiling I published in SAP Note All other cases the tool will indicate the reason (wrong CPU model or not enough cores and will return FAILED.> validatehypervisor : SUCCESS <Validates if the Hypervisor in lscpu is phype to verify this Partition runs on PowerVM. Others are not verified if they can operate HANA in a save and performant manner today.> validatehtt : SUCCESS <Validates enabled partition features and warns if this will impact production systems. In the initial GA HANA version this is: 1) If the partition does not run in dedicated or dedicated donating mode WARNING: only for none production systems. 2) If the partition runs below SMT 8 WARNING: This LPAR runs on less than eight SMTs what can cause lower performance. Use lscpu to verify CPU settings and decide for the preferred value. The command ppc64_cpu --smt=x can be used to dynamically increase SMT settings. 3) If AEM (Active Energy Manager) is active WARNING: This partition is using AEM (IBM Active Energy Manager). This can decrease or increase performance characteristics of this partition dynamically during operations.> 4) If AMS or AME (both memory virtualization features) are enabled the following text will be displayed: Memory visualization (AME or AMS) is not supported for SAP HANA production systems due to performance degradation. 5) DSCR setting = validatecorememoryratio : SUCCESS <This check applies for partitions which showed standard CPU and Memory configurations only. This check returns SUCCESS when the per core Memory is <= the max core to memory ratio documented in SAP Note > validatememory : SUCCESS <validates floor and ceiling memory configuration. If this partition runs additionally an application server this check is not applicable as the LPAR memory is taken for the test now containing

38 38 additional memory for the additional workload. If this partition is only used for a single SAP HANA Database the following rules are applied: 1) If less than 128 GB but more than 64 GB Memory is installed WARNING: Only none production systems 2) If less than 64 GB Memory is installed FAILED: This partition is so small that even a Sandbox installation as such might fail. 3) Is more than the general maximum Memory as outlined in SAP Note is installed WARNING: Apply SAP Note ) Returns SUCCESS for all Memory configurations between the floor for production (128 GB) and the ceiling (SAP Note ) Test Process Summary: Host configured prepared runtest gotresults cleanedup localhost:56809 OK OK OK OK OK <Verify if you see five OK to ensure all test have been executed> Tools, Links and References Hardware Configuration Check Tool - Central Note SAP HANA installation Setup Considerations For HANA installations always use hdblcm which is part of the SWPM (Software Provisioning manager) as some functionality is not included in hdbinst. Related Planning Sheets from the Planning Guide Table 12. SAP HANA Installation Planning Sheet Additional recommended documentation E_HANAINS131: Hana certification Tutorial Video Installing SAP HANA server - Concepts: (This link will provide additional education material) Tools to consider SAP SWPM (Software Provisioning manager)

39 39 X Sample Implementation See SAP Documentation. Migration Path Note: this chapter is only a summary and no guide to perform a database migration. It outlines important thoughts to take based on questions asked by other customers. Additional information can be added writing an to the IBM SAP International Competence Center (ISICC) info service: isicc@de.ibm.com. Introduction / Scenario Migrating data to a SAP HANA on POWER setup the SAP business application, the source database and the architecture has to be taken into account. The following three subchapters will describe the required steps to be taken into account in addition to the SAP HANA official product documentation. SAP Application Layer: Migrating an existing SAP system to the SAP HANA database it has to be ensured to upgrade the SAP release prior to migration, if the source system is not on a version of SAP which supports HANA DB. Please read the following documents as a starting point: SAP Business Suite on HANA: SAP NetWeaver BW Powered by HANA: Database Layer: Migrating an existing stand-alone and traditional database to the SAP HANA database, additional considerations have to be taken into account. Please contact you Database vendor and SAP for further information. Architectural Layer: Migrating an existing SAP HANA database is endian sensitive. This subchapter will provide a decision table. Migrate SAP Business Suite and SAP NetWeaver BW on traditional databases to SAP HANA database There are two methods to migrate an existing SAP system to run on SAP HANA shown in Figure 13. The one is the classical method (that is, the heterogeneous system copy using the classical migration tools software provisioning manager 1.0 and R3load) and the other one is the Database Migration Option (DMO) of the Software Update Manager (SUM) method. Both are valid and proved options.

40 40 Figure 13 SAP HANA migration methods for BW and Business Suite (example shows BS) Classical migration method The classical migration of a SAP system is documented for example at Best Practice Guide - Classical Migration of SAP NetWeaver AS ABAP to SAP HANA: Mainly it is a three step approach comprising of: Step 1: Upgrade to the level of HANA support (Application Version, Support Package and Kernel level) accordingly to the SAP HANA documentation or consulting. Step2: Execute Unicode Conversion if the source database is configured as Non-Unicode. Step3: Execute DB/OS Migration with SAP's standard tool (software provisioning manager 1.0 and R3load) by certified OS/DB Migration consultant. Database Migration Options (DMO) method in the SAP Software Upgrade Manager (SUM) Please follow the SAP Documentation. Good SAP documentation resources can be found through: Up-to-date information: SAP note for DMO for SAP Business Suite systems (Suite-on-HANA, including ERP, CRM etc.) SAP note for DMO for SAP BW systemshttp://service.sap.com/sap/support/notes/ Traditional database to SAP HANA database This section will refer to the overview of the migration method from the stand-alone and traditional database like Oracle, MS SQL and DB2 to a SAP HANA database. Database specifics related to the compatibility between the existing database and SAP HANA have to be taken into account. These comprise of:

41 41 Supported Platform Supported Languages and Code Pages Supported Function Also, Please see the "SAP HANA SQL and System Views Reference" in detail at: m?current_toc=/en/2e/1ef8b4f e55d4c127b/plain.htm&show_children=true Traditional Migration method: Export/Import From a high level the process consists of three steps: Step 1: Export the database contents. Step 2: Perform Language and Code Pages conversion (e.g. from ASCII to Unicode) if required Step 3: Import database contents to SAP HANA Please see the SAP Document: SAP HANA SQL and System Views Reference -> SQL Reference -> SQL Statements -> IMPORT, IMPORT FROM section. ETL (Extract, Transform, and Load) tool method There are some ETL products supporting the migration from a traditional database to SAP HANA. Examples are: SAP SLT (SAP Landscape Transformation server), SAP BODS (Business Object Data Services) and Sybase Replication Server. SAP HANA database to SAP HANA database Currently, SAP HANA is supported by two different platform architectures. These are Intel architecture and POWER architecture. The current implementation on Intel is using little endian and on Linux on POWER big endian mode. There are three major methods to migrate the database content from one SAP HANA database to another: SAP HANA Export/Import method using SAP standard tool like as sapinst SAP HANA Backup/Restore method using HANA Studio SAP HANA system replication method (There are options to have a zero downtime migration.) And these migration techniques could be used properly according to the situation like as follows accordingly to the existing SAP HANA documentation: Table 2 SAP HANA database migration options From / To Endian Method Notes Intel to Power Little to Big SAP heterogeneous system copy or SUM with DMO Power to Power Big to Big Backup/Restore System Replication XFS on Power to GPFS Big to Big Backup/Restore

42 42 on Power Scale up on Power to Scale out on Power Big to Big Copy Backup/Restore Copy SAP s tool Read the SAP Server Installation and Master Guides for BW on for initial information and further guidance. Backup This section gives a short overview about the basics of backup and recovery for SAP HANA systems. The tools to take a backup range from manual triggered mechanisms such as HANA studio backup wizard and SQL commands to scheduled operations from DBS Cockpit, con, HANA XS (xscron) or 3 rd party tools. Note: this chapter is only a summary and no guide to setup the backup infrastructure. It outlines important thoughts to take based on questions asked by other customers. Additional information can be added writing an to the IBM SAP International Competence Center (ISICC) info service: isicc@de.ibm.com. Additional recommended documentation Backup and Recovery Concepts Training: Data Backup File system Backint Storage Snapshot Advantage Data corruption protection (checksum) Integrated into Studio Backups immediately available for recovery Tooling included in SAP software Fastest Backup to resume from Storage level no network traffic Partially integrated Disadvantage No flat files included Monitoring of file system fill level No direct recovery Network load Consistency check on Block level Integrated into Studio Datacenter integration Features: encryption, de-dupe, Backups immediately available for recovery Vendors often provide Landscape backup options including flat files Network load No flat files included No OS level consistency check No direct recovery Volume Payload only Payload only Full volume size requirement (compression possible) Duration I/O and network bound I/O and network bound Instant (background) Contrary to the manually triggered data backup the log backup is done automatically after the data backup. The related SAP tunable with the default values are: log_mode = normal

43 43 enable_auto_log_backup = true log_backup_timeout_s = 900 log_mode = overwrite Operating System Backup The Linux OS must be backed up in addition to the SAP HANA database backup. This is a less frequent task as the Linux OS is not changed much. For example, updating Linux kernel, applying security patches. This method can be used to backup all flat files related to SAP HANA (executables, HANA Client, etc.), that aren t covered with SAP HANA database backup function (full system backup). A current full system backup will ensure to have a baseline to recover a crashed/destroyed faster than a manual install. Database backups and/or SAP HANA System Replication will complete the backup strategy. Note: Ensure the chosen method can handle files on IBM Spectrum Scale (GPFS) if used. In the following two options are explained. Other tools and methods can be used. Linux native method Boot the system into the rescue mode. Execute dump/restore or tar command( they're the standard Linux native commands).the backup files can be written to internal disks, tape, or shared file systems (NFS). We recommend saving files to an external system or environment. For more details see the Linux vendor specific documentation and SAP notes if applicable. In addition, if migrating the file system for SAP HANA database from XFS to clustered file system, or vice versa, you can migrate files using tar command. When using the Linux native method, SAP HANA database and related processes have to be stopped. The Linux native method doesn t cost extra, but is slightly more complicated than 3rd party software. 3rd party software method You can backup your HANA Operating System by 3rd party software. As one example, "Storix Backup Administrator" supports Bare-metal backup solution for Power Linux systems. "Storix Backup Administrator" works with Tivoli Storage Manager for flexible, central-managed backup/restore. More detailed information is documented at the Solution Brief from IBM: There are more tools available not listed here providing same functionality and comfort. For example, using PowerVC captured systems can be redeployed in minutes. HANA File System Backup With SAP HANA three different backup options are provided: Online Backup: SAP HANA provides database backup and restore function natively in the SAP HANA Administration tool based on file system copies. These copies have to be backed up by the standard backup tools to a remote destination regularly.

44 44 Offline Backup: Offline backups can be performed along a full system backup as described in the section Contrary to the manually triggered data backup the log backup is done automatically after the data backup. The related SAP tunable with the default values are: log_mode = normal enable_auto_log_backup = true log_backup_timeout_s = 900 log_mode = overwrite Operating System Backup. If you use IBM GPFS (General Parallel File System) for SAP HANA data files, logs, Kernel, etc., you can do file level backup using GPFS function (mmbackup command) with Tivoli Storage Manager. In addition, mmbackupconfig command will be useful for backup the GPFS configuration. For more detailed information, refer to the IBM GPFS Administration guide: HANA database Backup and Recovery You can get the SAP HANA database backup through the SAP HANA Studio or SQL interface. And DBA cockpit which is SAP NetWeaver ABAP stack function is also available in case of Suite on HANA or BW on HANA scenario. The backup files are saved to a defined staging area that might be on the internal disks, an external NFS shared disk, or a directly attached SAN storage. In addition to the data backup files, the SAP HANA configuration files (.ini files) have to be saved. For point in time recovery, the log area also has to be backed up. More detailed information about the backup and recovery processes for the SAP HANA database is provided in the SAP HANA Backup and Recovery Guide, available online at the following site: HANA Snapshots SAP HANA supports the creation of storage snapshots. Storage snapshots are created in the storage system and do not consume database resources. Furthermore they run in the background which allows the database to quickly resume writing its persistency. So snapshots can be created with minimal impact on the system. Step 1: Prepare the database for the storage snapshot using SAP HANA Studio or SQL command. It persists all finalized content to the disk and freezes the disk to create a copy. Step 2: Create a snapshot of the SAP HANA data area. FlashCopy on IBM Storage Systems The advantage is the time the data volume has to be freezed. The FlashCopy can be run in background. By that the data area can be instantly released (Step 3) to continue writing which will reduce the exposure from data loss. The disadvantage is that maintaining and enumerating the snaps is within the customer responsibility unless a tool serving the SAP Backint API is used (see HANA Backint Integration ). More detailed information about FlashCopy is provided in the storage user's guide. A

45 45 documented Solution can be found on 03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP GPFS snapshot If you use IBM GPFS, you can get storage snapshot. For more detailed information, refer to the IBM GPFS Administration guide. Other snapshot technology Step 3: To resume the write operations to the data area choose "confirm" or "abandon". In case of a SAP HANA file system backup the storage snapshots are listed in the backup catalog of SAP HANA. You can either display the backup catalog in SAP HANA studio, or SQL command (system view M_BACKUP_CATALOG). IBM provides following solution to optimize this process: IBM FlashCopy backup solution for SAP HANA TDI. Recovery A SAP HANA database can be recovered using GPFS, a native HANA file system backup, storage snapshots or full system backups. SAP HANA database provide the following three kinds of recovery scenarios (as of SPS9 functional scope). Recover up to the latest state Point-in-Time recovery Recovery to a specific data backup <more option will be provided over time> More detailed information about the backup and recovery processes for the SAP HANA database is provided in the SAP HANA Backup and Recovery Guide, available online at the following site: HANA Backint Integration SAP HANA provides the Backint API for third-party backup tools. The tool can backup the data through BACKINT and additional flat files on top. The third-party backup agent runs on the SAP HANA server and SAP HANA communicates with the third-party backup agent through the Backint interface. Currently, SAP HANA does not provide the backup compression function. However, thirdparty backup agents and servers will be able to compress the backup data. IBM provides the IBM Tivoli Storage Manager (TSM) supporting SAP HANA BACKINT. The documentation can be found on:

46 46 High Availability and Disaster Recovery Besides backup and restore protecting data from a disaster SAP HANA can be made highly available suiting individual RTO and RPO requirements. Besides SAP HANA internal functionality also a vital eco system exists for HA automation and DR optimization. All SAP HANA HA and DR functionality can be used directly for Power deployments. This includes Auto-Host-Failover for scale-up and later for scale-out deployments as well as SAP HANA System Replication. The RTO of a SAP HANA System replication can be improved using Cluster Manager. IBM has enabled SA MP to manage the SAP Landscape across operating systems. SA MP provides the capabilities to manage seamlessly Appservers residing on AIX or non-power Hardware. Looking into IBM features, such scenarios as DLPAR operations (which can lower the TCO dramatically) can be better managed using IBM Software and Hardware stacks. For pure data protection SAP HANA supports Storage technologies such as vdisk mirroring, and LUN Mirroring on storage Level. Also the hybrid solution Hyperswap can be used in conjunction with a cluster Manager such as IBM s SA MP. Note: this chapter is only a summary and no guide to deploy HA. It outlines important thoughts to take based on questions asked by other customers. Additional information can be added writing an to the IBM SAP International Competence Center (ISICC) info service: isicc@de.ibm.com. SAP HANA System Replication Setup SAP HANA System Replication with failover automation. Setup Considerations Network SAP documents their view of bandwidth requirements in the SAP HANA guides. This includes Appserver connectivity, Replication network between sites, admin network and for scale-out the Inter node communication.

47 47 Client network (Application Server) SAP HANA clients find the master HANA node by a) a virtual IP which is located on the valid HANA node, or b) a list of master candidates to go through, or c) a combination (one virtual IP per site). Primary Site HANA swarm communication network SAP HANA Application Secondary Site HANA swarm communication network For scale-out HANA nodes require internode communication For scale-out HANA nodes require internode communication SAP HANA System Replication Network The System Replication Network is used to shuffle the changed data from the primary to the secondary site. Figure 14 Schematic Network overview for SAP HANA System Replication deployments Additional recommended documentation SA MP HANA policies Documentation: 2/sampicpart4sapha_HANA.html SAP Note HANA high availability disaster tolerance configuration Tools to consider SA MP HANA policies : Auto-Host-Failover on shared disk using XFS Before planning for Auto-Host-Failover ensure to fully understand the impact of a cold restart of the entire or partial database. Having SAP HANA SR implemented as a Hot standby Solution it might be the better choice to failover to the HA site than restart. Figure 15 From Scale-up to SAP HANA Auto host failover describes the following considerations after the decision has been made to use Auto-Host failover (for scale-up HANA deployments reduce the picture to one master and one standby):