IBM SAP Technical Brief. Live Partition Migration of SAP Systems Under Load. IBM SAP International Competence Center Walldorf, Germany

Transcription

1 IBM SAP Technical Brief Live Partition Migration of SAP Systems Under Load IBM SAP International Competence Center Walldorf, Germany Version: 1.0 Status: April 2011 Page 1 of 19

2 Preface Edition Notice (April 2011) This is the first non draft edition of this document. Preface & Scope This paper documents the findings of a performance test using the PowerVM Live Partition Mobility feature to migrate an active SAP system at different work-load levels. The test sequence shows the impact of various workload levels on the duration of the partition migration. The setup of the test landscape is described in detail, however the operational aspects and steps required to enable a logical partition for Live Partition Mobility is not covered here. For those and for a more complete introduction into Live Partition Mobility, the reader is recommended to check the IBM PowerVM Live Partition Mobility Redbook (see chapter 6 References). Special Notices Acknowledgments The authors of this document are: Walter Orb, IBM SAP International Competence Center, Germany With special thanks to Daniel Balko from Adolf Würth GmbH & Co. KG for the contribution of a very interesting chart documenting the Live Partition Mobility runtime of dozens of active SAP systems, that were migrated from POWER6 to POWER7 systems during a hardware refresh. Feedback We are interested in any feedback you have. Please send your comments to isicc@de.ibm.com. Disclaimer This document is subject to change without notification and will not comprehensively cover the issues encountered in every customer situation. It should be used only in conjunction with the product literature accompanying the products listed above. The information contained in this document has not been submitted to any formal IBM test and is distributed AS IS. Edition history Version 1.0 = first final release Page 2 of 19

3 Table of Contents 1 LIVE PARTITION MOBILITY INTRODUCTION PREREQUISITES SAP SUPPORT SAP License Keys Flexible License Mechanism DEMO OF LIVE PARTITION MOBILITY WITH RUNNING SAP SYSTEMS POWER6 TO POWER7 MIGRATIONS TEST LANDSCAPE Detailed Hardware Setup Network Setup Software Infrastructure Test Workload Test Scenario TEST RESULTS CUSTOMER EXPERIENCES CONCLUSION REFERENCES COPYRIGHTS AND TRADEMARKS...19 Page 3 of 19

4 1 Live Partition Mobility Introduction Live Partition Mobility (LPM) is a key component of the IBM PowerVM virtualization technology. It is currently available for POWER6 and POWER7 processor based systems and enables the migration of entire logical partitions from one system to another without disrupting the application availability. SAP customers can move on-line SAP application and database services running in AIX or Linux partitions from one physical server to another without having to schedule downtime for their SAP applications. Taking advantage of the PowerVM virtualization technologies, customers typically run a number of different SAP systems in shared processor pool partitions on a single server. This can create an operational challenge to schedule planned downtime to perform system maintenance, as the unavailability of a single server might impact a number of different departments. With Live Partition Mobility one can vacate a complete server while keeping all SAP systems up and running, perform the required maintenance tasks and then move the systems back to the updated server. With LPM customers can eliminate the need to stop SAP applications to: Re-balance the distribution of partitions over multiple servers to adapt to changes in capacity requirements of individual SAP systems Reduce planned down time by dynamically moving applications from one server to another to vacate complete servers to: Perform disruptive firmware upgrades Perform hardware maintenance or upgrades Move to new servers during a hardware refresh, e.g. move partitions from POWER6 to POWER7 servers Consolidate workloads on fewer servers and power off unused servers, thus reducing the energy consumption A migration transfers the complete system environment. This includes the complete memory content, processor states, attached devices, and network sessions to connected users or other systems. The LPM process tries to establish a complete clone image of the memory content on a remote server and at the end of the process there is a short period of time where processing is suspended on the source server. This is required to perform some critical synchronizations, before the partition is resumed on the destination server again. To an active SAP end user this will mostly look as a temporary increase in response time (you would see the hour glass in a SAPGUI session), but running transactions continue seamlessly after the switch to the target server. Live Partition Mobility is not an alternative for high availability solutions for a number of reasons, for example: Live Partition Mobility requires manual intervention. The HMC that governs the migration might not have access to a malfunctioning LPAR and thus cannot launch the transfer. LPAR definitions are stored on the physical server, and those settings cannot be transferred if the server is not available. Page 4 of 19

5 1.1 Prerequisites PowerVM Live Partition Mobility is currently available on POWER6 and POWER7 processor-based servers. AIX 5.3, AIX 6.1, AIX 7.1, and Linux on POWER are supported. The OS and all other data of the mobile partition must reside on external storage subsystems. There must be at least one Virtual I/O Server (VIOS) active on the system and the mobile partition cannot use any physical adapter during the migration. Typical customer implementations use a dual VIOS configuration for redundancy. LPM will keep the same configuration when moving a partition to another server (presuming that the target server is also running with a dual VIOS setup). Requirements, setup, and operations including advanced migrations scenarios as for example the migration of partitions with dedicated I/O adapters are covered in detail in the product documentation and the IBM PowerVM Live Partition Mobility Redbook (see chapter 6 References). 1.2 SAP Support SAP-specific recommendations and requirements are documented in SAP note Live Partition Mobility is transparent to SAP applications and there are no code changes required to use this feature. However one should check the above SAP note for the minimum database levels that are supported for LPM SAP License Keys An SAP specific issue is that SAP license keys (for ABAP and Java systems) are based on a hardware key of the partition that is currently hosting the SAP message server. This hardware key changes after moving an SAP application server instance with a message server to another physical system. The hardware key is only read during startup of the SAP instance and then buffered in local memory. As the message server stays up and running during the migration, any subsequent logins will be checked against the old hardware key and will proceed without a problem. However the next time the message server is restarted the new hardware key is read and user logins may fail, unless a valid SAP license based on the new hardware key is installed. To avoid any problems regarding this license check, it is recommended to install the additional license keys for all potential target servers in the system landscape in advance. This is similar to what one would have to do for high-availability cluster setups and SAP allows to request license keys for different hardware keys (check SAP notes and ) Flexible License Mechanism To simplify the administration of SAP license keys in system landscapes where the message server can move between several physical servers, SAP has introduced a new license key feature called Flexible License Mechanism. With that method the license key is no longer tied to the hardware key of the message server. The flexible license mechanism uses a separate ID generator, which creates a unique network ID for each message server. This network ID is hardware independent and therefore it is possible to move the message server to a different host and retain the unique ID. It is possible to configure multiple ID generators to eliminate any potential single point of failures. Please refer to the current SAP NetWeaver documentation for a detailed description and instructions of how to setup this new Flexible License Mechanism: Page 5 of 19

6 SAP NetWeaver 7.0 EHP2 ntent.htm SAP NetWeaver Demo of Live Partition Mobility with running SAP systems The following two videos demonstrate the live migration of running SAP systems between two physical servers. The first demo shows the migration of an SAP system running under load using the Hardware Management Console (HMC) to manage the move. The second video shows the integration of IBM PowerVM technology in the SAP NetWeaver stack allowing to control Live Partition Mobility operations from the SAP Adaptive Computing Controller (ACC). It also highlights some of the PowerVM specific performance monitoring metrics that have been integrated in the operating system monitor in SAP CCMS (OS07n). POWER6 Live Partition Mobility Demo with SAP IBM PowerVM and SAP ACC Integration demo POWER6 to POWER7 Migrations As already mentioned Live Partition Mobility can be used to perform hardware refreshes without taking the application down. For example it is possible to migrate from existing POWER6 processor based servers to new POWER7 systems. In this case, the migrating partition will stay in a POWER6 compatibility mode. This is a partition attribute which can be set in the partition profile. The effect is that after the migration POWER7 specific processor enhancement are disabled, most notably the partition is still running in 2-way SMT (Simultaneous Multi-Threading) mode instead of the new 4-way SMT mode that is available with POWER7 processors. To fully exploit all of the new POWER7 features, a partition will eventually have to be stopped and started in POWER7 compatibility mode, which requires a scheduled downtime window. Please note that you cannot perform a live migration of a partition that is running in POWER7 mode back to a POWER6 system. Should this be necessary, you'd have to either restart the partition in POWER6 compatibility mode or stop the partition and perform an inactive migration. Page 6 of 19

7 2 Test Landscape Figure 1 shows the infrastructure setup used to execute the load tests. Figure 1: SAP Live Partition Mobility hardware setup Detailed Hardware Setup As shown in Figure 1 on page 7 two IBM Power 770 servers were used in this test. Each server was equipped with two POWER7 6-core processors (12-cores total) running at 3.5 GHz and 128 GB Memory. Table 1 shows the logical partition layout for both servers. Page 7 of 19

8 Logical Partition Processing Mode # dedicated processors Memory (GB) Running on VIO Server 1 Dedicated 1 4 Each server VIO Server 2 Dedicated 1 4 Each server Load Driver Dedicated 2 16 Server #2 only SAP and DB2 Dedicated 8 64 Migrating between both servers Table 1: Logical Partition Layout The VIO server partitions on each system were configured to use internal SAS disks for the root volume group. The load driver and the SAP/DB2 partitions had their root and data volumes allocated on a IBM DS3000 storage subsystem. The disks were attached to the client partitions using virtual fibre channel adapters. Virtual fibre channel adapters are based on a standard technology called N_Port ID Virtualization (NPIV) and enable client partitions to connect to a storage area network through a physical fibre channel adapter that is defined in a VIO server partition. Please refer to the virtual fibre channel section in the hardware information center and IBM PowerVM Virtualization Managing and Monitoring Redbook for detailed information about the design and configuration of virtual fibre channel adapters (see chapter References on page 17) Network Setup The load driver and the SAP partitions were configured with virtual Ethernet adapters connecting through SEA adapters on the VIO servers to an external Ethernet switch. Each VIO server was configured with a connection to two ports on an integrated Host Ethernet Adapter (HEA). One of the ports was used as the physical adapter associated with the SEA adapter, the second port was configured as a network interface of the VIO server itself (associated with an IP address and the hostname of the VIO server). With this setup the communication path between the load driver and the SAP partition was separated from the communication path between both VIO servers. We didn't use any other special network tuning parameters Software Infrastructure The software versions used for these tests were SAP ERP 6.0 Enhancement Package 4, DB2 Version 9.7, and AIX 6.1. The SAP system was running in a 2-tier configuration (database and all SAP application instances running within the same AIX image), albeit it was configured with multiple SAP application server instances. The SAP application server instances were configured as one central instance providing message/enqueue services and four SAP application server instances using four dialog work-processes and one update workprocess each. The end-user workload was spread over the four SAP application server instances. The main reason for splitting the load over more than one instance was just to allow for faster login of all users in the higher load scenarios (thus reducing the elapsed time to run each work-load scenario). Page 8 of 19

9 2.1.4 Test Workload The application work-load chosen for this study represented a sample set of standard SAP ERP business transactions. Multiple users were simulated on the load driver partition and were distributed over four SAP application server instances. All users executed the same set of predefined transactions in a fixed number of loops Test Scenario To create repeatable and comparable test scenarios, the overall test sequence was standardized as follows: startup database instance startup SAP central and dialog instances run a warm-up workload to fill all application and database buffers for each predefined SAP workload level start workload wait until the workload reaches a steady state (all simulated users are logged in and are executing transactions) start LPM performance data collection start the Live Partition Migration operation for the partition under load wait for completion of the partition migration operation stop LPM performance data collection wait for SAP workload to complete stop SAP dialog and central instances stop database instance We simulated a total of 9 different work-load levels. The first level represents an idle partition, all application servers and the database are started, but no active users are connected to the partition. The CPU utilization at this work-load level was slightly above 0%. At each subsequent work-load level, we added the same amount of additional users to the simulation. The lowest level of simulated users generated a CPU utilization of about 10% and the CPU usage gradually increased with each level until the maximum simulated workload close to 80% CPU utilization. To make the results comparable, the average CPU utilizations were calculated over the duration of the high-load intervals (the interval starts when all simulated users are logged in and ends when the first user has finished its complete set of business transactions). 3 Test Results The first chart in Figure 2 shows the total duration of partition migrations at different workload levels. For each workload level, the first bar shows the total duration of the partition migration operation and the second bar shows the average CPU utilization over the duration of the high-load interval. It took about 980 seconds to migrate an idle partition with 64 GB of memory. Each increase of workload also causes an increase of the time needed for the migration. When migrating a live partition, the LPM framework has to create a complete memory image copy on the target server, before it is able to suspend the partition on the source system, perform the remaining required synchronizations and then resume the partition on the target Page 9 of 19

10 server. To achieve this, the content of the memory pages are captured and copied over a network connection to the target server. In case a memory page that was already send to the target is modified again by the running application, it is put on a dirty list and has to be transferred again at a later stage. LPM repeats this process in multiple phases, trying to reduce the number of pages on the dirty list. Once the number of dirty pages reaches an internal threshold, the processing on the source partition is suspended. At this time, SAP users who submit new transactions will see an hourglass. LPM transfers the remaining dirty pages, performs other necessary synchronizations, and will then resume the partition on the target system. The SAP application instance will resume processing of the transactions in flight and the end-users will get their response after a little delay. With this process it becomes clear that the duration of a partition migration depends on the amount of memory that is configured and on the change rate of memory pages caused by the running applications. If the work-load is changing pages at a rate that is greater than the system is able to copy to the target server, the process will never reach a point where the final synchronization has a chance to succeed and the operation will eventually fail (at this time the partition would just continue to run on the source server). So increasing the number of simulated users will also increase the number of actively used memory pages in the system. Therefore it will take longer to migrate the partition. With our simulated workload scenario, the duration to migrate the partition configured with 64 GB real memory increased from 980 seconds at the idle level to 1720 seconds at workload level of 80% CPU usage. 2, , , , , Seconds 1, CPU Util Workload Level 0 Duration of LPM operation CPU Load Figure 2: Duration of LPM operation at different load levels The chart in Figure 3 shows the overall transfer rates at different work-load levels. The transfer rate is calculated as the memory size of the partition divided by the time needed for the partition migration. With an idle work-load we achieved a transfer rate of about 4 GB per minute. This rate decreased with each workload load level to about 2.2 GB per minute for the maximum load. Although the tests showed that the migration operation worked just fine and for this particular workload succeeded even at fairly high load levels, it is also clear that high system load Page 10 of 19

11 significantly adds to the duration of the migration operation. The general advise is that LPM activities should preferably be scheduled at periods of low activity, especially for systems with very large memory sizes GB/min CPU Util Workload Level 0 Migration Rate CPU Load Figure 3: Migration transfer rates The next chart (Figure 4) shows a comparison of the CPU utilization when running the workload level without performing partition migrations with our migration test scenario. It shows that there is virtually no impact on the CPU usage of the migrating partition itself CPU Util Workload Level without LPM with LPM Figure 4: Comparison of CPU load with and without LPM Page 11 of 19

12 The following chart in Figure 5 shows the throughput in transactions per second for both test scenarios. The first bar shows the throughput at normal operations, the second bar shows the throughput at the same workload level with an active partition migration. One can see that at low utilization levels, there is a only a very small impact on the throughput numbers. The impact becomes more noticeable at the high load levels. These throughput numbers are calculated by the workload simulation tool as the total number of transactions completed within the high-load interval divided by the duration of the highload interval. If one would compare the actual throughput numbers at specific times within this interval, one would see that there is only a small difference. The averages over the complete interval are different because during the LPM migration, the partition is suspended at a certain point in time. The load driver will continue to send user transactions during that phase. When the partition resumes its operation on the target system, the transactions that were sent during the suspend phase will be queued up on the application server. The SAP application instances will take a while to empty those queues, during that time the driver will see longer response times. Thus the duration of the high-load interval is longer and as the number of transactions that have to be processed within the high-load interval is virtually constant, the average throughput numbers will decrease Transactions / sec Workload Level without LPM with LPM Figure 5: Comparison of average throughput with and without LPM As explained above a Live Partition Migration will lead to a longer duration of the high-load interval. The chart in Figure 6 shows the duration of the high-load interval at the different workload level with and without an active LPM operation. This difference is basically the length of the suspend/resume phase and the amount of time the partition needs to fully recover from the stacked up queues after resuming its processing. This difference is very small at the lower utilization levels, but it takes almost eight minutes to fully recover from the suspend/resume phase when running the workload at the 80% utilization level. Page 12 of 19

13 Duration (min) Workload Level without LPM with LPM Figure 6: Duration of high-load interval The next three charts (Figure 7) show the CPU utilizations of the SAP partition and the two VIO servers that are involved in the LPM operation. As already explained above, the additional CPU usage on the SAP partition at each load level is caused by the additional workload. The LPM operation doesn't increase the CPU utilization on the migrating partition. Both VIO servers are simultaneously active for the duration of the LPM operation. However its CPU usage doesn't change a lot with the additional load on the SAP partition. Most of the CPU capacity is needed to transfer the copy of the memory image to the target server. The rate at which the VIO servers can transfer this data stays fairly constant, but the amount of time the VIO servers need to reach the suspend/resume phase becomes larger, as the need longer to reduce the amount of pages on the dirty list. The CPU demand on the VIO servers is relatively small during the phases between the LPM operations. Page 13 of 19

14 CPU Total 11/30/2010 User% Sys% Wait% :09 00:25 00:41 00:56 01:13 01:28 01:43 01:58 02:15 02:30 02:45 03:00 03:17 03:32 03:48 04:03 04:19 04:34 04:49 05:04 05:21 05:36 05:51 06:06 06:22 06:37 06:52 07:08 07:24 07:39 07:54 08:10 08:26 08:42 08:57 09:12 09:27 09:42 CPU Total 11/30/2010 User% Sys% Wait% :06 00:21 00:36 00:51 01:06 01:21 01:36 01:51 02:06 02:21 02:36 02:51 03:07 03:22 03:37 03:52 04:07 04:22 04:37 04:52 05:07 05:22 05:37 05:52 06:07 06:22 06:37 06:52 07:07 07:22 07:37 07:52 08:07 08:22 08:37 08:52 09:07 09:22 09:37 CPU Total 11/30/2010 User% Sys% Wait% :06 00:21 00:36 00:51 01:06 01:21 01:36 01:51 02:06 02:21 02:36 02:51 03:06 03:21 03:36 03:51 04:06 04:21 04:36 04:52 05:07 05:22 05:37 05:52 06:07 06:22 06:37 06:52 07:07 07:22 07:37 07:52 08:07 08:22 08:37 08:52 09:07 09:22 09:37 Figure 7: CPU Utilization during the complete test sequence Page 14 of 19

15 4 Customer Experiences The Live Partition Mobility feature became first available in 2007 and in the meantime a lot of customers have started to actively use it as one other step to achieve their continuous availability goals. Customers most commonly exploit the LPM feature to avoid downtime for scheduled hardware or firmware maintenance and for moving running systems to new servers during a hardware refresh. This especially includes the live migration of large production systems. Some customers have presented their experiences on user conferences or documented it on public web sites. The chart in Figure 8 by courtesy of Daniel Balko from Adolf Würth GmbH & Co. KG illustrates the durations of dozens of live migrations of SAP systems with different memory sizes, that were performed during a hardware refresh moving from POWER6 to POWER7 systems. During this hardware refresh all systems were migrated to different subnets and thus had go through additional network routers. Therefore the achieved transfer rates were a bit lower than the typical 2-4 GB/min seen in other cases. Dependent on the system load level and network utilization, systems with the same memory size took a different amount of time for the migration. System 1 is running in a PowerHA cluster configuration. Again one can see that the active node with active user load took a bit longer to migrate than the passive fail-over node with the same memory size. After the move to POWER7 was complete, the customer performed some additional LPM migrations to test the transfer rates within the same subnet. For these tests they achieved transfer rates of about 2.7 GB/min for System 2 and 3.3 GB/min for System 3. Figure 8: LPM Durations at Adolf Würth GmbH & Co. KG Page 15 of 19

16 5 Conclusion The workload tests documented in this paper show that Live Partition Mobility operations succeed even at fairly high system utilization levels. The amount of time it takes to complete a migration is dependent on a number of factors like the memory size of the migrating partition, the amount of memory changes in the partition, and the sustainable network bandwidth between the two VIO servers that perform the migration. Typical migration rates measured in the lab and in real customer live migrations are in the range of 2-4 GB/min. The suspend/resume phase during a migration will impact the application throughput and end-user response times for a few minutes. The overall impact and the amount of time required to reach normal processing levels again increases with the active load on the system. To avoid problems with LPM not being able to perform the switch because of too high memory change rates, it is still recommended to schedule LPM operations during off-peak hours, especially for systems with very large memory sizes. Page 16 of 19

17 6 References Documentation Live Partition Mobility in POWER6 Systems Hardware Information Center topic=/iphc3/iphc3kickoff.htm Live Partition Mobility in POWER7 Systems Hardware Information Center topic=/p7hc3/iphc3kickoff.htm Virtual fibre channel in POWER6 Systems Hardware Information Center topic=/iphat/iphatvfc.htm Redbooks IBM PowerVM Live Partition Mobility (SG ) SAP Applications on PowerVM (SG ) IBM PowerVM Virtualization Managing and Monitoring (SG ) Whitepaper Live Migration of Power Partitions Running SAP Applications IBM Power Systems Live Partition Mobility (LPM) and Oracle DB Single Instance SAP & IBM Power Systems DLPAR and LPM Customer Demo Demo Videos POWER6 Live Partition Mobility Demo with SAP IBM PowerVM and SAP ACC Integration demo Page 17 of 19

18 Other Setting Up Network To Improving Partition Mobility Performance DB2 and the Live Partition Mobility feature of PowerVM on IBM System p using storage area network (SAN) storage Power Systems Virtualization Wiki AIX Wiki JS22 Live Partition Mobility article on IBM developerworks S_TACT=105AGX20&S_CMP=EDU Page 18 of 19

19 7 Copyrights and Trademarks IBM Corporation All rights reserved. References in this document to IBM products or services do not imply that IBM intends to make them available in every country. The following terms are registered trademarks of International Business Machines Corporation in the United States and/or other countries: AIX, AIX/L, AIX/L(logo), DB2, e(logo)server, IBM, IBM(logo), pseries, System/390, z/os, zseries. The following terms are trademarks of International Business Machines Corporation in the United States and/or other countries: Advanced Micro-Partitioning, AIX/L(logo), AIX 5L, DB2 Universal Database, eserver, i5/os, IBM Virtualization Engine, Micro-Partitioning, iseries, POWER, POWER4, POWER4+, POWER5, POWER5+, POWER6, POWER7. A full list of U.S. trademarks owned by IBM may be found at: UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. SAP, the SAP Logo, mysap, R/3, SAP Enterprise Portal and other products are trademarks or registered trademarks of SAP AG in Germany and many other countries. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other company, product or service names may be trademarks or service marks of others. Information is provided "AS IS" without warranty of any kind. Information concerning non-ibm products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-ibm list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-ibm products. Questions on the capability of non-ibm products should be addressed to the supplier of those products. Page 19 of 19