Best practices for IBM ILOG CPLEX Optimizer on IBM POWER7 and AIX 7.1

Transcription

1 Best practices for IBM ILOG CPLEX Optimizer on IBM POWER7 and AIX 7.1 Performance optimization and scalability Yoaqing Gao, IBM SWG Compiler Optimization Team Venugopal Kailaikurthi, Jennifer Glover and Beth L. Hoffman, IBM Systems and Technology Group ISV Enablement Laszlo Ladanyi, IBM ILOG Product Team May 2013 Copyright IBM Corporation, 2013

2 Table of contents Abstract...1 Executive summary...1 Introduction...2 Overview of IBM ILOG CPLEX Optimizer...2 Performance aspects...2 Profiling tools... 2 System specification, software used, and topology...4 System specifications... 4 Software... 4 Topology... 5 Test methodology...6 Test workload and ILOG CPLEX Optimizer... 6 Performance concepts in Power Systems...7 The POWER7 processor chip... 7 Logical processors... 7 Cache affinity and processor binding... 7 Memory affinity... 8 Best practices and recommended tuning to optimize ILOG CPLEX Optimizer...8 Configuration 1 baseline configuration... 9 Configuration 2 Large page configuration Configuration 3 tuning memory allocation settings...13 Scalability Final results...16 Conclusion...17 Resources...18 About the authors...19 Trademarks and special notices...20

3 Abstract This paper describes the results of performance optimization and scalability tests and provides best practices for running IBM ILOG CPLEX Optimizer on an IBM POWER7 processor-based server running the IBM AIX 7.1 operating system. IBM conducted tests to evaluate IBM ILOG CPLEX Optimizer in standalone environments to improve performance by leveraging AIX memory management settings, large pages and by taking advantage of IBM POWER processor cache affinity (through processor binding) and memory affinity settings to obtain the best performance. Executive summary This paper describes the methods and settings that influence the performance of the IBM ILOG CPLEX Optimizer software on an IBM POWER7 processor-based server running the IBM AIX 7.1 platform. This paper is intended for businesses that are evaluating the use of or those that are already running ILOG CPLEX Optimizer on IBM Power Systems severs today. The performance optimization and scalability results documented in this paper provide insights into the performance characteristics of ILOG CPLEX Optimizer with the recommended approaches to tune the AIX environment on which the ILOG CPLEX Optimizer is running. An overview of IBM ILOG CPLEX Optimizer software and the system specifications of the test environment are also provided for the benefits of the readers The performance results demonstrated in this white paper show improvements that the authors have observed on a set of test instances in a standalone environment. These improvements are achieved by tuning operating system settings; they do not influence how CPLEX runs, just the execution speed. Note that not every problem instance showed improvements, and some have shown performance degradation. However, for approximately 70% of the instances, the improvements are significant and in the rest of this paper we highlight the best improvements the team has observed. Because the purpose of this paper is to provide tuning recommendations that are known to help at least with some problem classes and these improvements are sufficiently significant, the team recommends that users experiment with these settings to find what works best for their instances. Key findings include: The team observed improvements up to 14% with Barrier, 24% with Primal Simplex, 17% with Dual Simplex and 11% with Mixed Integer Programming (MIP) algorithms in IBM ILOG CPLEX Optimizer over the baseline when using a large page configuration. IBM ILOG CPLEX Optimizer takes advantage of AIX memory optimizations. The team observed up to 25% improved performance with Barrier and 68% with MIP algorithms of IBM ILOG CPLEX Optimizer. The scalability of IBM ILOG CPLEX Optimizer is good. The team tested from 4 cores to 16 cores and observed that the IBM ILOG CPLEX scaling was almost linear. 1

4 Introduction IBM ILOG CPLEX Optimizer is a component of IBM ILOG CPLEX Optimization Studio. Running IBM ILOG CPLEX Optimizer on an IBM Power Systems server with a default AIX operating system environment often does not enable optimal performance. We tested and analyzed the performance of ILOG CPLEX Optimizer on standalone environment to identify the scaling characteristics, variety of AIX memory management configurations and other tunings to determine the best settings. Overview of IBM ILOG CPLEX Optimizer IBM ILOG CPLEX offers C, C++, Java,.NET, and Python libraries that solve linear programming (LP) and related problems. Specifically, it solves linearly or quadratically constrained optimization problems where the objective to be optimized can be expressed as a linear function or a quadratic function. The variables in the model may be declared as continuous or further constrained to take only integer values. CPLEX comes in these forms to meet a wide range of users' needs: The CPLEX Interactive Optimizer is an executable program that can read a problem interactively or from files in certain standard formats, solve the problem, and deliver the solution interactively or into text files. The program consists of the file cplex.exe on Microsoft Windows platforms or cplex on UNIX platforms. The CPLEX Callable Library is a C library that allows the programmer to embed CPLEX optimizers in applications written in C, Visual Basic, Fortran or any other language that can call C functions. The library is provided as a dynamic link library (DLL) on Windows platforms and as an archive (that is.a file with.a extension) on UNIX platforms. ILOG Concert Technology libraries provide an object-oriented interface in C++, Java and.net. The Concert APIs provide modeling classes to be used with the IloCplex solver class. Performance aspects The objective and goal of IBM ILOG CPLEX Optimizer performance testing was to reduce the time taken to solve the problem using POWER7 and AIX 7.1 settings. The key performance aspects involved the collection of relevant data and deep analysis to identify any bottlenecks. During the performance analysis, we experimented with various test scenarios to reduce the time taken to solve the problem, identify bottlenecks, and work to remove the bottlenecks. The details of these test scenarios are explained in the Test methodology section. Profiling tools During the testing of various scenarios, performance reports were collected and analyzed to identify any bottlenecks. As part of performance testing, the following set of various tools were used to capture reports. Major tools used were: tprof: The tprof command reports processor usage for individual programs and the system as a whole. This command is a useful tool for Java, C, C++, or FORTRAN 2

5 programs that might be processor-bound and where the user wants to know which sections of the program are most heavily using the processor. nmon: The nmon tool is designed to use for monitoring and analyzing performance data on AIX and LINUX, specifically processor utilization, memory use, kernel statistics, run queue information, disks I/O rates, transfers, read/write ratios and more. curt: Generates processor utilization report from a trace. The curt command takes an AIX trace file as an input and produces a number of statistics related to processor utilization and process / thread / pthread activity. hpmcount: The hpmcount command provides the execution wall clock time, hardware performance counters information, derived hardware metrics, and resource utilization statistics (obtained from the getrusage() system call) for the application named by command. 3

6 System specification, software used, and topology This section describes the test environment created and used for the performance study, providing details about the hardware and software. The tests were conducted on an IBM POWER7 system and the OS level was AIX 7.1. An AIX 7.1 partition was created using the internal Small Computer System Interface (SCSI) associated with the server. No external storage was used. The benchmarking and performance tuning was done in a standalone configuration with all prerequisite software in a single AIX partition as depicted in Figure 1. The following section describes the system specifications and software configuration details and test topology. System specifications The test environment used one IBM Power Systems server with the following specifications: IBM Power Systems model cores 256 GB of memory Software This section describes the major software components that were used: IBM AIX 7.1 ( ) 64 bit IBM ILOG CPLEX Optimizer

7 Topology The test topology covers ILOG CPLEX Optimizer configured on a single standalone server. The test environment was a single physical system with a single AIX partition created. Input is a data model that represents a problem and output is the optimized solution. IBM ILOG CPLEX Optimizer IBM POWER System Logical Partition with AIX 7.1 SPSS ILOG CPLEX Modeler Optimizer Server Engine Version 15 Solution Result Flat file Data Test Harness Test Harness SPSS Modeler Client API (PSAPI 15) Figure 1: ILOG CPLEX Optimizer configured on a single server topology on a POWER7 and AIX 7.1 partition Main test system: This system is a single POWER7 48-core server with a single AIX 7.1 partition. 5

8 Test methodology The testing methodology focused on reducing the time taken to find the solution for a given problem on a standalone server. The test duration of different test variants depends on the time limit of each test. The main goal was to create an environment that replicated a scoring service from a user perspective and that produced an optimal level of scores for fixed time limit. This was achieved by running ILOG CPLEX Optimizer with a variety of AIX environment settings to identify the settings that produced the best results. This section initially addresses test scenarios experimented with to achieve the optimal performance and later in the section, different test workload parameters used during the course of performance testing are explained. Test workload and ILOG CPLEX Optimizer This section explains the different workloads and different types of ILOG CPLEX Optimizer used to solve the workloads. It also provides details about the test suite used for performance testing of ILOG CPLEX Optimizer. There are three different ILOG CPLEX Optimizers, 1) Barrier: Solves linear programming problems and parallelizes well. 2) Simplex: Solves linear programming problems but runs in single threaded mode. Dual and Primal are two types of simplex optimizers, which solve different types of simplex workloads. 3) MIP: Solves mixed integer programming (MIP) problems and parallelizes well. The team used 50 different problems for each ILOG CPLEX Optimizer type to benchmark and performance tune. The CPLEX Interactive Optimizer executable is used to solve each of these problems. CPLEX reads the problem and generates the solution by invoking the optimizer. For Barrier and Simplex (Dual & Primal), the measurement is the time taken to solve the problem which is called solution time. For MIP problems the measurement is the number of simplex iterations for a given time limit. The problems were available as files in MPS and SAV formats. For more information on these file formats and different Optimizers, refer to For example, the following code is the simple script that the team used for running ILOG CPLEX Optimizer to solve the barrier problems: foreach file (`cat FILELIST`) echo set log barrier.log >! test$$.in echo read $file >> test$$.in echo ch pr lp >> test$$.in echo set adv 0 >> test$$.in 6

9 echo set adv 0 >> test$$.in echo set thread 12 >> test$$.in echo set parallel 1 >> test$$.in echo set lpm 4 >> test$$.in echo set bar cross -1 >> test$$.in echo opt >> test$$.in echo display sol quality >> test$$.in./cplex < test$$.in rm test$$.in end All the barrier problems are listed in the file called FILELIST. CPLEX solves the problems and generates the time taken to solve the problem one by one. Performance concepts in Power Systems This section provides a high-level overview of Power Systems and AIX performance concepts The POWER7 processor chip POWER7 processor chip includes up to eight processor cores on a single-chip module. The POWER7 processor chip is available in configurations with 4, 6, or 8 cores per POWER7 processor chip, as compared to the POWER5 and POWER6, which have 2 cores per chip each. Along with the increased number of cores, the POWER7 processor chip implements simultaneous multithreading (SMT) supporting four hardware threads per core, as compared to the POWER5 and POWER6 that can support only two hardware threads per core each. Each POWER7 processor core supports running in a single threaded mode with one hardware thread, an SMT2 mode with two hardware threads, or an SMT4 mode with four hardware threads. Logical processors Each SMT hardware thread is represented as a logical processor in AIX. When running in SMT4 mode, the operating system will have four logical processors for each dedicated POWER7 processor core assigned to the partition. To gain full benefit from the throughput improvement of SMT, applications need to use all of the SMT threads of the processors. Cache affinity and processor binding The hardware threads for each core of a POWER7 processor share a core-specific cache space. For multithreaded applications where different threads are accessing the same data, it can be advantageous to arrange for those threads to run on the same core. By doing so, the shared data will remain resident in 7

10 the core-specific cache space, as opposed to moving between different private cache spaces in the system. This enhanced cache affinity can have the effect of allowing more efficient utilization of the cache space in the system and reducing the latency of data references. Similarly, the multiple cores on a POWER7 processor share a chip-specific cache space. Again, arranging the software threads that are sharing the data are run on the same POWER7 processor (when the partition spans multiple sockets) might allow more efficient utilization of cache space and reduced data reference latencies. Forcing the threads of an application to run on a specific set of logical processors is done using processor binding, for example by using the AIX execrset command. Processor binding is most effective when different applications (or different instances of the same application) are bound to nonoverlapping sets of logical processors. The bindings need to be individual cores or sets of cores within a single chip, and should avoid binding a single application to cores on different chips. Memory affinity Memory affinity is an approach to allocate memory that is closest to the processor where a thread is executing. IBM POWER processor-based SMP hardware systems contain multiple chip module (MCM), each containing multiple processors and system memory attached to these MCMs. The memory affinity support recognizes the relationship between processors, memory, and MCMs in IBM POWER processor-based systems. The support provides improved performance to some high-performance computing applications. Enabling memory affinity, the application is limited to run on a defined number of processors and the memory is preferentially allocated from memory behind the processor chip. To enable memory affinity, set the MEMORY_AFFINITY environment variable in the application environment. Applications must be bound to processors to use the AIX memory affinity support effectively. Best practices and recommended tuning to optimize ILOG CPLEX Optimizer Performance tuning is dynamic and there are potentially many parameters that can be changed, and arriving at an optimal set needs experimentation. It is always recommended to follow the best practices for an environmental setup. The following section describes the best practice recommendations for running ILOG CPLEX Optimizer on a POWER7 processor-based system running the AIX 7.1 operating system. Initial testing has been done on an ILOG CPLEX Optimizer setup with baseline settings to obtain baseline data, and similar tests were run on an optimized environment. The optimized environment includes memory tuning, processor and memory affinity settings. There are several key AIX environment variables that influence the performance of ILOG CPLEX Optimizer running on AIX 7.1 environment. The following sections explain each of the major AIX environment settings that have been set to provide optimal performance for ILOG CPLEX Optimizer. 8

11 Configuration 1 baseline configuration The baseline test variation was to use the default AIX settings for the ILOG CPLEX Optimizer. No changes were made to AIX settings other than user limits (as set in the following example). So you can very well consider this to be the default configuration, and set all parameters to unlimited. core file size data seg size file size max memory size open files (blocks, -c) unlimited (kbytes, -d) unlimited (blocks, -f) unlimited (kbytes, -m) unlimited (-n) unlimited pipe size (512 bytes, -p) 64 stack size cpu time (kbytes, -s) hard (seconds, -t) unlimited max user processes (-u) virtual memory (kbytes, -v) unlimited In Table1 there is a list of the models for Barrier, Simplex and MIP. These models saw the best performance improvements during the testing. It shows the initial baseline performance measurements before applying any tuning recommendations. Barrier models Baseline results Simplex models Baseline results MIP models Baseline results (seconds) (seconds) (million iterations) 104.sav pre lp MPS lp lp mps mps mps mps mps mps sav mps sav mps mps mps lp mps pre sav Table 1. Initial baseline performance measurements before applying any tuning recommendations 9

12 Configuration 2 Large page configuration In the initial testing, it was observed that ILOG CPLEX Optimizer can take advantage of large pages. The main purpose for large page usage is to improve system performance for high performance computing (HPC) applications or any memory-access-intensive application that uses large amounts of virtual memory. The improvement in system performance stems from the reduction of translation look aside buffer (TLB) misses due to the ability of the TLB to map to a larger virtual memory range. Large pages also improve memory prefetching by eliminating the need to restart prefetch operations on 4 KB boundaries. AIX supports large page usage by both 32-bit and 64-bit applications. For Barrier and Simplex Optimizers, the performance is better if the time taken to generate the solution (solution time) is less. For MIP, the performance is better if the number of iterations in a given time is higher. We have observed performance improvements up to 14% with Barrier, 24% with Primal Simplex, 17% with Dual Simplex and 11% with MIP algorithms with large pages. Figure 1 through Figure 4 list the models that take most advantage of large pages. For more information on large pages and how to configure large pages, refer to the section AIX 7.1 information > Performance management and tuning > Performance management > Memory performance in AIX information center at 10

13 Barrier Improvements with Large Pages Solution time sav 109.mps 130.mps 138.mps Data models Baseline Largepages Figure 1. Barrier improvements with large pages Primal Simplex Improvements with Large Pages Solution time pre 113.lp 138.mps 149.mps Data models Baseline Largepages Figure 2. Primal Simplex improvements with large pages 11

14 Dual Simplex Improvements with Large Pages Solution time mps 106.mps 108.mps Data models 140.sav Baseline Largepages Figure 3. Dual Simplex improvements with large pages MIP Improvements with Large Pages Iterations mps 117.lp 118.mps 119.mps Data models Baseline Largepages Figure 4. MIP improvements with large pages 12

15 Configuration 3 tuning memory allocation settings Initially, tests were run with different MALLOCTYPE (default, watson, and 3.1_64BIT) and MALLOCOPTIONS (multiheap, buckets, and pool) AIX environment setting combinations. Each setting was analyzed and arrived with the final settings of having 512 MB pool size and 32 heaps The (pool:0x ,multiheap:32) combination provided optimal performance. ILOG CPLEX Optimizer is a multithreaded application where multiple heaps are required so that the server process can issue malloc() and free() calls from more than one thread. With a single heap this was not achieved as seen from the baseline result, which was a serious performance bottleneck. With MALLOCOPTIONS set to pool and multiheap, it improved the performance of ILOG CPLEX Optimizer over baseline results. export MALLOCOPTIONS=pool:0x ,multiheap:32 The team tried different combinations of heap sizes and based on the results, it was concluded that 32 heaps was the best setting for ILOG CPLEX Optimizer. For Barrier Optimizer, the performance is better if the time taken to generate the solution (solution time) is less. For MIP, the higher the number of iterations, the performance is better. The team observed up to 25% with Barrier and 68% with MIP algorithms of IBM ILOG CPLEX Optimizer for various problems with the above MALLOCOPTIONS. Figure 5 and Figure 6 list the models that take most advantage of MALLOCOPTIONS pool and multiheap. For more details on malloc pool and multiheap options, refer to the section AIX 7.1 information > Programming for AIX > General Programming Concepts: Writing and Debugging Programs > System Memory Allocation Using the malloc Subsystem in the AIX information center at 13

16 Barrier Improvements with malloc pool and multiheap solution time MPS 131.sav 148.lp 149.pre Data models Baseline Pool&Mul Figure 5. Barrier improvements with malloc pool and mutiheap MIP Improvements with malloc pool and multiheap 150 Iterations lp 119.mps 132.sav 134.mps Data models Baseline Pool&Mul Figure 6. MIP improvements with malloc pool and multiheap 14

17 Scalability We tested the scalability of ILOG CPLEX Optimizer using SMT1 and SMT2 configuration, by varying the number of cores from 4 to 16 and workers from worker threads from 4 to 16. With SMT1 the scalability of IBM ILOG CPLEX Optimizer is good, resulting in very good performance improvement going from 4-core partition to 16-core partition. Figure 8 shows the scalability of two problems for Barrier Optimizer Scalability Solution time mps 106.MPS Figure 7. Scalability of two problems 15

18 Final results Table 2 lists models that are shown in the above bar charts with their performance improvements. Barrier models Tuning Results (seconds) Simplex models (Primal & Dual) Tuning results (seconds) MIP models Tuning results (million iterations) 104.sav pre lp MPS lp lp mps mps mps mps mps mps sav mps sav mps mps mps lp mps pre sav Table 2. Performance improvements 16

19 Conclusion The recommendations covered in this paper have demonstrated a performance boost of different proportions for different models over baseline configuration. For some models we have observed a performance boost of 68% over baseline configuration. ILOG CPLEX Optimizer is a multithreaded application that requires tuning beyond baseline settings. This paper documents baseline test results over memory optimized performance settings. The key tunings recommended are large page settings and memory allocator settings to achieve improved performance for ILOG CPLEX Optimizer. In summary, this exercise of performance optimization and scalability study has shown that IBM ILOG CPLEX Optimizer has been deployed to get best performance and the following key findings: Up to 68% performance improvement over baseline. Good scalability was observed with the increase of worker threads from 4 to 16. ILOG CPLEX Optimizer takes advantage of large pages and memory allocation tuning. Because the team observed varied performance improvements with different tuning and different models, it recommends clients to try each configuration in preproduction environment before applying the settings to production environments. 17

20 Resources The following websites provide useful references to supplement the information contained in this paper: Overview of IBM ILOG CPLEX Optimizer product ibm.com/software/websphere/ilog/ ibm.com/software/in/websphere/ilog/ IBM Power Systems hardware ibm.com/systems/power/hardware/ IBM Power Systems Information Center IBM AIX 7.1 Information Center TES%2FGI htm System memory allocation using the malloc subsystem (AIX online help) gd/mem_place_envir_var.htm Memory allocation mechanisms in AIX ibm.com/developerworks/aix/library/aumemoryallocation/index.html#overview%20of%20default%20allocation%20policy AIX Thread support tunable parameters gd/thread_supp_tun_params.htm nmon tool ibm.com/developerworks/aix/library/au-analyze_aix/ tprof, curt and hpmcount: Information about these tools and their usage can be found at 18

21 About the authors Yoaqing Gao is an IBM senior software engineer in the compiler group. He has been doing research and development for IBM XL C/C++ and Fortran compiler products on IBM POWER, z/architecture, CELL processors and IBM Blue Gene. He is the IBM XL compiler focal point for high performance computing and analytics. Beth L. Hoffman is an IBM Executive IT Specialist and ISV Technology Manager in IBM Systems and Technology Group. She leads optimization projects to improve the performance of ISV analytics solutions and IBM BAO products on IBM platforms. Venugopal Kailaikurthi is an IBM technical consultant in the STG ISV Solutions Enablement organization focusing on the enablement of the IBM Software Group BAO products on IBM Power Systems platforms. He specializes in testing IBM PowerVM virtualization with IBM Software Group products. Jennifer Glover is Smart Analytics ISV Program Manager. She leads optimization projects to improve the performance of ISV analytics solutions and IBM BAO products on IBM platforms. Laszlo Ladanyi is Software developer and works for the R&D team in the CPLEX group. 19

22 Trademarks and special notices Copyright IBM Corporation All rights Reserved. References in this document to IBM products or services do not imply that IBM intends to make them available in every country. IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. SET and the SET Logo are trademarks owned by SET Secure Electronic Transaction LLC. Other company, product, or service names may be trademarks or service marks of others. Information is provided "AS IS" without warranty of any kind. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Information concerning non-ibm products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-ibm list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-ibm products. Questions on the capability of non-ibm products should be addressed to the supplier of those products. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller for the full text of the specific Statement of Direction. Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is 20

23 presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here. Photographs shown are of engineering prototypes. Changes may be incorporated in production models. Any references in this information to non-ibm websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk. 21