Tuning AIX for Oracle Hyperion and Essbase Products Support documentation for Oracle Service......... Jubal Kohlmeier IBM STG Oracle Applications Enablement November 2013 Copyright IBM Corporation, 2013. All Rights Reserved. All trademarks or registered trademarks mentioned herein are the property of their respective holders
Table of contents Abstract... 1 Introduction... 1 Prerequisites... 1 Tuning AIX with Oracle Essbase products... 1 AIX Libraries... 1 Recommended Environment Variables.... 1 AIX Virtual Memory Manager Settings... 2 LDR_CNTRL settings... 3 64K Page Size Support... 3 Ulimit values... 4 NOLOG on JFS2 file systems... 4 Statistic and Configuration Tools... 5 Specialized AIX Tools... 5 Larger tools... 5 nmon... 6 topas... 6 perfpmr... 6 obimon... 6 obidoc... 6 Other Known Issues... 7 Localhost host name degrades performance... 7 Technote on DNS Lookups... 7 Synopsis:... 7 Summary... 8 About the author... 8 Trademarks and special notices... 9 IBM White paper title here Copyright 2013, IBM Corporation
Abstract This document is comprised of tuning recommendations and helpful tips, tool suggestions for aid in improving the performance of Oracle Essbase products, and for helping Oracle and IBM customers. Introduction Oracle and IBM customers sometime require help in tuning and identifying potential problems with products that execute on Power7 AIX systems. This document was based on a previous suggestion document provided by Oracle. This document has been updated to include AIX 6.1 and AIX 7.1 version. There are three different sections to the paper. They are as follows: AIX and Oracle Hyperion and Essbase tuning recommendations. Recommendations on performance and configuration gathering tools. Issues that have been seen with Oracle Essbase customers that are OS issues that are not directly related to Oracle Essbase. Prerequisites It is assumed that standard AIX installation procedures and recommendations are applied prior to the additional tuning in this document. It is not the intent of this document to bypass either IBM or Oracle standard installation processes. AIX documentation provides you with information regarding tools and manuals. Tuning AIX with Oracle Essbase products This section provides examples of the correct use of subheadings, bullets, figures, code examples, and other elements used in creating white papers. AIX Libraries It is recommended that the version of xlc runtime library be at least version 11.1.0.4 or later. To find the current version type: lslpp l xlc.aix61.rte The version of xlc compiler can be found by the command: xlc -qversion Recommended Environment Variables. It is recommended to set the following variables for Oracle Essbase version 11.x or greater products. Check the values within the current environment using the AIX env command. Copyright 2013, IBM Corporation 1
# within format of ksh environment variables. AIXTHREAD_COND_DEBUG=OFF AIXTHREAD_MUTEX_DEBUG=OFF AIXTHREAD_MNRATIO=1:1 AIXTHREAD_SCOPE=S AIXTHREAD_MUTEX_FAST=ON SPINLOOPTIME=1000 # CC is replaced with the maximum number of physical Cores within the LPAR. MALLOCOPTIONS=multiheap:CC,considersize YIELDLOOPTIME=CC #Where CC is set to the number of physical cores (not logical) export AIXTHREAD_COND_DEBUG export AIXTHREAD_MUTEX_DEBUG export AIXTHREAD_MNRATIO export AIXTHREAD_SCOPE export AIXTHREAD_MUTEX_FAST export SPINLOOPTIME export YIELDLOOPTIME export MALLOCOPTIONS Listing 1: Environment Variables required within the Essbase products. AIX Virtual Memory Manager Settings The table below provides the recommended values for virtual memory management. If upgrading the operating system from AIX 5.3 to either AIX 6 or AIX 7, the recommendations do not carry forward during the upgrade and must be modified manually. You can implement these settings with the AIX vmo command, which is dynamic and takes effect immediately. For example: Copyright 2013, IBM Corporation 2
vmo -p -o maxperm%=90 -o maxclient%=90 -o minperm%=3 -o lru_file_repage=0 LDR_CNTRL settings Ensure AIX shared libraries have "read-other" permissions so that AIX will load them in global memory. Otherwise concurrent use by multiple processes will require their own CPU cache blocks. Substantial performance gains are possible when libraries are shared globally. export LDR_CNTRL=LOADPUBLIC@IGNOREUNLOAD The environment variable, LDR_CNTRL, provides a means of modifying memory and libraries behavior for applications. It is recommended that LOADPUBLIC and IGNOREUNLOAD bet set. This will improve the performance and use of memory. See the following for description of LOADPUBLIC and IGNOREUNLOAD. The LDR_CNTRL environment variable can be used to control one or more aspects of the system loader behavior. You can specify multiple options with the LDR_CNTRL variable. When specifying the option, separate the options with the '@' sign. An example of specifying multiple options is: LDR_CNTRL=PREREAD_SHLIB@LOADPUBLIC. Specifying the PREREAD_SHLIB option causes entire libraries to be read as soon as they are accessed. With VMM readahead is tuned, a library can be read from the disk and be cached in memory by the time the program starts to access its pages. While this method might use more memory, it might also enhance the performance of programs that use many shared library pages if the access pattern is nonsequential (for example, Catia). Specifying the LOADPUBLIC option directs the system loader to load all modules requested by an application into the global shared library segment. If a module cannot be loaded publicly into the global shared library segment then it is loaded privately for the application. Specifying the IGNOREUNLOAD option prevents the application from unloading libraries. This specification might prevent memory fragmentation and eliminate the overhead incurred when libraries are repeatedly loaded and unloaded. If you do not specify the IGNOREUNLOAD option, you might end up with two data instances of a module if the module was loaded at application load time and the module was then requested to be dynamically loaded and unloaded multiple times. 64K Page Size Support On POWER5+ systems, AIX 5L Version 5.3 with the 5300-04 Recommended Maintenance Package supports a new 64KB page size when running the 64-bit kernel. AIX 5L has rich support around 64KB pages, and 64KB pages are intended to be general purpose. 64KB pages are very easy to use, and it is expected that many applications will see performance benefits when using 64KB pages rather than 4KB pages. No system configuration changes are necessary to enable a system to use 64KB pages. On systems that support 64KB pages, the AIX 5L kernel will automatically configure 64KB pages for the system. 64KB pages are fully pageable, and the size of the pool of 64KB page frames on a system is Copyright 2013, IBM Corporation 3
dynamic and fully managed by AIX 5L. AIX 5L will vary the number of 4KB and 64KB page frames on a system to meet demand on the different page sizes. Both the svmon and vmstat AIX commands can be used to monitor the number of 4KB and 64KB page frames on a system. For forcing the use of 64K pages, the LDR_CNTRL environment variable can be modified with memory settings. export LDR_CNTRL=DATAPSIZE=64K@TEXTPSIZE=64K@STACKSIZE=64K Note: the @ operator separates options within LDR_CNTRL, previous options within the current environment should be preserved when adding additional arguments to LDR_CNTRL. INFORMATIONAL: there is a tradeoff of CPU performance gain versus additional memory usage when applying 64K page sizes. Ensure the customer is willing to account for the additional memory use with 64K pages. Ulimit values It is recommended that the following values for ulimit be in affect for the Essbase application environment. To check the current values type: ulimit a The recommended values for ulimit are: time(seconds) unlimited file(blocks) unlimited data(kbytes) unlimited stack(kbytes) 131072 (128MB) memory(kbytes) unlimited coredump(blocks) unlimited nofiles(descriptors) 4096 threads(per process) unlimited processes(per user) unlimited NOLOG on JFS2 file systems There is the ability to improve performance during data loading by usage of a RAMDISK, or AIX JFS file system with the NOLOG option. You can use NOLOG JFS2 file systems which skip the Journal logging for file allocation/deallocation intensive access. JFS logging and RAID 5 is a deadly combination if you want speed. Usage of NOLOG on non temporary file system is not recommended, as file system recovery will be much longer. The AIX mkramdisk command provides the tool to create a RAMDISK. Copyright 2013, IBM Corporation 4
Statistic and Configuration Tools This section provides support the information on what tools are useful for testing, configuration and statistic gathering. Each section describes the purpose of the tools in question. Specialized AIX Tools The AIX operating system has many tools to help the developer and administrator diagnose and test the performance of the system. This document does not attempt to go in any detail regarding the specific tools, see the AIX documentation for further information. The table below provides a means of specifying what tools are useful depending on the type of object under investigation. Most of the AIX tools listed are character based tools. The exception to the tools below is the AIX topas command, which is curses-based. Larger tools Two major tools come standard with most versions of AIX. These are nmon and topas. The AIX nmon tool became standard with AIX 6.1 and AIX 7.1. Copyright 2013, IBM Corporation 5
nmon nmon it is fast becoming a popular tool supported on AIX. It is also possible to collect nmon performance data daily. Nmon is a free tool and can be found on the internet. topas The other standard tool supported on AIX is the AIX topas command. The AIX topas command reports selected local and remote system statistics. Example of a daily report with topas is: usr/bin/topasrec -L -s 300 -R 1 -r 6 -o /etc/perf/daily/ -ypersistent=1 -O type=bin -ystart_time=17:25:45,feb27,2013 perfpmr IBM provides a data collection utility called "perfpmr". Used by IBM diagnostics teams when asking a customer for data for a customer problem, but it is a very good way to collect a snapshot of a system. It does a great job of archiving the configurations of the machine. This is useful for when reviewing the configuration of a customer s machine. obimon If it appears that an Essbase process is using a lot of CPU resources, run thee Essbase obimon tool. Collecting information about system behavior will enable support to focus on the changes that need to be made. obimon ESSSVR & The Essbase obimon command will start the information collection process. Let it run to completion. Tail the output file if you wish to see information "now". obidoc Run the Essbase obidoc tool with the Essbase user id each time you have a configuration change. It is also helpful to run it before/after a major test. Running the tool helps to generate AIX statistics that may help us determine the root cause. Copyright 2013, IBM Corporation 6
Other Known Issues This section contains issues that have been seen by Essbase customers, but not directly related to the Essbase product. Localhost host name degrades performance Check the customers host name. If the machine thinks its name is "localhost", the cost of reverse DNS authentication among machines might be disrupted, leading to large network delays. It is recommended that the hostname not be set to localhost. Technote on DNS Lookups The technote url is: http://www-01.ibm.com/support/docview.wss?uid=swg21212940, with title, Client on AIX generates excessive IPV6 DNS lookups for new connections. Synopsis: Very slow network traffic on IBM Tivoli Directory Server 5.x or 6.x on an AIX system might occur because the client generates excessive IPV6 DNS lookups for every new connection it makes even if IPV6 is not enabled. Some AIX systems have very slow DNS requests. Solution is to disable the IP6 DNS lookups. Copyright 2013, IBM Corporation 7
Summary The tools and tips recommended help to provide for a performing and robust Oracle Hyperion Essbase environment. It is recommended that customers of Oracle and IBM follow these tips and suggestions when implementing Oracle Hyperion Essbase on IBM AIX operating system. About the author Jubal Kohlmeier is an advisory software engineer in the IBM Systems & Technology Group (STG), Business Systems Division. Jubal has more than 30 years of industry experience in the computing industry and more than 15 years with Oracle products. Jubal has worked with customers with Oracle applications and Oracle databases in achieving high volume mission critical solutions on IBM POWER and System x. Copyright 2013, IBM Corporation 8
Trademarks and special notices Copyright. IBM Corporation 2013. All rights reserved. References in this document to IBM products or services do not imply that IBM intends to make them available in every country. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows Server, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel and Xeon are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. AMD and AMD Opteron are trademarks of Advanced Micro Devices, Inc. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. The information provided in this document is distributed AS IS without any warranty, either express or implied. The information in this document may include technical inaccuracies or typographical errors. All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Information concerning non-ibm products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-ibm list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-ibm products. Questions on the capability of non-ibm products should be addressed to the supplier of those products. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local IBM office or IBM authorized reseller for the full text of the specific Statement of Direction. Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending Copyright 2013, IBM Corporation 9
upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here. Photographs shown are of engineering prototypes. Changes may be incorporated in production models. Any references in this information to non-ibm Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. Copyright 2013, IBM Corporation 10