Oracle JD Edwards EnterpriseOne Object Usage Tracking Performance Characterization Using JD Edwards EnterpriseOne Object Usage Tracking ORACLE WHITE PAPER NOVEMBER 2017
Disclaimer The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remain at the sole discretion of Oracle. ORACLE JD EDWARDS ENTERPRISEONE OBJECT USAGE TRACKING PERFORMANCE CHARACTERIZATION
Table of Contents Executive Summary 2 Introduction 3 Object Usage Tracking Concepts 3 Data Collection Techniques 5 Use Cases 5 Results 5 Analysis 11 Conclusion 11 Appendix A: Test Configuration 12 1 ORACLE JD EDWARDS ENTERPRISEONE OBJECT TRACKING PERFORMANCE CHARACTERIZATION
Executive Summary Following the release of Object Usage Tracking in EnterpriseOne Tools 9.2.1.2 for application and batch process tracking and EnterpriseOne Tools 9.2.2 for business function tracking, the Oracle JD Edwards EnterpriseOne development team initiated a project to characterize the performance of interactive applications and batch processes. The purpose of the performance characterization was to demonstrate that there is minimal impact to performance after enabling Object Usage Tracking. The resulting performance characterization concluded that there was no significant impact to interactive applications or batch processes with Object Usage Tracking enabled. Object Usage Tracking has two distinct feature sets: the first is the recording of application and batch processes, and the second is the enablement of the Object Usage Tracking exclusion list. There was a consistent minimal impact of 130ms or less than 0.5% of the total end-to-end server processing time for both interactive and batch processes at all scaling levels of users tested (100/250/500) for object tracking of the application and batch processes. The exclusion list added an additional 65ms to the total end-to-end server processing time. The data presented in this document will isolate each of the performance impacts of these two aspects of the Object Usage Tracking feature of EnterpriseOne. The Oracle development team recommends that Object Usage Tracking remain enabled at all times. The recommendation is that customers follow the best practices for proper sizing of EnterpriseOne database tables that store the Object Usage Tracking information as well as initiate scheduled cleanup of the object tracking detail tables so as to avoid performance issues due to large table sizes. 2 ORACLE JD EDWARDS ENTERPRISEONE OBJECT TRACKING PERFORMANCE CHARACTERIZATION
Introduction Oracle JD Edwards EnterpriseOne is an integrated applications suite of comprehensive enterprise resource planning (ERP) software that combines business value, standards-based technology, and deep industry experience into a business solution. The JD Edwards solution architecture can exist on multiple platforms and on multiple database architectures. This document describes performance characterization testing performed with JD Edwards EnterpriseOne Object Usage Tracking running on an Oracle Linux-based virtual machine. Object Usage Tracking Concepts Object Usage Tracking provides enterprise-wide visibility on JD Edwards interactive applications and batch jobs (UBEs) executed and business function execution, customized, and patched in customer environments. Out of the box object usage analytics helps customers identify modules in use, usage patterns, as well as peak usage and load across environments to give greater insight for usage, performance, issue analysis, and managing compliance. Object Usage Tracking allows the recording of batch and interactive applications (and forms) and business functions accessed by users. Object Usage Tracking stores tracking records in the Object Usage Tracking Detail table (F98911) for interactive and batch processes and Business Object Usage Tracking Detail(F98912) for business functions on the EnterpriseOne Database Server. Object Usage Tracking is disabled by default. You enable Object Usage Tracking through the Working with Object Usage Tracking Configuration (P980042T) application as illustrated in the figure below. Object Usage Tracking Configuration (P980042T) Select the System Level Tracking Enabled check box and enable the DV920 path code as illustrated above. Row Exit for Object Type Tracking The Row exit allows Object Type Tracking at a lower level of control. 3 ORACLE JD EDWARDS ENTERPRISEONE OBJECT TRACKING PERFORMANCE CHARACTERIZATION
Revise Usage Tracking by Object Type Object tracking for the Application (APPL), Business Function (BSFN), and batch (UBE) are enabled as shown in the figure above. Records are not written to the respective tables (F98911 and F98912) when these features are disabled. The Exclusion List screen from the P980042T application Row exit allows for further Object Usage Tracking configuration. Object Exclusion list From this EnterpriseOne application configuration screen, adding the function code object name to the exclude list will prevent them from recording record entries to the EnterpriseOne database. Reducing the number of objects tracked by adding them to the exclusion list allows a limited set of only those business functions that are key to the business and allows a more succinct view of the business processes used in the EnterpriseOne software. Note: You must first enable Object Usage Tracking for the system before enabling the feature at the path code level. An Object Usage Tracking record for application and batch processes is inserted in the F98911 table every time a user enters an interactive application or form and for every submission of a batch job. Object Usage Tracking of business function records for application processes are accumulated in the EnterpriseOne kernel process cache and then inserted into the F98912 from information in the cache to the database server table every 45 minutes for active EnterpriseOne kernel processes. The trigger to the writing of records to the database server table also occurs when the last user exits the individual EnterpriseOne kernel process. Object Usage Tracking of business function records originating from a batch process are inserted into the F98912 database table only when the batch process completes. 4 ORACLE JD EDWARDS ENTERPRISEONE OBJECT TRACKING PERFORMANCE CHARACTERIZATION
The above transactional records of interactive and batch processes and business functions assume that Object Usage Tracking is enabled for those features and that they are not filtered by entries in the exclusion list. The exclusion list only limits the records in the database server for business functions calls (F98912). Data Collection Techniques Appendix A contains a diagram of the EnterpriseOne architecture used for testing and the specific EnterpriseOne component configurations. The items listed below are the metrics and data collection techniques used for testing Object Usage Tracking:» CPU/Memory/Network operating system metric collection on the EnterpriseOne Logic and JAS / HTML servers was performed using scripts with the standard commands of ps, iostat, vmstat, netstat, and sar.» JVM heap memory on the JAS / HTML server collected statistics using the verbose garbage collection (GC). The following arguments to the JVM collect all GC activity and heap size data over time: -Xloggc:<path to log directory>/gc.log -XX:-PrintGCDetails -XX:+PrintGCTimeStamps Verbose GC statistics are an excellent method of Java memory analysis and the impact of this utility on the EnterpriseOne JAS server is negligible, making it an ideal tool for ensuring stability of the EnterpriseOne application and identifying possible memory shortages and memory leaks.» The EnterpriseOne Call Object Kernel process performs Object Usage Tracking of interactive processes. Call Object response times metric collection used the EnterpriseOne Server Manager Console facility included in the EnterpriseOne base product offering.» Response times of all interactive transactions were collected using the Oracle Application Testing Suites (OATS) built-in analysis tools, which provided server end-to-end timing metrics. Use Cases The Oracle EnterpriseOne Day-in-the-Life (DIL) Kit was used for testing EnterpriseOne Object Usage Tracking. The DIL Kit is a set of Oracle internally automated load testing scripts for the purpose of generating various interactive loads in an EnterpriseOne architecture environment. The DIL Kit comprises 25 interactive applications across five functional modules, including Customer Service Management, Finance Management, Human Capital Management, Supplier Relationship Management, and Supply Chain Management. Another component of the DIL Kit is the 1.2 TB foundational database that application and batch processes use, designed to represent an average-sized JD Edwards EnterpriseOne customer. Exercising the DIL Kit for 100, 250, and 500 interactive users included testing for concurrent batch processing and comparing the metrics with the Object Usage Tracking feature enabled. Metrics such as end-to-end response time and operating system metrics (processor, memory, and network) statistics comprised the bulk of the comparisons used for measuring the performance characterizations. Results Discussion 5 ORACLE JD EDWARDS ENTERPRISEONE OBJECT TRACKING PERFORMANCE CHARACTERIZATION
Object Usage Tracking recorded an entry for every instance an interactive application or form was instantiated by the DIL Kit tests. A tracking detail record was also recorded for every batch job process submitted. When enabled, the business function tracking inserted records for the interactive and batch function code executed. The DIL Kit has 31 unique application/form entries that add records to the F98911 table. An individual DIL Kit script may record 1 to 4 records per iteration, which will continue for the testing period of an hour for each use case described above. For example, during DIL Kit testing of one hour, the test recorded 3800 records for a 100 users interactive load and 9700 records for a combined 100 users interactive and batch load, assuming a rate of 10 UBE submissions per minute. Each batch (UBE) process entered a single record into the F98911 table. All of the batch processes tested were single-threaded and did not launch any child batch processes. Unlike the interactive and batch processes adding records for each interactive and batch events in the F98911 table, the business function table (F98912) is a summary of the function code requests as is illustrated in the table below: Thus in the example of a 500 interactive users initiated concurrently with a large UBE load (75 UBE/min) scenario, only 315,000 records in the F98912 were recorded, this however represented over 15 million business function code events. Keeping the number of records smaller in the business function table design addresses any concern to performance with large table growth over time. Applications and Batch Processes Metrics As mentioned in the summary section of this document, the impact of Object Usage Tracking is a delay of approximately 130ms for Object Usage Tracking of the application and batch processes. The below figure shows that this value is observed for each of the use cases tested. 6 ORACLE JD EDWARDS ENTERPRISEONE OBJECT TRACKING PERFORMANCE CHARACTERIZATION
Server end-to-end response times Interactive User Characterization of Object Usage Tracking In isolated interactive user testing (100/250/500 users), there was a consistent baseline measurement of the server end-to-end timing of approximately 25.60 seconds without Object Usage Tracking enabled. When Object Usage Tracking is enabled, the timing of the server end-to-end time rose to an average time of 26.73 seconds or 130ms difference. This baseline and time difference was independent of the scaling of interactive users. Interactive User and Batch Characterization of Object Usage Tracking Similar to the interactive user testing, interactive and batch load testing results were predictable in the number of entries in the F98911 table and overhead to the end-to-end server response times. In the use cases of interactive and batch, a baseline (Object Usage Tracking disabled) of 26.00 and 26.08 seconds was observed for 100/batch and 250/batch, respectively. Enabling Object Usage Tracking yielded a similar 130ms differential. As before, scaling both interactive users with batch yielded the same 130ms result, making the performance impact of the Object Usage Tracking feature independent of load. Operating System Server CPU Profile Application and Batch The illustration of the operating system processor (CPU) is in the figure below. The figure shows the CPU resource consumption for the EnterpriseOne architecture component servers. CPU Usage for EnterpriseOne Components As anticipated for both interactive and interactive with batch testing, the CPU load increases on the Logic and Web Servers as the testing load increases and there was no significant impact to CPU performance between having Object Usage Tracking enabled or disabled. Although there is minor improvement to the CPU for the Logic and Web Server CPU statistics, this is a result of the EnterpriseOne application waiting to write records and not consuming CPU resources. The observed CPU statistics are not an actual performance enhancement, but a downstream effect of the code wait states resulting from the Object Tracking records inserted into the F98911 table. Operating System Server Memory Profile Application and Batch For the operating system memory profiles, illustrated below are the EnterpriseOne component servers. In the case of the JAS Server, the garbage collection (GC) metrics, described later in this document, is a better observational tool for observing memory consumption on the JAS server. The GC metrics measures the memory heap usage of the WebLogic managed instance memory pool. 7 ORACLE JD EDWARDS ENTERPRISEONE OBJECT TRACKING PERFORMANCE CHARACTERIZATION
Memory Usage for the EnterpriseOne Component Servers The illustration above is the memory profiles of the EnterpriseOne components. Similar to the CPU profiles, there is not a significant impact to memory consumption on the component servers with Object Usage Tracking enabled for interactive user scaling and interactive user scaling with batch processing. The EnterpriseOne architecture is based on the Linux operating system. This operating system uses a complex algorithm of memory buffers and cache with regard to memory consumption and reporting of available memory. Small variances in the reporting of memory consumption as was observed above are normal. Object Usage Tracking with Exclusion Lists Server End-to-End Response Time This section of the document illustrates that adding records to an exclusion list created a nominal impact to the endto-end server metric. An additional 64ms was observed between interactive only processes with an exclusion list and 23ms for an interactive and batch process scenario with an exclusion list. 8 ORACLE JD EDWARDS ENTERPRISEONE OBJECT TRACKING PERFORMANCE CHARACTERIZATION
In the testing results above, 617 record entries were added to the database exclusion list table (F98900) through the EnterpriseOne application. In the above example, a sales order business function name OBNM (B4200310) was added to the exclusion list table (F98900) this would prevent all five (F4211ClearWorkFile, F4211FSBeginDoc, F4211FSEditLine, F4211FSEditLinePreProcess, and F4211FSEnddoc) function event names (FCTNM) to be written to the object tracking business function list table (F98912). The table above also illustrates the total number of records for each of the object types (BSFN, APPL, or UBE) in the EnterpriseOne application for scale. Object Usage Tracking with Exclusion Lists Operating System Metrics The CPU usage and memory usage comparison figures below illustrate that the Object Usage Tracking feature using exclusion list operating system metrics were consistent with their baseline counterpart, only encountering a 5% increase in additional CPU usage for the EnterpriseOne Logic and Database servers. CPU Usage Comparison for Object Tracking Exclusion List An exclusion list of 617 records prevented any of the business functions in the DIL kit to be written to the Object Tracking functions table (F98912). The actual number of reasonable records in an exclusion list would likely be much less than this and thus present a nominal impact to the operating system processor consumption. 9 ORACLE JD EDWARDS ENTERPRISEONE OBJECT TRACKING PERFORMANCE CHARACTERIZATION
Memory Usage Comparison for Object Tracking Exclusion List The operating system memory profile was only slightly increased for the 500 interactive and batch processing with an exclusion list. An increase of 2% memory was observed in testing, which represents an increase in the number of connections to the database resources needed to write the records to the F98912 Object Tracking functions table. JAS Server Garbage Collection Profile The garbage collection metrics shown in the figure below illustrate the memory profile of the WebLogic heap usage (shown in blue). The figure shows the memory profile of the WebLogic Java managed instance heap usage. The test is 100 users with the inclusion of batch with Object Usage Tracking enabled. Garbage Collection metrics Without going into details about all aspects of this graph, the profiles demonstrate that no memory issues are observed through the garbage collection profile and a consistent memory usage and freeing of memory is seen that is identical to a similar profile. Tuning Recommendations With just a small impact,130ms for application/batch and an additional 64ms for the exclusion list, to the overall server end-to-end response time, the following is the recommended performance tuning for Object Usage Tracking: EnterpriseOne Datasource System - 920 The F98911 and F98912 tables are located in the System 920 EnterpriseOne data source. The EnterpriseOne System 920 data source is used to mainly read information from the database for normal EnterpriseOne operations. Object Usage Tracking changes the dynamics of this data source to be write operation heavy. With this requirement comes the need for the database administrator to ensure that the default size of the database data 10 ORACLE JD EDWARDS ENTERPRISEONE OBJECT TRACKING PERFORMANCE CHARACTERIZATION
source is large enough to handle the influx of records inserted into the F98911 and F98912 tables. The System 920 data source must also be set to auto-extend if need be; otherwise, tracking information and EnterpriseOne functionality in general might be compromised. There is a performance impact if the database must perform frequent auto-extension of the System 920 data source and therefore it is recommended that a large tablespace size is chosen instead of auto-extension. Purging the Object Usage Tracking Detail Table Oracle recommends that the user follow the documentation for Object Usage Tracking with regard to purging and summarizing detail tracking information for future use. The F98911 and F98912 tables can accumulate a large number of records in a short time interval. Incorporating processes to ensure that this table s growth is checked periodically is a concern of any performance analyst. A few key factors to this performance degradation include the size of F98911/F98912, frequency of inserts, the database s need to enforce indexes and constraints, and the database s need to maintain current table statistics. Of course, performance degradation will vary depending on the customer s use of Object Usage Tracking and the speed of the hardware on the EnterpriseOne Database Server component. Analysis Analysis of the cumulative metrics showed a minimal 130ms for application and batch Object Usage Tracking and an additional 64ms impact for exclusion list performance impact to response time for the server end-to-end response time. Furthermore, CPU/memory did not show any significant impact to warrant any architecture changes to the EnterpriseOne component servers. Conclusion When Object Usage Tracking is enabled, it has minimal impact to the end users, whether from the perspective of scaling interactive users or interactive users with batch. Testing with the Oracle DIL Kit demonstrated the feature impact was consistent throughout all the testing performed and was independent of the scaling of load. The impact of enabling the Object Usage Tracking exclusion list also had negligible impact. The average CPU/memory profiles for application/batch and exclusion list aspects of Object Usage Tracking did not show a concern with regard to the operating system metrics. 11 ORACLE JD EDWARDS ENTERPRISEONE OBJECT TRACKING PERFORMANCE CHARACTERIZATION
Appendix A: Test Configuration Below is the list of the EnterpriseOne component machines that made up the architecture for testing: Machines and Platforms Enterprise Server:» Oracle Enterprise Linux 6» Oracle Database12c (12.1.0.2) 32-bit Client» 4 VCPUs x Intel Xeon CPU E5-2697 @ 2.90 GHz» 12 GB RAM Database Server:» Oracle Enterprise Linux 6» Oracle Database12c (12.1.0.2) Enterprise Edition 64 bit» 8 VCPUs x Intel Xeon CPU E5-2697 @ 2.90GHz» 30 GB RAM WEB / HTML Server:» Oracle Enterprise Linux 6» 6 VCPUs x Intel Xeon CPU E5-2697 @ 2.90 GHz» 16 GB RAM» WebLogic Server 12c (12.1.3); Java JDK (1.8)» Single Managed Instance (4GB Heap Size) Deployment Server:» Windows 2012 R2 Enterprise Edition» 2 VCPUs x Intel Xeon CPU E5-2697 @ 2.90 GHz» 8 GB RAM Server Manager Console:» Oracle Enterprise Linux 6» 1 VCPUs x Intel Xeon CPU E5-2697 @ 2.90 GHz» 132 GB RAM OATS Test Controller: Software» Windows 2012 R2 Enterprise Edition» 8 VCPUs x Intel Xeon CPU E5-2697 2.90 GHz» 32 GB RAM» Oracle Application Testing Suite 12.3.0.1.0.376» JD Edwards EnterpriseOne Application 9.2 with Tools 9.2.1.2 12 ORACLE JD EDWARDS ENTERPRISEONE OBJECT TRACKING PERFORMANCE CHARACTERIZATION
EnterpriseOne Architecture Resources: For more information, see the JD Edwards EnterpriseOne Tools Software Updates guide at: https://docs.oracle.com/cd/e53430_01/eotsu/objectusage.htm#eotsu275 Another useful link for the Object Usage Tracking Functionality is found in this knowledge document entitled: E1: CNC: Object Usage Tracking Functionality Available With Tools Releases 9.2.1.2, 9.2.2.0 (Doc ID 2267215.1) https://mosemp.us.oracle.com/epmos/faces/searchdocdisplay?_adf.ctrlstate=5gz6588xk_4&_afrloop=284139534569970#aref_section32 Additional content on Object Usage Tracking can be found on LearnJDE.com at: https://docs.oracle.com/cd/e84502_01/learnjde/tools_and_technology.html?4 13 ORACLE JD EDWARDS ENTERPRISEONE OBJECT TRACKING PERFORMANCE CHARACTERIZATION
Oracle Corporation, World Headquarters Worldwide Inquiries 500 Oracle Parkway Phone: +1.650.506.7000 Redwood Shores, CA 94065, USA Fax: +1.650.506.7200 CONNECT WITH US blogs.oracle.com/oracle facebook.com/oracle twitter.com/oracle oracle.com Copyright 2017, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only, and the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document, and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group. 0617 Oracle JD Edwards EnterpriseOne Object Tracking Performance Characterization Using JD Edwards EnterpriseOne Object Tracking