An Oracle White Paper October Benchmarking Oracle Real-Time Decisions 3.2 on Exalytics Hardware

Transcription

1 An Oracle White Paper October 2014 Benchmarking Oracle Real-Time Decisions 3.2 on Exalytics Hardware 1

2 Contents Executive Overview... 3 Benchmark Objectives... 4 Product Overview... 4 Oracle RTD Decisioning Process Framework... 4 Oracle RTD Decision Management... 4 Oracle Decision Manager... 5 Oracle RTD for Marketing Optimization... 5 Learning Graph... 5 Benchmark Scenario Summary... 6 Business User Scenario... 6 Benchmark Setup... 6 Benchmark Details... 7 Summary of Benchmark Results... 7 Benchmark Results... 7 Throughput... 7 Response Times... 8 Learning Record and Model Sizes... 8 CPU Utilisation Conclusion Appendix A: Configuration RTD Database Exalytics CPU Speed RTD Platform Logging Performance Logging RTD Call Forwarding NUMA Node Affinity JRockit Decision Servers Learning Server Appendix B: Inline Service Appendix C: Data Choice Data Customer Data Appendix D: Oracle Load Testing Configuration Available Ports Load Testing Scenario

3 Executive Overview Oracle Real-Time Decisions (RTD) is a highly scalable Enterprise Decision engine that combines closed-loop predictive analytics automation, business rules and optimization to automate high-volume operational decisions, allowing organizations of all sizes to deeply embed analytics in their processes and push better decisions to the front line. For more information see As they plan to integrate RTD to provide cross-channel Decision Service capabilities to their operational business processes, companies need to understand the performance characteristics of this platform. Oracle recognizes the importance of characterizing the performance of any enterprise system, and in particular of systems like RTD that provide services to mission critical operational systems. In this document we discuss throughput related performance metrics for the Oracle Real- Time Decisions platform to help the reader understand how many requests per second a server farm can process given a specific RTD Decision Logic. We describe various aspects of the overall performance of RTD Decisions on the basis of an example that exercises many significant features of the product. A strong emphasis of the test was to demonstrate the scalability of the RTD Learning Server running on Exalytics hardware when updating multiple models for a given session while serving approximately ten thousand concurrent customer sessions on the front-end. This test exercises a feature introduced in RTD called RTD Learning Graph which permits the association of a business event (say a click on an offer) with multiple predictive models linked to each other by way of a graph. With this capability, RTD can "learn" the characteristics associated with a click on a specific offer as well as the characteristics associated with a click on the specific creative associated with that offer presented or with a specific placement in your CRM system. The benchmarking tests have been run with the hypothesis that 8 such models would be updated for every single business event from a total universe of 800 candidate outcomes. 3

4 Benchmark Objectives The objective of this benchmark is to highlight how the RTD Base Marketing Inline Service supports high decision rates on Oracle Exalytics Hardware. An Exalytics X3-4 server was used to run eight Decision Servers and a Learning Server, under RTD 3.2, WebLogic Server and JRockit. Oracle Application Testing Suite was used to simulate a large customer population invoking RTD for recommendations and supplying feedback. Product Overview Oracle RTD Decisioning Process Framework The heart of Oracle RTD is a decisioning process framework that takes into account the overall performance goals which an organization would like to optimize, the performance metrics that measure those goals, the action required to score each of the available choices, and a weighting of those scores based on segments of the population. This is illustrated in Figure 1. Figure 1. The RTD Decisioning Process Business organizations can use Oracle RTD to decide the best products to offer their customers, by different customer segments and demographics. Oracle RTD Decision Management Oracle RTD Decision Management extends the Oracle RTD decisioning process framework, by extending the scope of the decision framework itself and by providing an application development environment for creating business user friendly applications with which to manage the lifecycle of choices available for recommendation. 4

5 Oracle Decision Manager Oracle Decision Manager is an end-user oriented web tool that allows business users to control and manage their choices in external choice repositories. Business users can read, create, update and delete choices and choice rules. When these choices are fed back into the external choice repositories, they become available to the other applications and transactions that use Oracle RTD for choice recommendations. Decision Manager also enables users to access select Decision Center analytic reports on their choices directly from the Decision Manager interface. Using the Oracle JDeveloper platform and through definition of metadata, the Decision Manager interface can be configured specifically to each organization's particular types of choices. Oracle RTD for Marketing Optimization A reference configuration of Decision Manager, called Oracle RTD for Marketing Optimization, and its associated Inline Service, RTD Base Marketing, serve as an application for any marketing organization that is interested in managing their marketing objects. Learning Graph This key decision framework enhancement introduces the ability to define explicit relationships between choices beyond hierarchical relationships. This enables related choices to be actioned and processed together in a way that supports customer workflow operations and procedures. For example, a marketing organization could have planned their marketing campaigns to be publicized on a variety of external media outlets, and have a number of possible choices to make as to what to recommend, to whom, and how. The choices could be categorized under the general headings of campaigns, offers, channels, placements, and so on. These types of choices could be treated as independent, standalone categories, but that could lead to marketing opportunities being lost due to missing connections between objects during the business intelligence gathering process. Oracle RTD Decision Management enables relationships to be defined between the choice categories, as shown in Figure 2 (the basis for the Oracle RTD for Marketing Optimization reference application). Figure 2. Relationships Between Choice Categories in Oracle RTD for Marketing Optimization 5

6 The benefits of inter-choice relationships in the decisioning process include: Enhanced rule evaluation - how choice eligibility is determined. For example, for a creative in a campaign, if the campaign is deemed ineligible for a particular customer, the creative will also be ineligible for this customer due to the propagation of eligibility defined at the campaign-creative relationship level. Enhanced event propagation - how the success and failure of recommendations are fed back into the Oracle RTD predictive models. For example, if Oracle RTD recommends a creative, and the creative was presented to the customer, this presentation event will be recorded: o o For that particular creative in the predictive model associated with creatives For the offer and campaign in which the creative was presented, in the predictive models associated with offers and campaigns Benchmark Scenario Summary Business User Scenario The benchmark scenario consists of the following typical call sequence to RTD. The calls are part of the RTD Base Marketing Inline Service as supplied with Oracle RTD Applications. Session Start is called to start the customer s session with RTD. Get Creative is called to obtain a creative recommendation for the customer. The decision is made from 800 creatives according to a slot ID selected at random and passed as a parameter to the call. In 5% of cases, Creative Feedback is called to record that the customer clicked on the creative. In 20% of cases where the customer clicked on the creative, Creative Feedback is again called to record that they subsequently confirmed the creative. Session Resolution is called to end the RTD session. Benchmark Setup An Exalytics X3-4 server hosted eight RTD 3.2 Decision Servers and a Learning Server, on WebLogic Server running JRockit R The Exalytics server had four ten-core Xeon E processors, with each core having two virtual execution threads giving 80 logical cores in total, and 2 TB of RAM. A Windows 2008 R2 server was also used in the test, with 12 Intel Xeon X GHz cores and 144 GB RAM. Hyper-threading was turned on giving 24 logical cores. This server ran three Oracle 11g R2 databases, one holding the RTD schema, one holding customer and choice data used by the Inline Service and one being the Oracle Load Testing database (part of Oracle Application Test Suite ). The Windows server also ran the OLT Java components. The setup is illustrated in Figure 3. 6

7 RTD Database DS1 DS2 DS3 DS4 DS5 DS6 Customer and Choice Database DS7 DS8 LS OLT OLT Server OLT Agents Admin Server Windows 2008 R2 Server Exalytics X3-4 Server Figure 3. Test Configuration Benchmark Details An OpenScript script integrated with the RTD Java Smart Client replicated the user scenario. Oracle Load Testing was used to run 17,000 virtual users, each iterating through the test script. Every iteration of the script used a customer ID chosen at random from a population of 5,000,000 customers, with random think times between 1 and 14 seconds. Load balancing across the Decision Servers was achieved programmatically. Within a particular iteration of the script, the Java Smart Client sent all its requests to a Decision Server selected on a round-robin basis when the script started. The test started with a ramp-up period, as the load built to its maximum level, after which the system settled into a steady state. The results presented here are from 18 hours of steady state running. Summary of Benchmark Results An Exalytics X3-4 server running 8 RTD 3.2 Decision Servers and a Learning Server comfortably supported 10,978 concurrent RTD sessions 2,160 requests per seconds, two thirds of which are informants and one third advisors 706 decisions per second, with 800 eligible choices per decision a 72 ms decision response time processing of up to 1,000 learning records per second The Learning Server applied the learning records generated to eight learning models and a statistics model in real-time. Benchmark Results Throughput Table 1 shows the rate at which requests of all types were made to the Decision Servers, taken from the individual Decision Server performance logs and totalled. Also shown are the rate of decisions 7

8 made, assuming that on average 1 out of every 3.06 requests is a Get Creative advisor call, and the number of concurrent RTD sessions, again totalled from the Decision Server performance logs. TABLE 1. RTD REQUEST THROUGHPUT REQUESTS/SECOND DECISIONS/SECOND CONCURRENT RTD SESSIONS 2, ,978 Response Times Table 2 shows the average response time for each RTD request type, as measured by the Oracle Load Testing agent, across the entire test period. These end-to-end times include Java Smart Client processing and network latency. TABLE 2. END-TO-END RESPONSE TIMES REQUEST AVERAGE RESPONSE TIME/MS Session Start 12 Get Creative 72 Creative Feedback (Clicked) 10 Creative Feedback (Converted) 10 Session Resolution 10 Figure 4 shows how the response time for the Get Creative request varied with time as measured by the Oracle Load Testing agent. Figure 4. Variation of Get Creative Response Time with Time Learning Record and Model Sizes Learning records were generated at the rate of 706 records per second. Learning record sizes are shown in Table 3, queried from the SDLEARNING table during the test. TABLE 3. LEARNING RECORD SIZES MIN SIZE/BYTES MAX SIZE/BYTES AVERAGE SIZE/BYTES 849 2,486 1,083 Figures 5 and 6 show the learning record backlog and the rate at which the Learning Server processes learning records, taken from the Learning Server log. During learning model persistence 8

9 (the larger peaks), which occurs every two hours, no learning records are processed; during this time a backlog of unprocessed learning records builds up. Once the models have been persisted, the Learning Server works through the backlog until a steady state is one again achieved (in which the rate of learning record creation equals the rate of processing). The same effect is observed during prediction model persistence (the smaller peaks), which occurs when the number of learning records processed has increased by 20% over that at the time of the last persistence. Since persistence models are smaller than learning models, a smaller backlog of learning records accumulates. Consequently learning records are processed at one of two distinct rates during the run. The average learning record processing rate during the steady state is 706 records per second, the same rate at which learning records are generated. This rises to 1,000 records per second while reducing a backlog built up during model persistence. Figure 5. Learning Record Backlog Figure 6. Rate of Learning Record Processing Table 4 shows typical learning model sizes for the current time window, both compressed (as stored in the database) and uncompressed (as held in memory in the Learning Server), taken from the Learning Server log. TABLE 4. LEARNING MODEL SIZES LEARNING MODEL SIZE/KB (COMPRESSED) SIZE/KB (UNCOMPRESSED) CreativeAcceptance 224,939 1,108,842 OfferAcceptance 21,182 55,570 TagAcceptance 6,417 14,173 CampaignAcceptance 1,542 3,168 SlotAcceptance 798 1,761 SlotTypeAcceptance 798 1,761 PlacementAcceptance

10 ChannelAcceptance Statistics Table 5 shows the compressed and uncompressed size of the prediction model. TABLE 5. PREDICTION MODEL SIZE PREDICTION MODEL SIZE/KB (COMPRESSED) SIZE/KB (UNCOMPRESSED) CreativeAcceptance 120, ,657 CPU Utilisation Table 6 shows average CPU utilisation during the test of various components in the RTD infrastructure. The utilisation of individual Exalytics components was measured using the top command, and the server as a whole using sar. The utilisation of components on Windows was measured using Windows Performance Monitor. Each CPU utilisation figure is as a percentage of the total processing available on that server. TABLE 6. AVERAGE CPU UTILISATION OF KEY COMPONENTS COMPONENT SERVER AVERAGE CPU UTILISATION/% java.exe (Decision Server 1) Exalytics 6.9 java.exe (Decision Server 2) Exalytics 6.9 java.exe (Decision Server 3) Exalytics 7.0 java.exe (Decision Server 4) Exalytics 6.9 java.exe (Decision Server 5) Exalytics 6.9 java.exe (Decision Server 6) Exalytics 7.0 java.exe (Decision Server 7) Exalytics 7.0 java.exe (Decision Server 8) Exalytics 7.0 java.exe (Learning Server) Exalytics 6.7 Server as a whole Exalytics 60.8 oracle.exe (RTD database) Windows 2008 R2 1.1% oracle.exe (CUST database) Windows 2008 R2 0.4% Figures 7, 8 and 9 show how CPU utilisation varies with time for key components. The CPU used by a Decision Server falls slightly during the first few hours of the run, rising again slightly twelve hours in. As learning models and persistence model are persisted periodically, the CPU used by the Learning Server (which cannot process learning records during this time) dips. Once model persistence has completed there is an elevation in the Learning Server CPU as it works to process the backlog of learning records illustrated in Figure 5. 10

11 The overall utilisation of the Exalytics server reflects a combination of the utilisation patterns of the Decision Servers and Learning Server. Figure 7. Variation of User CPU for Decision Server 1 Figure 8. Variation of User CPU for the Learning Server Figure 9. Variation of User CPU Utilisation for the Exalytics Server Overall Conclusion The test demonstrated the performance which could be achieved with the RTD Base Marketing Inline Service for customers with 100 attributes and 800 choices per decision. The data presented shows that an Exalytics X3-4 server running 8 Decision Servers and a Learning Server under RTD 3.2 supports almost 11,000 concurrent RTD sessions, making over 700 decisions per second at 60% CPU utilisation. The Learning Server maintained in real-time the models associated with the Learning Graph, processing up to 1,000 learning records per second. 11

12 Appendix A: Configuration RTD Database Oracle Database was installed on the Windows 2008 R2 database server and three General Purpose or Transaction Processing instances created. Each instance was given 20 GB of memory with Automatic Memory Management enabled, and six redo logs each having a single 4 GB logfile. Large Page Support was enabled as described in the Oracle Database Platform Guide for Microsoft Windows. The database parameter processes was set to 1,000. One database instance held the SDDB data schema used by Real-Time Decisions, another held the customer and product data used by the Inline Service, and the final instance held the database schema used by Oracle Load Testing. Exalytics CPU Speed For maximum RTD throughput, the Exalytics server s CPU governor was placed in performance mode rather than the default ondemand. RTD Platform Version of RTD was installed, under WebLogic Server with JRockit R Note that: HTTP access logging was turned off for all Decision and Learning Servers. The maximum number of connections in both SDDS and CLMDS data sources was set to 30. The Wrap Data Types option was unchecked for both data sources. This feature enables the tracking and debugging of JDBC connections by WebLogic Server. Turning it off can lead to a performance increase. Logging Logging levels were set as follows: MBEAN ATTRIBUTE DECISION SERVERS LEARNING SERVER InlineServiceLoggingLevel INFO INFO LoggingLevel INFO DEBUG Performance Logging Performance logging was enabled on the Decision Servers as follows: MBEAN ATTRIBUTE DSPerfCounterEnabled DSPerfCounterAppend VALUE true true DSPerfCounterLogInterval 15,000 RTD Call Forwarding RTD call forwarding was disabled, since the Java Smart Client ensured that all requests made during particular script iteration went to the same Decision Server: 12

13 MBEAN ATTRIBUTE ManageSessionAffinity VALUE false NUMA Node Affinity The Learning Server Java process was pinned to a single NUMA node, using the Linux numactl command. The Exalytics server has 4 NUMA nodes, each having 20 logical cores. Physical memory is divided equally among the NUMA nodes and it is more efficient for a thread to access memory belonging to its local NUMA node than remote NUMA nodes. This is particularly relevant to the Learning Server because it uses several threads to apply Learning Records to models in parallel. Learning Server efficiency has been seen to increase if all its threads run on the same NUMA node. The Decision Server Java processes were pinned to the three NUMA nodes not running the Learning Server, again using the Linux numactl command. JRockit The following JVM parameters were set in addition to those specified in the RTD documentation. Decision Servers -Xms4096m -Xmx4096m -Xns512m -XpauseTarget:200ms -Xgc:pausetime -jrockit -XX:+UseCallProfiling -XXnoSystemGC -XXcompressedRefs:enable=true -XX:+UseLargePagesForHeap -XX:+ForceLargePagesForHeap -XX:+UseLargePagesForCode -XX:MaxLargePageSize=2048k Learning Server -Xms20g -Xmx20g -Xgc:throughput -jrockit -XX:+UseCallProfiling -XXnoSystemGC -XXcompressedRefs:enable=true -XX:+UseLargePagesForHeap -XX:+ForceLargePagesForHeap -XX:+UseLargePagesForCode -XX:MaxLargePageSize=2048k The learning process was optimised using the following settings (see the RTD documentation for details): -Drtd.LearningDispatcher.LearningSleepMS=100 -DlsLearningRecordTimeout_sec=180 -Drtd.LRQueueCapacity= Drtd.numCPUs=18 13

14 Appendix B: Inline Service The Inline Service used in the test was based on the Base Marketing Inline Service included with RTD Applications, with the following modifications: Different choice data was used, as described in Appendix C. Customer data was created in a bespoke PERF_TEST_CUSTOMERS table, as described in Appendix C. The Customer Preferences attribute of the Session entity was unmapped from Customer Preferences Data Source, and the data source deleted. All columns from Customer Data Source were deleted and new ones imported from the PERF_TEST_CUSTOMERS table rather than the default CROSSSELLCUSTOMERS. All attributes from the Customer entity (other than the Customer ID key) were deleted, and new ones based on the PERF_TEST_CUSTOMERS table created. These were mapped to the corresponding columns in the data source. The logging function loginfo() was changed to logdebug() in the following: o Function CLM Is Choice Eligible o Function CLM Record Event Including Related Choices o Advisor Get Creative The application parameter CLM Database Polling Delay In Seconds was changed to 600. The study persistence time was set to 120 minutes. 14

15 Appendix C: Data Choice Data The choice data used in the test was based on that shipped with the Base Marketing Inline Service in RTD Applications, with the following modifications: There were 10 slot types, and one slot of each type. There were 20 campaigns, each campaign had 20 offers (400 offers in total), and each offer had 20 creatives (8,000 creatives in total). Each creative was assigned to a single slot such that each slot (and each slot type) was associated with 800 creatives. One hundred different tags were defined. Each creative was associated with one randomly chosen tag. Each campaign and each offer had a unique eligibility rule, which referenced the choice s region and a customer attribute, and always evaluated to true. Customer Data The PERF_TEST_CUSTOMERS table was populated with 5,000,000 customers. The table had 100 columns mapped to customer attributes in the RTD Inline Service; the types and ratio of columns were: 10% dates 20% doubles 10% integers (cardinality 100) 50% strings (cardinality 100) 10% booleans 15

16 Appendix D: Oracle Load Testing Configuration Available Ports Windows 2008 R2 server, which was used to run Oracle Load Testing, has a default number of 16,383 usable ports. The test script creates one Java Smart Client per Virtual User, with 17,000 Virtual Users, requiring that the number of usable ports had to be increased. An extra 10,000 ports were added using the netsh command as described in Load Testing Scenario The following non-default scenario settings were used in Oracle Load Testing : SETTING VALUE # VUs 17,000 Iteration Delay 1 VU Pacing (Think Time) Recorded/Random VU Pacing Lower Bound 0 VU Pacing Upper Bound 0 Maximum Users Per Process 3,000 Think time was randomly selected between 1 and 14 seconds within the script. Oracle Corporation World Headquarters 500 Oracle Parkway Redwood Shores, CA U.S.A. Worldwide Inquiries: Phone: Fax: oracle.com Copyright 2014, Oracle. All rights reserved. This document is provided for information purposes only and the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. 16