Manjunath Subburathinam Sterling L2 Apps Support 11 Feb Lessons Learned. Peak Season IBM Corporation

Transcription

1 Manjunath Subburathinam Sterling L2 Apps Support 11 Feb 2014 Lessons Learned Peak Season

2 Agenda PMR Distribution Learnings Sterling Database Miscellaneous 2

3 PMR Distribution Following are the areas where PMRs were logged in the area of performance in this season. Highlights of this report: Less than 10% of these issues required product enhancements/fixes Bulk of these issues are implementation and design issues About 72% of the database issues were related to DB maintenance Solution - Includes Solution and Custom code issues build on Sterling, Agent Server and Application servers Database Query Tuning via indexes, histograms and database parameter changes Misc JMS, XSD, Hardware and network related issues 3

4 Part I: Solution Learnings 4

5 Solution Learnings Blocking locks in application In simple terms blocking lock occurs when one user activity is blocking other. Blocking locks are seen more prominently during peak days due to volume and increased user activities. Blocking locks can be avoided by carefully considering the following: Extended transaction boundary For example, Create Order ON_SUCCESS Schedule and Release Create Shipment This design is a high potential blocking lock candidate as locks are held in the application for a extended period of time This transaction can be broken to smaller chunks of APIs, Txn#1 Create Order. JMS Txn#2 Schedule and Release JMS Txn#3 Create Shipment. This approach would enhance throughput and provide better memory management. Tip Review your design for large transaction boundaries like the one above and break them into smaller ones. 5

6 Solution Learnings Unsupported API Output template Solution Developers tweaking the standard API output template to obtain a desired output. For example: Above output template may yield and fetch the OrderLine data as part of getshipmentlist API, but this approach can lead to API issuing a FOR UPDATE statement on Order or Order Line tables. This locking can manifest as a blocking lock in application if getshipmentlist call returns a large set of Shipments. The optimum design would be to call getorderlinelist by massaging the output of getshipmentlist and then merging those outputs to derive the required XML. Tip Use only supported output XML mentioned in javadocs 6

7 Solution Learnings Enabling Hot SKU feature Hot SKU is a powerful feature that can minimize the Inventory Item locking providing great throughput benefits and better response times. (For details on hot SKU parameters visit Hot SKU Controls section in Performance Management guide) When do I enable HotSKU? HotSKU can be enabled during any point in time. Enabling HotSKU does not cause any negative performance impact. Do I need to consider additional memory requirements if I enable HotSKU? No, HotSKU feature runs on the App/Agent server memory and there is no significant memory impact on the JVM Do I need to run Consolidate Additional Inventory always? Yes, Consolidate Additional Inventory takes care of synchronizng _Addnl and Inventory Supply and Demand tables in the background. Frequency and timing of this agent needs to be derived based on your solution. 7

8 Solution Learnings Will HotSKU completely eliminate row lock contention? HotSKU is a powerful feature to minimize the Inventory Item Locking but it does not guarantee elimination of row lock contention. Can I configure Items to be hot beforehand? Yes, if you have the set of Items which are going to be on sale, you can configure them to be hot using yfs.yfs.hotsku.skiplockinventoryitemlist What should be the value of these parameters: yfs.yfs.hotsku.numberofabnormallocksforswitchtohotsku and secondstoclassifyas AbnormalTime? Values for these and other hot sku parameters are to be derived based on your performance testing exercise. Why do I find out whether an Item is tuning hot? By looking at the application logs or querying the _Addnl tables. Please refer HotSKU documentation for more information on this. Tip- HotSKU now works for items with Infinite Inventory 8

9 Solution Learnings Sourcing and Scheduling Size of Distribution Groups In most cases response times of Inventory Availability and other sourcing and scheduling related transactions linearly increases with the size of distribution groups. If you have large distribution groups, consider breaking them into smaller groups. User Exit and External Systems response times Latency in external systems response is one of the major areas of bottleneck in Scheduling and Sourcing transactions. It is a good idea to have the external systems tuned to have a sub-second response times. Any latency in external system will also increase the lock holding time on the database. Wherever possible make calls to external systems asynchronously 9

10 Solution Learnings Store Fulfillment If you have store fulfillment enabled and if stores that can frequently run out of capacity use the Minimum Available Capacity configuration to avoid sourcing engine from scanning unnecessary nodes Tip If you are invoking an another Sterling system to get Inventory Availability, do you know using out of the box HTTP in place of a WebService call gives a performance boost? 10

11 Solution Learnings Order and Shipment Monitor Order and Shipment Monitor are non Task Q agents which scans the Order and Shipment table for qualifying records, so the key here is do regular maintenance activities on Order and Shipment tables. Close Order Configure Close Order to run frequently(atleast once in a day during non-peak hours). Running Close Order will reduce the number of Orders/Shipments monitor needs to scan. Events Configured on Monitor Monitors are primarily designed to raise events. However these events can also be used to do extended business logic. For example: Order Monitor Event Send an getshipmentlist Cancel Order Calling Cancel Order asynchronously would yield performance benefits in this case Order Monitor Event Send an getshipmentlist JMS Cancel Shipment 11

12 Solution Learnings Maintaining Monitor Rules Throughput of Monitor decreases with increase in number of Order/Shipment Monitor rules. It is a good idea to review the rules and disable insignificant rules during the peak season. Do you know setting Collect Pending Jobs to N under Agent Criteria Details can reduce the Monitor load on the database? Collect Pending Jobs is used for gathering stats to display on System Management Console and disabling this does not interfere with processing of this agent 12

13 Solution Learnings Payment Processing Payment Processing are performance intensive non-task q based agents which plays a critical role in Order Processing. Trigger Interval and Number of Records to buffer Trigger Interval and Number of Records for Payment Agents are to be derived carefully. Aggressively triggering this agent can lead to expensive database queries on Order Header table. User Exit and Webservice Calls UserExit and Webservice calls are to be optimally tuned for the Payment Agents. Request Collection during Create Order Calling requestcollection during Create Order and receiving Orders from Order Capture channel in Authorized status will significantly reduce the load on Payment Processing. 13

14 Solution Learnings Real Time Availability Monitor Enable JMS pooling using the following parameter yfs.jms.session.disable.pooling This enhances the performance of JMS sender side processing Enable JMS bulk sender mechanism using following parameters, yfs.agent.bulk.sender.enabled yfs.agent.bulk.sender.batch.size 14

15 Part II: Database Learnings 15

16 Database Learnings YFS_INBOX Queries YFS_INBOX table is in the centre of Sterling exception and alert mechanism. YFS_INBOX table should be well maintained. Inbox Purge Configure and run Inbox purge regularly. Reasonable size of Inbox table is critical for user login and other critical synchronous operations. Design to close exceptions Inbox table purges only closed exceptions and alerts. It is imperative to have a alert closing mechanism for Inbox queries to be effective. Watch out for getexceptionlist API calls It is typical to have getexceptionlist API calls in the custom code. Queries resulting out of getexceptionlist can be a heavy on database. Ensure that you index Inbox table in accordance with getexceptionlist API input. 16

17 Database Learnings YFS_AUDIT Abandoned Orders in System It is not abnormal to have Orders which are not being fulfilled for few months. These Orders over a period of time accumulate a huge set of audit records. For example An order being unsuccessfully processed by Schedule Order agent for few months. Such Orders become a bottleneck in the system and slows down the Order Processing. Device a mechanism to identify those Orders and put them on hold. Audit Purges Running Audit Purges regularly can reduce the database overhead created by YFS_AUDIT queries. Lesson Watch out for getorderauditlist API calls in solution. If getorderauditlist API is invoked for abandoned Orders with huge number of audits can cause performance impact. Use MaximumRecords to limit the number of records being fetched. 17

18 Database Learnings YFS_PERSON_INFO YFS_PERSON_INFO table is frequently accessed by core DOM APIs like Create, Schedule and Release. Run PersonInfoPurge regularly to keep this table size under check. Tip In lower environment (like stress), ensure that the addresses being used are randomized. For example: If the same Name/ Id is used for all the Orders, it can slow down Order Creation and Sourcing operations. Order Search based on status If your solution is designed to have Order Searches based on status, ensure that YFS_ORDER_RELEASE_STATUS table is regularly purged(order Purge cleans this table). Collaborate with your DBAs to index this table based on your input. 18

19 Database Learnings Lock Time-out errors Lock time-out errors are common whilst changing Orders/Shipments. Lock Time-Out errors are typically harmless errors thrown to avoid a blocking lock situation in database. However at times this error can impact user experience. You can explore the option of using SelectMethod in API input to make a user wait until the transaction holding the lock completes. Tip Setting SelectMethod as WAIT for synchronous transactions can infinitely take away a thread from the pool in case of rogue locks on the database side. 19

20 Database Learnings Deadlocks ORA Deadlock occurs when two or more oracle sessions compete for mutually locked resources. Deadlocks can be best illustrated with two multiapi calls Session#1 Session#2 In the above example Session#1 will lock Order Header in the sequence of ABC, LMN and XYZ and on the other hand session#2 will follow the sequence of XYZ,LMN and ABC, hence the deadlock error In the above scenario, XML for Session#2 should be massaged and the OrderHeaderKeys should be sorted as Session#1 20

21 Database Learnings Can deadlocks cause blocking locks in application? Deadlocks do not cause database blockage. As soon as the database detects a deadlock situation it terminates the one of the deadlocked sessions. What are the diagnostics does product provide when there is a deadlock error? Run your JVM with DDLOCK_LOG_DIR=<Dir Name> and yfs. yfs.app.identifyconnection=y, When deadlock detected, system dumps the snapshot of DB (active sessions and locks held by them) in directory specified with filename Yantra_Deadlock_trace.nnnnnnnnnn. I see a deadlock error when a user performs confirmshipment action and there is absolutely no other process running at that point? Deadlocks can occur only if there are two conflicting sessions. You may need to do further investigation to find out whether the user action is spawning multiple threads. I have enabled HotSKU, why do I still see deadlocks on Inventory Item table? Deadlocks can happen whenever there are conflicting sessions and HotSku has no direct role in minimizing deadlocks. You may explore the option of yfs.inventory.sortandlock parameter. 21

22 Part III: Miscellaneous Learnings 22

23 Miscellaneous Learnings The Funnel Effect Funnel Effect is noticed pre-dominantly in solution which has inadequately tuned application thread pool and data source connections. Symptoms like the one in logs shown below may indicate such an issue, When there are more application threads and less datasource connection available, application threads start hanging depicting this behavior. For example: When there are 50 WebContainer threads available with Datasource Maximum Connection configured as only 25 is most likely going to cause hung sessions. It is a good practice to follow below formula to set these values, Datasource MaxConnections(X) > WebContainer threads(y). X = Y

24 Miscellaneous Learnings CPU Soaking CPU Soaking is a phenomenon observed when the FULL GCs triggered by JVMs is hogging all the CPU resources and time. CPU soaking does not indicate a memory leak pattern in the code but it shows that there is a high GC overhead(upto 98%) because of running the JVM with insufficient memory. While it is a challenge to predict how much memory a JVM would need, it can be derived out of a solution benchmarking exercise. 24

25 Miscellaneous Learnings High GC Overhead - Xmn While there could be multiple reasons for high GC overhead, quite often insufficient nursery space allocation in an JVM can cause high GC overhead. Setting Xmn can reduce high number of minor GCs and yield performance benefit. Tip -Xmn can be set as 1/4 th of total heap size 25

26 Questions? 26