WebLogic JMS System Best Practices

WebLogic JMS System Best Practices Daniel Joray BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA 1 View > Header and footer Date

Unser Unternehmen Trivadis ist führend bei der IT-Beratung, der Systemintegration, dem Solution-Engineering und der Erbringung von IT-Services mit Fokussierung auf und Technologien im D-A-CH-Raum. Unsere Leistungen erbringen wir aus den strategischen Geschäftsfeldern: B E T R I E B Trivadis Services übernimmt den korrespondierenden Betrieb Ihrer IT Systeme. 2 Trivadis das Unternehmen

Mit über 600 IT- und Fachexperten bei Ihnen vor Ort Hamburg 12 Trivadis Niederlassungen mit über 600 Mitarbeitenden 200 Service Level Agreements Düsseldorf Frankfurt Stuttgart Freiburg München Basel Brugg Bern Zürich Lausanne Wien Mehr als 4'000 Trainingsteilnehmer Forschungs- und Entwicklungsbudget: CHF 5.0 Mio. / EUR 4.0 Mio. Finanziell unabhängig und nachhaltig profitabel Erfahrung aus mehr als 1'900 Projekten pro Jahr bei über 800 Kunden 3 3 3 Trivadis das Unternehmen Stand 12/2013

Trivadis an der DOAG Ebene 3 - gleich neben der Rolltreppe Wir freuen uns auf Ihren Besuch. Denn mit Trivadis gewinnen Sie immer. Trivadis das Unternehmen

AGENDA 1. Java Messaging Systems 2. Java Messaging System in Weblogic 3. JMS Servers 4. Persistent Stores 5. JMS Modules 6. JMS Resources 7. Best Practices 5

Java Messaging Systems 6 View > Header and footer Date

Java Messaging Systems Messaging systems are used to build highly reliable, scalable, and flexible distributed applications. A messaging system allows separate, uncoupled applications to reliably communicate asynchronously. The messaging system architecture generally replaces the client/server model with a peer-to-peer relationship between individual components, where each peer can send and receive messages to and from other peers. This permits dynamic, reliable, and flexible systems to be built, whereby entire ensembles of sub-applications can be modified without affecting the rest of the system.. 7

Java Messaging Systems Messaging systems include high scalability (ability to support tens of thousands of clients and tens of thousands of operations per second), easy integration into heterogeneous networks, and reliability due to lack of a single point of failure. Messaging systems provide a host of powerful advantages over other, more conventional distributed computing models. They encourage "loose coupling" between message consumers and message producers. There is a high degree of anonymity between producer and consumer. The message consumer doesn't matter who produced the message, where the producer lives on the network, or when the message was produced. 8

Java Messaging Systems Two messaging systems models are in common use. Point to Point messaging systems, messages are routed to an individual consumer which maintains a queue of "incoming" messages. Messaging applications send messages to a specified queue, and clients retrieve messages from a queue. Publish/Subscribe (pub/sub) messaging system supports an event driven model where information consumers and producers participate in the transmission of messages. Producers "publish" events, while consumers "subscribe" to events of interest, and consume the events. Producers associate messages with a specific topic, and the messaging system routes messages to consumers based on the topics the consumers register interest in. 9

Java Messaging Systems A JMS message structure Message Header Properties Message body Message header is used for message identification. The header is used to determine if a given message is appropriate for a "subscriber" In message header you find field like MessageID, Timestamp, DeliveryMode, Expiration, Priority, etc. Properties (optional) is used for application-specific, provider-specific, and optional header fields Body holds the content of the message. Several formats are supported, including TextMessages, ObjectMessages, BytesMessages, etc. 10

Java Messaging Systems Message size has a direct impact on the message throughput 2000 Message in Kb / Store Location Disk 1800 1600 1400 Msg /Sec 1200 1000 800 Write Read 600 400 200 0 1 2 5 10 20 50 11

Java Messaging Systems JMS Connection qconfactory = (QueueConnectionFactory) ctx.lookup(jms_factory); qcon = qconfactory.createqueueconnection(); qsession = qcon.createqueuesession(false,session.auto_acknowledge); queue = (Queue) ctx.lookup(queuename); qsender = qsession.createsender(queue); msg = qsession.createtextmessage(); 12

Java Messaging System in Weblogic 13 View > Header and footer Date

Java Messaging Systems Weblogic Weblogic server is compliant with the JMS 1.1 specification As JMS provider, Weblogic Server includes the use of JNDI for lookup purpose, JMS servers, JMS configuration modules and persistent stores. Additional JMS-related resources can be used by JMS Servers and JMS Module like Path Service JMS-Store-and-forward Messaging bridges 14

Java Messaging Systems Weblogic Schema from a Weblogic JMS System JNDI Tree CF Weblogic Domain JMS Server Queue JMS Module Messages Producer Persistent Store Messages Consumer 15

JMS Servers 16 View > Header and footer Date

JMS Servers A JMS server is an environment-related configuration entity that acts as management container for JMS queue and topic resources defined within JMS modules that are targeted to specific that JMS server. A JMS server's primary responsibility for its targeted destinations is to maintain information on what persistent store is used for any persistent messages that arrive on the destinations, and to maintain the states of durable subscribers created on the destinations. You can target a JMS server to either an independent Weblogic Server instance or to a migratable target server where it will be deployed. 17

JMS Servers A JMS Server belongs to a single Weblogic Server instance, and the same instance can host multiple JMS Servers You can target multiple JMS modules to each JMS Server If You need persistence for your messages, You must configure the JMS Server to use a persistent store and target the JMS Server to the same target as that for the persistent store Multiple JMS Servers can share the same persistent store 18

JMS Servers (Paging) With the message paging feature, JMS servers automatically attempt to free up virtual memory during peak message load periods. Message paging is always enabled on JMS servers, and so a message paging directory is automatically created without having to configure one, but you can specify a different directory if you wish. JMS message paging saves memory for persistent messages, as even persistent messages cache their data in memory. If a JMS server is associated with a file store (either user-defined or the server s default store), paged persistent messages are generally written to that file store, while nonpersistent messages are always written to the JMS server s paging directory. If a JMS server is associated with a JDBC store, then both paged persistent and non-persistent messages are always written to the JMS server s paging directory. 19

JMS Servers (Paging) If a Paging Directory is not specified for a JMS server, paged-out message bodies are written to the default./tmp directory inside the host server's root directory (<domain>/servers/<server>). The Message Buffer Size JMS server attribute specifies the amount of memory that will be used to store message bodies in memory before they are paged out to disk. The default value is approximately one-third of the maximum heap size for the JVM or 512Mb, whichever is larger. Paged-out message does not free all the memory that it consumes, because the message header remains in memory for use with searching, sorting, and filtering Expiration Scan Interval set the number of seconds between this JMS server's cycles of scanning local destinations for expired messages (Default 30 sec) 20

JMS Servers (Paging) Paging cost can be very expensive 1800 1600 1400 1200 1000 800 600 Messages Per Second (Paging File Size 0) Messages Per Second (Paging File Size 2Gb) 400 200 0 00:00.0 00:03.0 00:06.0 00:09.0 00:12.0 00:15.0 00:18.0 00:21.0 00:24.0 00:27.0 00:30.0 00:33.0 00:36.0 00:39.0 00:42.0 00:45.0 00:48.0 00:51.0 00:54.0 00:57.0 01:00.0 21

JMS Servers (Parameters) Bytes Threshold High and Bytes Threshold Low specifies upper and lower threshold for the number of bytes stored on the JMS server, those triggers flow control and logging events Messages Threshold High and Messages Threshold Low specifies upper and lower threshold for the of messages stored on the JMS server, those triggers flow control and logging events Blocking Send Policy determines whether the JMS server delivers smaller messages before larger ones when a destination has exceeded its maximum number of messages (default is FIFO) Maximum Message Size sets the maximum number of bytes allowed in individual messages on this JMS server 22

JMS Servers (Logging) JMS server logs provide useful information relating to the production and consumption of messages. you can configure how the server write this information with parameters like Log file name, Rotation type, Rotation file size and Limit number of retained files To activate Messages Logging you should enable the logging information when you create the JMS destination such a queue or a topic 23

JMS Servers (Monitor) You can monitor running JMS Servers from Administration Console or from WLST. 24

Persistent Stores 25 View > Header and footer Date

Persistent Stores A persistent store provides a built-in, high-performance storage solution for Weblogic Server subsystems and services that require persistence The persistent store supports persistence to a file-based store (File Store) or to a JDBC-enabled database (JDBC Store). You can target a persistent store only to one servers or to a migratable target server To create a File Store the path set in parameter Directory muss exist. For highest availability, use either a Storage Area Network (SAN) or a dual-ported SCSI disk. With the parameter Synchronous Write Policy you can define how hard a file store will try to flush records to the disk. The available values are: Direct-Write (default), Cache-Flush, and Disabled. 26

Persistent Stores You can configure for each FileStore the Synchronous Write Policy 2000 1800 1600 1400 1200 1000 800 600 Messages Per Second (Direct Write) Messages Per Second (Disable) Messages Per Second (Cash Flush) 400 200 0 00:00.0 00:03.0 00:06.0 00:09.0 00:12.0 00:15.0 00:18.0 00:21.0 00:24.0 00:27.0 00:30.0 00:33.0 00:36.0 00:39.0 00:42.0 00:45.0 00:48.0 00:51.0 00:54.0 00:57.0 01:00.0 27

Persistent Stores Best Practices For subsystems (JMS Server) that share the same server instance, share one store between multiple subsystems rather than using a store per subsystem. Sharing a store is more efficient for the following reasons A single store batches concurrent requests into single I/Os which reduces overall disk usage. Transactions in which only one resource participates are lightweight one-phase transactions. Conversely, transactions in which multiple stores participate become heavier weight twophase transactions. Add a new store only when the old store(s) no longer scale. 28

Persistent Stores To create a JDBC Store you need a Datasource You cannot specify a JDBC data source that is configured to support global (XA) transactions The specified JDBC data source must use a non-xa JDBC driver You cannot enable Logging Last Resource or Emulate Two-Phase Commit in the data source This limitation does not remove the XA capabilities of layered subsystems that use JDBC stores Weblogic JMS is fully XA-capable regardless of whether it uses a file store or any JDBC store Multiple stores cannot share the same table. 29

Persistent Stores WLS will automatically attempt to create the required table if The table is not already present The data source credentials have Create rights The data source vendor type is not Other A matching DDL file is found for the data source type You can chose the prefix from the table [prefixwlstore] Store DDL files for standard vendors are found within the archive: <WL_HOME>/modules/com.bea.core.store.jdbc_xxx.jar For Oracle databases, the default DDL used is oracle.ddl, this DLL create a table who the record is save in a LONG RAW Colum. You can change the default DDL whit Create Table from DDL File field. oracle_blob.ddl is available that uses a BLOB data type instead of the default LONG RAW type. 30

Persistent Stores Performance Not all stores have the same performance 2000 1800 1600 1400 1200 1000 800 600 400 Write MPS (Disk) Write MPS (NFS) Write MPS (Memory) Write MPS (JDBC) Read MPS (Disk) Read MPS (NFS) Read MPS (Memory) Read MPS (JDBC) 200 0 00:00.0 00:02.0 00:04.0 00:06.0 00:08.0 00:10.0 00:12.0 00:14.0 00:16.0 00:18.0 00:20.0 00:22.0 00:24.0 00:26.0 00:28.0 00:30.0 00:32.0 00:34.0 00:36.0 00:38.0 00:40.0 00:42.0 00:44.0 00:46.0 00:48.0 00:50.0 00:52.0 00:54.0 00:56.0 00:58.0 01:00.0 31

JMS Modules 32 View > Header and footer Date

JMS Modules JMS modules are application-related definitions that are independent of the domain environment JMS modules are globally available for targeting to servers and clusters configured in the domain, and therefore are available to all applications deployed on the same targets and to client applications The following configuration resources are defined as part of a system module or an application module Connection factories Queue and topic destinations Templates Destination keys Quota Distributed destinations (Queue or Topic) Foreign servers JMS store-and-forward (SAF) configuration items 33

JMS Modules Not all JMS Resource in a JMS Module have the same target. In this case you can use Sub-Deployment to define a specific physical target to a JMS Resource JMS Module Queue Sub- Deployment Server JMS Server 1 Queue Topic Sub- Deployment Server JMS Server 2 Connection Factory 34

JMS Resources 35 View > Header and footer Date

JMS Resources (Connection Factory) A Connection Factory encapsulates connection configuration information, and enables JMS applications to create a Connection. You can use the preconfigured default connection factories provided by Weblogic JMS, or you can configure one or more connection factories to create connections with predefined attributes that suit your application The Connection Factory allow you to configure parameters like timeout, delivery mode, transaction management and load balancing management 36

JMS Resources (Connection Factory Parameters) Default Delivery main parameter are: Default Priority: The default priority used for messages when a priority is not explicitly defined Default Time-to-Live: The maximum length of time, in milliseconds, that a message exists Default Time-to-Deliver: The delay time, in milliseconds, between when a message is produced and when it is made visible on its destination Default Delivery Mode: The default delivery mode used for messages when a delivery mode is not explicitly defined Send Timeout: The maximum length of time, in milliseconds, that a sender will wait when there isn't enough available space (no quota) on a destination to accommodate the message being sent. 37

JMS Resources (Connection Factory Parameters) Client main parameters are: Client ID Policy: The Client ID Policy indicates whether more than one JMS connection can use the same Client ID Subscription Sharing Policy: Specifies the subscription sharing policy on this connection. Maximum Messages per Session: The maximum number of messages that can exist for an asynchronous session and that have not yet been passed to the message listener Transactions main parameters are: XA Connection Factory Enabled: Indicates whether a XA queue or XA topic connection factory is returned, instead of a queue or topic connection factory. Transaction Timeout: The timeout value (in seconds) for all transactions on connections created with this connection factory 38

JMS Resources (Connection Factory Parameters) Load Balance main parameters are: Load Balancing Enabled: Specifies whether non-anonymous producers created through a connection factory are load balanced within a distributed destination on a per-call basis. Server Affinity Enabled: Specifies whether a server instance that is load balancing consumers or producers across multiple members destinations of a distributed destination, will first attempt to load balance across any other physical destinations that are also running on the same server instance. 39

JMS Resources (Connection Factory Parameters) Flow Control main parameters are: Flow Control Enabled: Specifies whether a producer created using a connection factory allows flow control. If true, the associated message producers will be slowed down if a JMS server or a destination reaches its specified upper byte or message threshold Flow Maximum: The maximum number of messages-per-second allowed for a producer that is experiencing a threshold condition. When a producer is flow controlled it will never be allowed to go faster than the Flow Maximum messages per second 40

JMS Resources (Destination Keys) Destination Key defines a unique sort order that destinations can apply to arriving messages. As messages arrive on a specific destination, by default they are sorted in FIFO (first-in, first-out) order, which sorts ascending based on each message's unique JMSMessageID. You can use a destination key to configure a different sorting scheme for a destination, such as LIFO (last-in, first-out). Parameters Sort Key: Specifies a message property name. Like JMSMessageID, JMSTimestamp, JMSPriority, User-Defined, ect. Key Type: the expected property type for this destination key Direction: The direction 41

JMS Resources (Quota) Quota controls the allotment of system resources available to destinations Bytes Maximum: The total number of bytes that can be stored in a destination that uses this quota Messages Maximum: The total number of messages that can be stored in a destination that uses this quota Policy: For destinations that use this quota, this policy determines whether to deliver smaller messages before larger ones when a destination has exceeded its message quota, you can choose between FIFO or PREEMTIVE Shared: Indicates whether this quota is shared by multiple destinations that refer to it 42

JMS Resources (Templates) JMS Template defines a set of default configuration settings for multiple destinations The configurable options for a JMS template are the same as those configured for a destination If the destination that is using a JMS template specifies an override value for an option, the override value is used The JNDI Name, Enable Store, and Template options are not defined for JMS templates You can configure subdeployments for error destinations, so that any number of destination subdeployments will use only the error destinations specified in the corresponding template 43

JMS Resources (Queue) Queue defines a point-to-point destination type, which are used for asynchronous peer communications. A message delivered to a queue is distributed to only one consumer Message Message Producer Queue Consumer 44

JMS Resources (Topic) Topic defines a publish/subscribe destination type, which are used for asynchronous peer communications. A message delivered to a topic is distributed to all topic consumers Message Consumer 1 Message Consumer 2 Producer Topic Consumer 3 Message Consumer 4 45

JMS Resources (Distributed Destination) Distributed Queue defines a set of queues that are distributed on multiple JMS servers, but which are accessible as a single, logical queue to JMS clients Ditributed Topic defines a set of topics that are distributed on multiple JMS servers, but which are accessible as a single, logical topic to JMS clients. JNDI Name Jms/myDistributedQueue Producer Message Queue Server1 Message Consumer Queue Server 2 46

JMS Resources (Distributed Destination) Two types of distributed queues are available Uniform Distributed Queue Queue members are created uniformly from a common configuration. 47

JMS Resources (Distributed Destination) Distributed Queue Queue members are created and weighted individually to fine tune performance 48

JMS Resources (High Available) You can establish load balancing of destinations across multiple servers in the cluster by configuring multiple JMS servers and targeting them to the defined Weblogic Servers Each JMS server is deployed on exact one Weblogic Server instance and handles requests for a set of destinations Load balancing is not dynamic. During the configuration phase, the system administrator defines load balancing by specifying targets for JMS servers If your JMS resource should be high available, you have to migrate your JMS Server Weblogic support whole server migration and service migration (automatic and manual) 49

JMS Resources (Server Migration). Host 1 Testsrv2 Ip 10.0.10.70 Deployed applications tqueuesrv2 JMSServer2 Filestoresrv2_NFS Host 2 Testsrv3 Ip 10.0.10.80 Testsrv2 Ip 10.0.10.70 Deployed applications tqueuesrv2 JMSServer2 Filestoresrv2_NFS NFS / JDBC stores 50

JMS Resources (Service Migration). Host 1 Testsrv2 Ip 10.0.10.70 Deployed applications tqueuesrv2 JMSServer2 Filestoresrv2_NFS Host 2 Testsrv3 Ip 10.0.10.80 tqueuesrv2 JMSServer2 Filestoresrv2_NFS NFS / JDBC stores 51

JMS Resources (Distributed Destination) Distribution of application load across multiple JMS servers through connection factories, thus reducing the load on any single JMS server and enabling session concentration by routing connections to specific servers. 2500. 2000 1500 1000 Queue Distributed Queue 500 0 00:00.0 00:03.0 00:06.0 00:09.0 00:12.0 00:15.0 00:18.0 00:21.0 00:24.0 00:27.0 00:30.0 00:33.0 00:36.0 00:39.0 00:42.0 00:45.0 00:48.0 00:51.0 00:54.0 00:57.0 01:00.0 52

Best Practices 53 View > Header and footer Date

Best Practices Tuning a JMS System in Weblogic can be very complex Client Server 1 CPU IO Memory Parameters WLS Server1 JVM Memory Parameters Server 3 CPU IO Memory Network WLS-Cluster CPU IO Memory Database Memory Parameters Server 2 CPU IO Memory Parameters WLS Server2 JVM Memory Parameters NFS Shares 54

Best Practices (How to choice the right configuration) What is important for your Application? Producer insertion Consumer reliability Throughput Message Order Persistence And you should answer those questions for each JMS resource 55

Best Practices (How to make your tests) Monitor CPU, disk I/O, memory, network interfaces from all involved systems At the beginning of each test, stop the Weblogic systems and clean-up or remove persistent store, paging files, truncate store tables Configure your environment with WLST scripts ( TVD-Basenv can help you) Use a load generator (Jmeter, LoadUi, Grinder, Java function) Document each test case, a the beginning, use only short test case, when you have reached your targets with short test, you can make longterm test 56

Best Practices (JMS Server and Store) Create a custom store on each Weblogic server which will host a JMS server. (Why use a custom store? Custom stores provide more flexibility in tuning and administration. In addition, the default file store is not migratable -- only custom stores are migratable.) It is recommended to always target to migratable targets when available Configure message count quotas on each JMS server. There is no default quota, so configuring one helps protect against out-of-memory conditions. Rule of thumb: conservatively assume that each message consumes 512 bytes of memory even if it is paged out. 57

Best Practices (JMS Server and Store) Although JMS paging is enabled by default, verify that the default behavior is valid for your environment Specify a Message Paging Directory. The default value is $MW_HOME\user_projects\domains\domainname\servers\servername\tmp Message Paging Directory must not be placed on the same disk as persistent store Tuning the Message Buffer Size Option, to specifies the amount of memory that will be used to store message bodies in memory before they are paged out to disk. It is important to remember that this parameter is not a quota 58

Best Practices (JMS Module) It is almost always preferable to use system modules instead of deployable application modules. The only way to modify deployable modules is to manually edit the XML and redeploy. Create exactly one subdeployment per module. It's much easier for third parties to understand the targeting, and it reduces the chances of making configuration errors. If a single subdeployment is not sufficient, create two modules. Populate the subdeployment only with JMS servers - not Weblogic servers. For modules that support non-distributed destinations, the subdeployment must only reference a single JMS Server. 59

Best Practices (JMS Module) Configure the distributed queue to enable forwarding. Distributed queue forwarding automatically internally forwards messages that have been idled on a member destination without consumers to a member that has consumers It is highly recommended to always configure message count quotas. Quotas help prevent large message backlogs from causing out-ofmemory errors, and Weblogic JMS does not set quotas by default Configure Messages Expiration Policies Configure Flow Control Configure One-Way Message Send and Windows Size 60

Best Practices (JMS Module) Configure Redelivery Limit, the number of redelivery tries a message can have before it is moved to the error Configure Error Destination, name of the target error destination for messages that have expired or reached their redelivery limit. If no error destination is configured, then such messages are simply dropped. 61

Best Practices (Controlling the Flow of Messages) Specifically, when either a JMS server or it s destinations exceeds its specified byte or message threshold, it becomes armed and instructs producers to limit their message flow (messages per second). As producers slow themselves down, the threshold condition gradually corrects itself until the server/destination is unarmed Producers receive a set of flow control attributes from their session, which receives the attributes from the connection, and which receives the attributes from the connection factory 250 200 150 100 50 0 Message Flow Max250 Min 200 MSG / Ssec 62 100 800 1500 2200 2900 3600 4300 5000 5700 6400 7100 7800 8500 9200 9900

Best Practices (Controlling the Flow of Messages) First enable flow control on JMS Servers 100 Then configure the queue or the connection factory 400 350 300 250 200 150 50 0 One Producer Message size 5k 100 900 1700 2500 3300 4100 4900 5700 6500 7300 8100 8900 9700 Message / sec 63

Best Practices (One-Way Message Send ) One-way message send can greatly improve the performance of applications that are bottle-necked by senders but do so at the risk of introducing a lower quality-of-service When active, the associated producers can send messages without internally waiting for a response from the target destination's host JMS server 64

Conclusion It is not possible to predict the JMS performances of a system without testing You should never see a JMS Resource like a container. The physical implementation can impact the application. Not only the performances but the workflow of application. You can only choose the right JMS resource if you know the application. 65

Further information... Tuning Weblogic JMS http://docs.oracle.com/cd/e14571_01/web.1111/e13814/jmstuning.htm#chdebfdf Configuring and Managing JMS for Oracle Weblogic Server http://docs.oracle.com/cd/e14571_01/web.1111/e13814/jmstuning.htm#chdebfdf JMS Performance Tuning Series (Darrel Sharpe) http://developsimpler.blogspot.ch/2011/11/jms-performance-tuning-series-part-1.html Messaging in Weblogic Server: Best Practices (René van Wijk) http://middlewaremagic.com/weblogic/?p=6334 66 View > Header and footer Date

Questions and answers... Daniel Joray Principal Consultant Tel. +41 58 459 50 26 Daniel.Joray@trivadis.com BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA View > Header and footer Date