WebSphere MQ Low Latency Messaging (LLM)

Transcription

1 04 April 2013 WebSphere MQ Low Latency Messaging (LLM) Kiran Darbha 2010 IBM Corporation

2 Agenda Building a smarter planet What is messaging WebSphere MQ LLM Introduction Software overview MQ LLM - Features WebSphere MQ Vs MQ LLM 2 04/04/13

3 What is Messaging Application to application communication Event distribution on any scale Reliable delivery of information wherever it is needed 3 04/04/13

4 Challenging Times for Financial Markets Legacy Messaging cannot address today s challenges Increasing market-data volumes Fragmented orders and higher message traffic due to growth in electronic exchanges, high-frequency trading strategies and longer trading hours Spiralling market data traffic Driving higher volume and throughput requirements Extreme spikes in capacity demand Increasing cost of redundant premium hardware Increasing regulatory requirements Determination of best price for all client orders, with execution against that price Excessive message latency Caused by legacy daemon-based messaging inserting excessive network connections 4 04/04/13 Dropped messages and network storms When high volumes of messages interfere with one another

5 IBM WebSphere MQ Low Latency Messaging Ultra Low Latency and high throughput delivery Peer-to-peer daemon-less messaging Based on patented technology from IBM Research Low Latency capabilities Ultra-low latency & high throughput Shared memory support Stream failover for high availability Dynamic congestion traffic control Flexible message filtering Multicast & Unicast distribution Message store for reliable delivery Highly configurable API Ordered (FIFO) delivery Infiniband & 10GbE support WebSphere MQ MQ Low Latency Messaging Ultra-fast delivery with low latency & high availability 5 04/04/13

6 Differentiating Features of IBM Low Latency Messaging Highest Performing Messaging Middleware Highest message throughput and lowest networking latency across commodity platforms Native OFED support for InfiniBand and 10GbE Patented batching algorithms to handle bursty traffic and deliver predicable latency High Availability with No Performance Compromise Sub-second failover with zero message loss for systems that cannot tolerate downtime Ease of Use Features for Rapid Development and Deployment Supports simultaneous publish to multiple networks over multiple transports Multicast and unicast message distribution over UDP, TCP and Shared Memory Centralized configuration and monitoring Configurable levels of topic reliability range from none to assured Focus on Monitoring that Matters Built-in performance and latency statistics pinpoint system and application bottlenecks APIs for easy integration into the application and your existing monitoring infrastructure Congestion Management to Ensure System Stability Automatic detection and suspension of individual slow consumers Real-time application notification of high re-transmission rates and queue depths Message Filtering to Help you Scale Support for millions of logical flows on a single topic 6 04/04/13

7 IBM LLM Drives Down Latency & Increases Throughput Setting the bar for Throughput 98 million messages per second on Native Infiniband and Shared Memory 75 million messages per second on 10 Gigabit Ethernet Industry-leading Latency 770 nanoseconds for shared memory 2 microseconds over Native InfiniBand 3.6 microseconds over Ethernet (10 GbE*) System Configuration IBM HS22 blades: 2 x Quad core Intel Xeon E GHz 14GB RAM. Linux RHEL 5 update 3 (x86_64 64 bit) Voltaire 40 Gb IB Switch Module / BNT 10Gb Ethernet Switch Module Mellanox ConnectX MT26428 HCAs / Chelsio T320 Dual Port 10GbE Adapter * 10GbE using RoCEE Shared Memory Test on HS22 Blade with 3.33GHz Intel Westmere processors Low Latency Single hop Average Message Transmission Network size rate (msgs/sec) (bytes) 10K 100K 1 GbE Ethernet μs 34 μs 10GbE Ethernet* μs 4.5μs InfiniBand μs 3 μs Shared Memory ns 810ns High Throughput 10 Gigabit Ethernet Message size (bytes) Message Rate (msgs/sec) 12 75,914, ,253, ,724, ,846 12,000 98, /04/13 IBM Confidential

8 Software Overview LLM is a pure messaging library Transport provides one-to-one, one-to-many and many-to-many data exchange. Peer-to-peer messaging, no centralized bottleneck Two transport-specific APIs Reliable Multicast Messaging (RMM) UDP unicast and multicast Reliable Unicast Messaging (RUM) TCP unicast Shared objects loaded into the application process

9 Software Overview, cont. Publish-Subscribe No centralized queue for messages; daemon-less. Transmitter publishes on a topic (RMM) or queue (RUM) Receiver subscribes to a topic or queue Topic usually identified by a unique name. Can have multiple transmitters/receivers for the same Topic Each transmitter has a unique stream

10 LLM Platforms Current platform support: Linux (RedHat (4/5) and SuSE (10/11), 32 and 64) Windows* (XP, 2003, and Vista, 32 and 64, 2008) Solaris 10* (UltraSparc, x86-32 and 64-bit) zlinux*, plinux* HP-UX* (ia64) AIX (ppc64) Languages supported C,Java,.Net APIs Reliable Multicast Messaging (RMM) Reliable Unicast Messaging (RUM) * Some limits on functionality support. Please review Software Requirements topic of the product Infocenter (internal link provided,

11 WMQ LLM 2.6 Platform support Features RMM over Ethernet RMM over InfiniBand RUM over Ethernet RUM over InfiniBand Shared Memory Windows 32-bit & 64- bit Linux x86 32-bit & 64-bit Linux on pseries Redhat MRG 1.1 Linux on System z AIX HP-UX Solaris x86 32-bit & 64-bit Solaris Ultra SPARC Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported Supported RCMS Supported Supported Supported Supported Supported Supported Supported IPv6 3 Supported Supported Supported Supported Supported Supported Supported LLM-CM 1 Supported Supported Supported Supported Supported Supported Supported 1 LLM-CM is WebSphere MQ Low Latency Messaging Coordination Manager 2 CCT is Coordinated Clustered Time. This is available as the technology preview only on Linux x86 platforms. 3 IPv6 is not supported on Red Hat Enterprise Linux 4 (x86 or x86-64)

12 LLM Application Communication LLM 2.3 Topology showing key elements in particular peer to peer nature Applications use the Reliable Multicast Messaging (RMM) API to communicate via UDP topics. A RMM topic can use the following types of transport RMM to RMM via UDP network multicast or unicast RMM to RMM via shared memory multicast or unicast NAK RMM to RMM Multicast by Unicast (MBU) topics can combine any of the above Receiver Receiver multicast group address NAK Transmitter stream of UDP packets ACK or IP address Router Standard multicast routing and Receiver LAN1 ACK Router udp routing ACK Receiver NAK Receiver Receiver NAK ACK Receiver LAN2 Receiver 12 04/04/13

13 LLM Application Communication LLM 2.3 Topology showing key elements in particular peer to peer nature Applications use the Reliable Unicast Messaging (RUM) API to communicate via TCP queues A RUM queue can use the following types of transport RUM to RUM via TCP unicast RUM to RUM via shared memory unicast Receiver Receiver Transmitter stream of TCP packets IP address Wide Area Network Router Firewalls Proxies LAN1 Router Receiver Receiver Receiver Receiver LAN2 Receiver 13 04/04/13

14 RMM Objects Instances Distinct transmit and receiver instances Configured via initialization structures (preferred) or local text files Topic Defines a uniquely named, logical channel of communication between transmitter(s) and receiver(s) using a destination address and destination port Logical Flows TurboFlow filtering Message Properties

15 RMM Objects Events Asynchronously delivered information related to instance, topics or streams Contains type, source and descriptive information. May require application intervention (e.g. congestion related events) Log events Asynchronously delivered information for the application log file Log event codes also used as reason codes for failed RMM/RUM API calls Contain severity, module and descriptive information. Messages Transmitted piece of data between transmitter and receiver. Consists of raw bytes. Application is responsible for marshalling and byte-order conversion. Packed by RMM/RUM layer into packets for transmission. No code-enforced limits on message size.

16 RMM Objects - Events Delivered to callback for instance, topic/queue, stream Step 1: implement an RMM event handler typedef void (*rmm_on_event_t)(const rmmevent *event, void *user) ; void my_rxinstance_handler(const rmmevent *event, void *user) {} Step 2: register the event handler and create the RMM receive instance int returncode=0, errorcode=0; FILE *logfile = fopen("logr","w"); rxconfig->on_event = my_rxinstance_handler; apphandlercontext->rxinst = rx; rxconfig->event_user = apphandlercontext; returncode = rmmrxinit(rx, rxconfig, on_log_event, logfile, &errorcode); if(returncode!= RMM_SUCCESS){ char buf[256]; rmmgeterrordescription(errorcode, buf, sizeof(buf)); fprintf(stderr, "Could not initialize an RMM receiver. Error code: %s\n", buf); }

17 RMM Objects Log Events Important to log for debugging Step 1: implementing the log event listener typedef void(* llm_on_log_event_t)(const llmlogevent_t *log_event, void *user) void on_log_event(const llmlogevent_t *log_event, void *user) { } Step 2: register the listener with the transmitter or receiver instance int returncode=0, errorcode=0; FILE *logfile = fopen("logt","w"); returncode = rmmtxinit(tx, txconfig, on_log_event, logfile, &errorcode); if(returncode!= RMM_SUCCESS){ char buf[256]; rmmgeterrordescription(errorcode, buf, sizeof(buf)); fprintf(stderr, "Could not initialize an RMM transmitter. Error code: %s\n", buf); }

18 Transmitting Messages (& Packets) Each time a transmitter app executes a submitmessage() API call a message is placed in a packet A packet only contains messages destined for the same Topic or Queue The transmitter can control when packet is sent: By configuring how long packet is kept open By flushing message buffer By setting don t batch flag in message Packet C: rmmtxsubmitmessage() Java: RmmTxTopic.submitMessage() C: rmmtxflushmessagebuffer() Java: RmmTxTopic.flushMessageBuffer() C: rmmtxmessage.dont_batch Java: RmmTxMessage.dontBatch Msg Msg Msg Msg Msg A Receiver ACKs or NAKs a Packet not individual messages

19 Receiving messages (& packets) Explicit Poll (Synchronous) o User thread o RMM uses rmmrxreceivepacket() o RUM uses rumrreceivepacket() o These methods return a rmmpacket or rumpacket. o The number of messages in the packet specified by num_msgs. Callback (Asynchronous) o System thread o Specify on_message() or on_packet() callback routine o These callback routines are specified in rmmrxtopicparameters or rumrxqueueparameters

20 LLM Filtering Filtering within LLM occurs at several levels. Each mechanism is more efficient than the next. Hardware filtering Packets addressed to specific multicast groups are filtered by the network element (switch) based on the multicast groups the RMM instance has joined. Packets addressed to specified unicast destinations are filtered by the network switching infrastructure and the network interface card of receivers on the network segment. Operating System filtering Packets addressed to a specific port are filtered by the kernel based on the ports the RMM receiver is using. Packet filtering Packets are filtered by RMM based on the acceptance criteria of the instance reception topics (e.g. by topic name) Message filtering Turbo Flow labels Message Properties

21 Turboflow Filtering TurboFlow performs per-message filtering based on a label attached to each message by the transmitter. Integers The integer label attached to the message must match one of the integer labels that the receiver has chosen to accept. Bitmaps label must mask the bitmap label that the receiver has chosen to accept. Name String label hashed by LLM. Integer-based labels/filtering; Comparison very efficiently Topic can efficiently support hundreds of thousands of individual TurboFlows. This is useful, for filtering by individual Stock Exchange symbol. Usually, TurboFlow mode for the RMM topic must match for both transmitter and receiver. C:rmmTxTopicParameters.turbo_flow_mode, rmmrxtopicparameters.turbo_flow_mode Receiver specifies requested flows: C: rmmrxaddacceptedflow()/rmmrxremoveacceptedflow() A receiver can specify the following filter by stream mode when creating an RMM Receiver topic Allows receiver filtering to not match transmitter topic TurboFlow configuration C: rmmrxtopicparameters.filter_by_stream_mode Possible values: RMM_FILTER_BY_STREAM_DISABLED - Default RMM_FILTER_BY_STREAM_TF - TurboFlow filtering RMM_FILTER_BY_STREAM_PROPS - Message property filtering RMM_FILTER_BY_STREAM_ALL - Filter with TurboFlows and Message Properties

22 Message Properties Transmitter side data set in the message RmmTxMessage.msgProps Both sender and receiver must enable message properties(set enable_msg_properties to 1) Set of Name=Value pairs Attached to message value typed Different from user data Name can be character and/or binary integer Set using rmmmsgproperty structure in C. Powerful message selection using arithmetic & logical expressions. Property types Byte arrays String Integer(32bit/64bit) Double(64bit floating pt).

23 Building a smarter planet Message Selectors Method of selecting messages Applied after multicast group, topic and turbo flow Two ways to use a selector SQL 92 based selection clause Evaluates to a boolean Only property names and numeric constants allowed Can reference message properties Examples: price > 3 AND quantity > 10 Precedence of AND over OR operation is not supported - logical expressions containing AND and OR operations are evaluated from left to right. The application should use parenthesis to enforce the right precedence. For example the selector "Prop1='A' OR Prop1='B' AND Prop2='C'" will be evaluated as "(Prop1='A' OR Prop1='B') AND Prop2='C'". Note that NOT operations have the right precedence. All properties filter Define using rmmmsgselector structure

24 RMM Reliability Features Buffering (sender/receiver) Feedback modes Negative Acknowledgement (NAK) Positive Acknowledgement (ACK) With Positive Acknowledgement, additional levels of assurance Single ACK Multiple ACK Mandatory ACKers (sender/receiver) Application level ACK (receiver) Late join Using Transmitter history buffer Using Message Store

25 No Compromise High Availability Reliable and Consistent Message Streaming Tier 1 Independent data sources Message streaming Hot-hot, lossless failover availability framework for applications Relieves application from dealing with high availability issues Tier 2 Replicated server components Tier 3 Replicated client components Total ordering High availability: - failure detection - tier management - component failover High availability High availability Reliable multicast New component synchronization Message streaming Reliable multicast Stream failover Late joiner support Rich set of features Fast failure detection and failure recovery Application state synchronization Intra-tier communication Coordinated application state changes Total Ordering preserved among multiple streams 25 04/04/13

26 Message Persistence Message Recovery Support for latejoining receivers Late-joining receivers retrieve messages for topic from message store Transparent transition to live message stream when receiver has caught up Allows joining prior to the history in the transmitter 26 04/04/13

27 Centralized Topic Namespace Management Key challenges for large distributed messaging installations Managing the topic and network address space Distributing this information to the end point applications The Coordination Manager Topic Mapper enables simpler management Administrator define rules mapping topic & receiver properties (e.g. topic name, client IP) to topic network destination details (e.g. multicast address, port) Application can specify topic details or use the mapper when creating topic 27 04/04/13

28 Load Balancing with IBM Low Latency Messaging Queue-less load balancing message delivery style Support for load-balanced and broadcast partitions on a single topic Dynamic management of load based on receiver acknowledgement speed Support for out-of-order acknowledgements and transmitter notification of acknowledgement 28 04/04/13

29 Monitoring Features of WebSphere MQ LLM LLM provides a robust monitoring API with access to various statistics Instances General transmitter and receiver instance statistics including aggregate message rates, bytes sent, buffer and memory utilisation, and the current configuration. Topics Receiver topic statistics such as accepted streams, queue utilisation and topic configuration Streams Transmitter and receiver stream statistics including messages and packets processed, received, filtered, duplicated or lost, NAKs generated, and stream configuration Receiver sets Transmitter statistics on the current known receivers, their buffer utilisation, NAK generation and congestion management policy. Latency Transmitter and receiver topic latency information. The level of detail for these statistics is configurable and ranges from basic buffer utilisation information to detailed histograms of internal and external latency timings. Detailed statistics can negatively affect performance.

30 30 04/04/13 Building a smarter planet Congestion Management Features of WebSphere MQ LLM Memory limits with event notification Per-instance limits for the amount of memory each instance may consume. Per-topic limits on the size of transmit and receiver buffers. Configurable time or space-based queue cleaning parameters. Configurable event notification thresholds for danger of memory or buffer exhaustion. Rate limitation Static rate limits prevent transmitter from exceeding a set message rate Dynamic rate limitation adjusts transmitter message rate to prevent loss based on receiver feedback Repair traffic limitation with event notification Receiver limits on repair traffic : NAK/ACK generation rates, retries and response timeout values Transmitter limits on repair traffic : NAK response rate, NAK processing limit Configurable event notification thresholds for when generation limits are exceeded Slow consumer policies Local NAK suppression features Transmitter policies that for suspension or expulsion of receivers when they exceed NAK generation thresholds Configured on a per-topic, per-receiver basis.

31 Configuring Low Latency Messaging LLM can be configured programmatically or using local configuration files. Programmatic configuration is recommended. LLM requires a set of basic configuration settings. Network interface, default port, TTL, maximum packet size, memory buffer sizes LLM can be fine-tuned using a set of advanced configuration settings. Default queue sizes, threading, batching algorithm LLM can be fine-tuned programmatically on a per-topic basis Ports, queue sizes, reliability, congestion management policy Near optimum performance is quickly achievable out of the box but requires several system and network configurations to be properly set.

32 Configuring - C The approach to configuration of LLM objects and LLM actions is the same LLM objects require a set of basic configuration settings to function. Instances, Topics, etc. LLM actions require a set of basic configuration settings to run Monitor, Change Configuration, etc. Each object or action has a configuration structure associated with it. Objects are configured programmatically. Each configuration structure must be initialized before use. This sets structure defaults. Required fields in each structure must be set before use. The object is initialized using the initialized and modified configuration structure The action is run using the initialized and modified

33 Application Development Include files (for C) rmmcapi.h rumcapi.h llmcommonapi.h (included by above files) rmmexternalmonitoring.h (optional) Libraries (32 bit & 64 bit versions) Windows librmm.lib / librmm.dll librum.lib / librum.dll libcdcs.lib libllmi.lib Unix librmm.so librum.so libcdcs.so

34 Sample Transmitter Program // define some structures rmmtinstance instance_t; rmmtxmessage msg; rmmtopict topic; rmmtxtopicparameters tparms; // transmitter instance // message to be sent // topic structure // topic parameters // initialize the (default) transmitter instance rc = rmmtxinitdefaultconfig(&instance_t, &expcode); // initialize topic parameters structure rc = rmminitstructureparameters(&instance_t, RMM_SID_TxTOPICPARAMETERS, &(tparms), &expcode); // set the on_event callback API. the same on_event callback will be used for all topics tparms.on_event = on_event; tparms.event_user = (void*) sender; // user data to be passed to on event routine // set the topic name strcpy(tparms.topic_name, topicname); // set the multicast address strcpy(tparms.dest_address, mcastgroup); // Create the topic rc = rmmtxcreatetopic(&instance_t, &(tparms), &(topic), &expcode);

35 Sample Transmitter Program (Cont d) // initialize the message structure Rc = rmmtxsetstructureparameters(&instance_t, RMM_SID_TxMESSAGE, &(msg), &expcode); // Set the payload of the message msg.msg_buf = msgbody; msg.msg_len = datalen; // set the turboflow label pointer msg.msg_label = &(turboflowlabel); // perform the send rc = rmmtxsubmitmessage(&(topic), &(msg), &expcode); // shut down the topic, lingering for 500 milliseconds = ½ second rc = rmmtxclosetopic(&(topic), 500, &expcode); // stop the publisher instance rc = rmmtxstop(&instance_t, 500, &expcode);

36 36 04/04/13 Building a smarter planet Comparing IBM WebSphere MQ and IBM WebSphere MQ LLM Speed Quality of Service Reliability Quality of Service WebSphere MQ Time Independent As-Soon-As-Possible Either non-persistent or persistent Ultra-Reliable, Bet-The-Business WebSphere MQ Low Latency Messaging Ultra-Fast Microseconds Ranges from unreliable to assured Variable, based on class of data Target Market Enterprise Messaging Low Latency Messaging Target Industries Competes with All TIBCO EMS SonicMQ Financial Markets Others with similar QoS needs TIBCO RV 29West LBM and UME Solace, Tervela Real-Time Innovations (RTI) APIs JMS, XMS and MQI Unique Platform coverage Interoperates with 80 platforms WAS, WMB, WESB, DataPower XI50, DB2, CICS, IMS, etc, etc Windows, Linux on x86, zlinux, plinux, Solaris on x86 and SPARC Included in DataPower XM70 and WebSphere Front Office 36

37 Differentiating IBM WebSphere MQ and IBM WebSphere MQ LLM What do I need in terms of...? Message Throughput (using a standard blade server) Message Latency Message Reliability (see later slides) WebSphere MQ 30K+ 2KB messages/topics per sec (Note we have no data for GbE envs) From milliseconds to seconds Once and once-only delivery (persistent) At most once delivery (non-persistent) WebSphere MQ Low Latency Messaging Full network bandwidth utilization e.g. 2.5 million 45-byte messages on 1GbE From single digit usecs to sub-100 usecs for high message throughput Unreliable, reliable and assured levels out-of-the-box. Approaching guaranteed with customer development Transaction Support All transactional envs None High Availability Message Delivery Active/Active, Active/Passive with either 3 rd party or built-in S/W failover Asynchronous and Synchronous delivery Hot-hot with transparent, lossless failover Asynchronous and Synchronous delivery Queuing Support Yes None Security Access Control, Channel Security, End to End security with ESE product Limited authentication and authorization with customer development Administration Tooling Yes None Configuration Tooling Yes Limited only to topic mapping server 37 04/04/13 37

38 References/Links

39