Copyright Push Technology Ltd December Diffusion TM 4.4 Performance Benchmarks

Transcription

1 Diffusion TM 4.4 Performance Benchmarks November 2012

2 Contents 1 Executive Summary Introduction Environment Methodology Throughput Latency Results Summary Throughput without Conflation Throughput with Replace Conflation Throughput Comparison: Conflated vs Non- Conflated Throughput Summary Latency Summary Conclusion of 16

3 1 Executive Summary Diffusion is the most efficient data distribution technology on the market and this whitepaper describes the improvements achieved with respect to performance and scalability in Diffusion TM 4.4. A selection of benchmarks were used to demonstrate the improvement and quantify Diffusion s unique approach to data distribution. In particular we introduce Structural Conflation ; in its simplest form this replaces messages yet to be distributed to a client with a more current message ensuring the clients receive the most up- to- date information. This has the dual benefit of reducing bandwidth usage whilst enabling the server to service a larger number of concurrent connections. We demonstrate how Diffusion s intelligent approach to data distribution adapts to individual client conditions, markedly improving scalability as can be seen from the conflated versus non- conflated benchmark results. In particular attention is drawn to the reductions in disconnections when under maximum stress. The unique integration of messaging with in- memory data caching and in- flight data processing allows consistent, current data to be delivered to tens of thousands of client devices with optimal resource utilisation. Diffusion TM is a high performance data distribution technology with excellent latency and throughput characteristics whether deploying data distribution solutions behind or across the firewall. The results speak for themselves. 2 Introduction One of the principal objectives of the latest Diffusion TM 4.4 release was improving performance and scalability. This was achieved by targeting performance bottlenecks measured through an objective methodology; by optimising the critical pathways in the product and in particular through adoption of lock- free wait- free concurrency techniques. Lock- free wait- free algorithms essentially increase efficiency when services are heavily loaded through reducing the cost of certain frequent operations. The compound effect is more graceful, robust and scalable performance even when finite resources are saturated. Diffusion TM is designed to go the extra mile to better service the last mile. This report outlines the performance characteristics of Diffusion TM 4.4 using commodity off the shelf hardware. The full testing methodology, details of the hardware and software stack used for the system under test, and full test results are provided. 3 of 16

4 3 Environment The test environment consists of 2 machines, with 2 sockets per machine and 6 hyper- threaded cores per socket. The machines are directly connected. Box A Box B Machine Configuration The server is deployed on one of the machines and constrained to a single processor. All clients are deployed on the other machine and constrained to a single processor. Servers and clients are constrained to a single processor on each machine to force saturated conditions in the throughput benchmark earlier in test runs. The Diffusion TM server is developed in Java. The following Java runtime was used on both benchmark machines: Java(TM) SE Runtime Environment (build 1.6.0_33- b03) Java HotSpot(TM) 64- Bit Server VM (build b03, mixed mode) The benchmarks were run with the machines directly interconnected using Solarflare Communications SFC Ge network cards. It is recommended not to exceed 30,000 concurrent connections with these cards, although our benchmark tests up to 45,000 concurrent connections. 4 of 16

5 4 Methodology Two technical benchmarks were undertaken to determine the performance profile of a single running instance of a server. A throughput benchmark designed to stress the server demonstrates how far it can be pushed against a single network card. The latency benchmark shows how low the latency can go between servers and clients. A single server instance can easily saturate multiple 10Ge network interfaces on commodity hardware. Saturating critical resources such as network IO, CPU or memory resources is not a recommended practice outside of lab or benchmarking conditions as service levels degrade beyond this point. The server was constrained to a single socket on the test server and only one network interface was used for all test runs. Diffusion TM supports multiple transports. The most widely deployed transport carrying the Diffusion TM protocol over the web and mobile today is the IETF WebSocket 1 protocol. This is the transport documented in this whitepaper. Diffusion TM browser clients will automatically cascade across available transport implementations choosing the best available, but degrading gracefully through Silverlight- based, Flash- based, XML Http Request and IFrame based transports for legacy environments. Legacy transports are not explored in this whitepaper. The same configuration of Diffusion TM was used for both benchmarks. This configuration is the out of the box default configuration and is neither optimised for throughput nor latency. Finer grained configuration for the respective benchmark improves the results but has no bearing on the overall characteristics. 4.1 Throughput Overview The throughput benchmark model is designed around a number of configurable dimensions. The test is run repeatedly from a cold start of a Diffusion TM server instance and for a fixed duration of 5 minutes. Each benchmark run uses a different message payload size. Each set of runs is repeated with no conflation configured and with replace conflation configured. Without conflation Diffusion TM acts analogously to traditional enterprise messaging technologies. With conflation, however, it diverges from traditional messaging. 1 IETF WebSocket Specification - editor.org/rfc/rfc6455.txt 5 of 16

6 The difference is clearly evidenced in the results and summarised in this report based on the evidence captured. All other configurable dimensions do not vary across benchmark runs. Client connections are ramped continuously for the duration of each run. Clients are ramped at a fixed periodic interval of 5 seconds. All new clients being connected in an interval are started simultaneously. This exaggerates disconnections and adverse conditions in a controlled way. Each client subscribes to a fixed subset of 50 available topics. Data is being published at a designed rate of 100 messages per second per client. Diffusion TM servers are typically deployed at or behind the edge of private networks where services are exposed to public networks, with multiple instances running behind a load balancer. Each instance typically serves circa 10,000 to 30,000 concurrently connected clients with small message payloads of between 20 and 100 bytes. It is a real- time stream oriented distribution technology. It intelligently batches, fragments and conflates messages to optimize distribution whilst guaranteeing the currency, and timeliness of delivery. Benchmark Duration Message Payload Sizes Mode of Operation Ramping Interval New Client Connections Per Interval Messages Per Second Per Client - 5 minute runs , 250, 500, 1000 and 2000 bytes. - Without conflation, with replace conflation. - 5 seconds per interval messages. 4.2 Latency Overview The latency benchmark model is designed around a number of configurable dimensions. The latency benchmark is a simple round trip benchmark. A single client is used. The client sends a message to the server. This ping or request message is received by and responded to by the server. The client then receives the pong or response message. This process continues for the benchmark duration. This benchmark therefore represents a best- case latency profile. Benchmark Duration Message Payload Sizes - 5 minute runs , 250, 500, 1000 and 2000 bytes. 6 of 16

7 5 Results Summary This section provides a high- level summary and brief discussion of the benchmark results. 5.1 Throughput without Conflation Smaller message sizes have increased message- processing overhead and tend to saturate allocated CPU resources sooner than IO resources for payload sizes of 500 bytes or less. Larger payload sizes tend to be limited by allocated network IO. Diffusion TM can saturate Ge interfaces with a single instance configured as in this benchmark with 1000 byte message payloads. So, depending on available compute resources and the working set of a real- world use case multiple 10Ge network devices may be appropriate, or where message sizes are typically small, increased compute capacity. Throughput increases linearly as client connections are increased at a constant rate Larger message sizes saturate available network IO sooner than smaller messages Utilized bandwidth scales linearly as expected until network IO saturation point. Optimal utilisation at saturation. 87.2% (1.09 of 1.25 GB/sec) is payload data for large messages - not TCP/IP or protocol framing (data with no business value). CPU not Network capacity limits small message throughput. 7 of 16

8 At least 1 million 1K messages per second, sustained. New connections are handled gracefully until resources saturate. New connections are managed out beyond resource saturation. 45K concurrent clients at 125 and 250 bytes per message can be sustained. 24K concurrent clients at 500 bytes per message can be sustained Circa 13K concurrent clients at 1000 bytes per message can be sustained. Benchmarking based on typical and peak utilisation for a realistic model of your system will help understanding of your provisioning requirements and enable you to plan capacity effectively. Diffusion TM is designed to minimize the impact to service levels to already connected clients. This is confirmed by the disconnection rates. No disconnections are recorded until we reach saturation. Beyond this point the rate is a function of size. The benchmark is designed to deliver a soft constant rate of 100 messages per second per client. The back- pressure waveform at 1000 bytes or more correlates well with the rate of acceptance, refusal or dropping of connections once service levels degrade at saturation. Scaling is linear until saturation occurs. Connection attempts are made continuously throughout the benchmark run. Only disconnected clients are summarised and disconnections only occur once critical resources (CPU and IO here specifically) saturate. It is not recommended to provision at or near saturation for production use. 8 of 16

9 5.2 Throughput with Replace Conflation Diffusion TM is not a traditional messaging technology. It can cache data in memory and allows deployment of user designed distribution logic. Diffusion TM servers can be networked together in broker- based or brokerless networks to collaborate on larger distribution tasks. One of the key advantages of Diffusion TM in low fidelity high latency environments is the ability for the product to actively conflate and tune data to the capabilities of each connected device and to adapt rates of distribution dynamically. Messaging technologies have no sympathy for tailoring individual SLAs to each device. Virtualizing client side queues to the server side has significant advantages where data is mostly being streamed from servers to clients. The disadvantage of course is increased complexity of guaranteeing delivery of transactional data. The wire protocol uses snapshot (delivered on subscription to a topic) and delta (delivered most of the time during the lifetime of a connection) messages. In mobile contexts clients can disconnect and reconnect frequently, so a delta of what has changed since that client last connected can be sent rather than a full snapshot, thereby saving considerable bandwidth on recovery. During normal operation deltas or changes are sent in order to conserve bandwidth. Clients, similarly, can update the services they provide based on state changes in the connection. Replace conflation essentially replaces outgoing messages yet to be distributed to a client with a more current message. This ensures clients receive the most up- to- date message and protects downstream environments from a degree of back- pressure due to needing to distribute stale data. Diffusion TM s virtualization of client side queues gives the server telemetry to detect when clients cannot cope with distribution volumetrics or conversely when rates can be increased. Performance characteristics of the server are in line with and on the same order as replace conflation as without conflation. However, the server can sustain more concurrent clients. Intelligent conflation that takes data structures into consideration can considerably increase the level of conflation and further reduce bandwidth utilization. 9 of 16

10 Bandwidth utilisation is similar to the non- conflated case with the following exceptions More concurrent clients can be serviced compared to the non- conflated case. Optimal utilisation. 87.2% (1.09 of 1.25 GB/sec) is data for large messages. Reduced bandwidth utilisation when compared to the non conflated case for smaller message sizes. 45K concurrent clients at 125 bytes using 22% less bandwidth than non conflated case at the end of the test run 45K concurrent clients at 250 bytes using 27% less bandwidth than non conflated case at the end of the test run 28K concurrent clients at 500 bytes using 1% less bandwidth and servicing 16% more clients than non conflated case 14K concurrent clients at 1000 bytes using 1% less bandwidth and servicing 4% more clients than non conflated case 10 of 16

11 Replace conflation manages bandwidth utilization and per client service levels actively. As the server becomes more saturated, more benefit is extracted from conflation. This is reflected in messages per second per client that tends to reduce over time as load increases. As the server handles more concurrent clients less disconnections occur relative to the non- conflated case. Beyond saturation per client service levels reduce accordingly. For larger message sizes this can mean up to 30% less per client. For smaller message sizes service levels reduce less until CPU resources saturate. The Diffusion TM server adapts to adverse conditions gracefully under both conditions. 5.3 Throughput Comparison: Conflated vs Non- Conflated In this section we compare non- conflated message style throughput with data distribution based on the results of those benchmarks to this end. The following graph shows both the replace conflated and non- conflated message rates in millions of messages per second over the duration of the benchmark run. We observe linearly increasing message rates until, at a certain message payload size, we reach available network IO saturation. For large message sizes of 1000 bytes or more there is no perceivable difference in messaging rates when viewed in this way. For smaller message sizes of 500 bytes or less however we see increasing benefits of actively conflating data for smaller message sizes. 11 of 16

12 Diffusion TM clients in capital markets, online gaming and gambling are typically distributing very small messages, typically less than 100 bytes per message, very frequently. The benefits of conflation are even greater at these low rates because the wire protocol buffers and packs multiple logical messages in order to maximize bandwidth utilization and minimize the distribution overhead incurred by TCP, IP and transport level framing. Distribution can also be prioritized (price information over news, for example) essentially partitioning low priority large data items over time so that small urgent messages are delivered in a timely fashion. 12 of 16

13 If we analyse the data through the lens of concurrently connected clients we see a very different view of these results. Even when a server supporting these clients is saturated, it can distribute messages to a larger number of clients. For data that is likely to change frequently, replace conflation offers distinct benefits. For data where only subsets are volatile and frequently changing then merge conflation may be a better candidate. Some data cannot be conflated at all. For example orders, trades and other transactional events should neither be lost nor forgotten. Messaging systems typically handle transactional delivery very well. Data distribution takes this further so that server side resources and client- side devices, connectivity and other constraints are utilised more efficiently. We can see clearly in the chart above that messaging style distribution does not cope well once IO saturates. High frequency recoverable data adapts to increasing demand, even beyond saturation point more gracefully where conflation can be used. Messaging systems have limited or no capability to peek inside the content and make intelligent decisions. Diffusion TM leverages data and distribution expectations smartly. 13 of 16

14 5.4 Throughput Summary Data distribution, as distinct from messaging, is concerned with delivering the right data, at the right time, in the right way. Messaging treats data as opaque blobs that it distributes between producers and consumers. Data distribution, on the other hand, understands enough about data so that it can actively manage timeliness and responsiveness in order to ensure only relevant data gets distributed. This allows the same service to distribute data both inside the firewall where 1Ge and 10Ge connectivity are common and over the firewall to clients where latency is relatively high, bandwidth is shared, and connectivity is prone to failure. The server adapts to client conditions individually. This has an effect of improving serviceability overall as can be seen from the conflated verses non- conflated throughput benchmark results. Internal clients or participants simply receive an improved quality of service. External clients receive data at an optimal rate. Both are serviced in exactly the same way, by the same services. In practice, the rate of distribution for small payloads is such that on mobile handhelds connected over 3g verses local area connectivity there is no human perceivable difference over short (on the same continent) distances. 5.5 Latency Summary The latency histogram below characterizes the best case latency achievable with Diffusion TM. The measurements plotted are less than 1 millisecond wide in total. 14 of 16

15 The histogram measures round trip time, not single hop latency. Average latency for messages of 500 bytes of less sub 100 microseconds. 99% of messages round trips in 250 microseconds or less. All messages at the th percentile deliver sub- millisecond latencies. The above measured results can be used (halved) to approximate single hop latency. Average latency for messages of 50 0 bytes or less is sub 50 microseconds. 99% of message single hops in 125 microseconds or less. All messages at the th percentile delivered in half a millisecond or less. 15 of 16

16 6 Conclusion Diffusion TM 4.4 is a high performance data distribution technology with excellent latency and throughput characteristics whether deploying data distribution solutions behind or across the firewall. This whitepaper compared throughput and latency of messaging and data distribution styles with Diffusion 4.4. Data distribution offers reduced bandwidth utilisation, increased stability and robustness, and fairer service levels than plain old messaging. The unique integration of messaging with in- memory data caching and in- flight data processing allows consistent, current data to be delivered to tens of thousands of client devices with optimal utilisation. Diffusion TM demonstrates that adaptive, smart exploitation of client and server side constraints delivers better bang for your bytes, both for individual devices, and overall. 16 of 16