Enterprise Networks for Low Latency, High Frequency Financial Trading

Transcription

1 Enterprise Networks for Low Latency, High Frequency Financial Trading Dr. Casimer DeCusatis Distinguished Engineer, IBM Corporation, Poughkeepsie, NY Presented at the 2011 Enterprise Computing Community conference, Marist College

2 Legal Slide Trademarks and disclaimers Building a smarter planet: Financial Markets 8 IBM Corporation All rights reserved. References in this document to IBM products or services do not imply that IBM intends to make them available in every country. Trademarks of International Business Machines Corporation in the United States, other countries, or both can be found on the World Wide Web at Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office. UNIX is a registered trademark of The Open Group in the United States and other countries. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. Information is provided "AS IS" without warranty of any kind. The customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Information concerning non-ibm products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-ibm list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-ibm products. Questions on the capability of non-ibm products should be addressed to the supplier of those products. All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here. Prices are suggested U.S. list prices and are subject to change without notice. Starting price may not include a hard drive, operating system or other features. Contact your IBM representative or Business Partner for the most current pricing in your geography. Photographs shown may be engineering prototypes. Changes may be incorporated in production models.

3 Overview Motivation for low latency enterprise networking Low latency within the data center IBM WebSphere LLM solution System X server with Mellanox/SolarFlare adapters IBM/BNT Rack Switch STAC benchmarking results Low latency between data centers Fiber path considerations Dispersion compensation OTN and FEC signal processing

4 M. Loro and J. Gerrity, Optical networks for low latency applications, Ciena webinar and white paper, March 29, 2011

5 M. Loro and J. Gerrity, Optical networks for low latency applications, Ciena webinar and white paper, March 29, 2011

6 M. Loro and J. Gerrity, Optical networks for low latency applications, Ciena webinar and white paper, 03/29/11

7 WebSphere MQ Low Latency Messaging Ultra low-latency & high-throughput transport for financial markets Peer-to-peer, daemon-less messaging Based on patented technology from IBM Haifa Research Lab Branded part of MQ family of transports Strategic part of messaging portfolio Now used as multicast technology for WebSphere MQ V7 Version 2.5 General Availability: November,

8 WebSphere MQ Low Latency Messaging Usage Initially built for Financial Markets customers requiring high performance, reliable messaging in the front office for market data and trade data Included with WebSphere Front Office for Financial Markets Integrated with Partners such as Data Vendors/ISVs for internal application communication and external application connectivity Production deployments in market data distribution systems, internal trade execution systems and exchange execution systems Pilots in Industries with similar quality of service requirements Financial Services Defense and Aerospace Telecommunications Logistics 8

9 IBM Messaging in the Trade Process Exchanges Data Aggregators Execution Venues Compliance Reporting Counterparty Confirmations Custodial Settlement Instructions Payments External Feed Handlers Market Data Platform Analysis, Pricing, Trade, and Order Management Pre- / Post-Trade Compliance Internal WebSphere MQ w/ High Performance Interconnect Low Latency Messaging Mission-Critical Messaging Internal / Reference Data Positions / Portfolios Market & Credit Risk Analytics Security Master Accounting and General Ledger Front Office Responsible for sales and trading operations Middle Office Manages market & credit risk, calculates profits and losses Back Office Includes settlements, clearances, record maintenance, and accounting

10 WebSphere MQ Low Latency Messaging Highest Performing Messaging Middleware Peer-to-peer daemonless messaging Native OFED support for InfiniBand and 10GbE in addition to standard IP Highest message throughput and lowest message latency over any transport Patented batching algorithms to handle bursty traffic High Availability with No Performance Compromise Sub-second failover with zero message loss for systems that cannot tolerate downtime Focus on Monitoring that Matters Built-in performance and latency statistics pinpoint system and application bottlenecks APIs for easy integration into the application and existing monitoring infrastructure Congestion Management to Ensure System Stability Automatic detection and suspension of individual slow consumers Real-time application notification of high re-transmission rates and queue depths 10

11 Combining LLM and IBM-BNT G GbE First single chip switch to pass the 1 Terabit per second barrier 64 ports, 10 Gigabit Ethernet = 1.28 Terabits (full duplex) First 10G switch to support OpenFlow First production 40 Gigabit Ethernet for the data center Advantages of a Single Chip Switch Lower latency Every port provides identical latency Lower power and heat Higher reliability Rich set of L2 & L3 Features for HFT CEE, Multicast Support, IEEE 1588* Tbps *Firmware Upgrade (Future)

12 RackSwitch G8264 Specs 4 QSFP+ Ports 10 / 40 Gig E 48 SFP+ Ports 1 / 10 Gig E 1U Leading performance Up to 64 10G ports 1.28 Terabits / second Flexible Ports 4 QSFP+ ports 1 x 40G or 4 x 10G 48 SFP+ ports 10G or 1G Every port: Copper or Fiber cable

13 HFT Messaging Solution: IBM BNT G8264 and WMQ LLM WebSphere MQ LLM Red Hat Enterprise Linux X-series 3650 M3 w/ Intel Xeon Mellanox CX-2 w/ RoCEE Solarflare w/ OpenOnload IBM BNT RackSwitch G8264 IBM and its partners have demonstrated ultra low latency messaging solutions. IBM Offers: IBM BNT G / 40 GbE High Perf switch IBM Websphere MQ Low Latency Messaging Choice of High Performance Network Adapters Mellanox ConnectX-2 Solarflare 10GbE w/ OpenOnload (VLH Program US only) High Performance X-series servers.

14 Network Adapters Mellanox CX-2 EN with RoCEE Supports RoCEE (RDMA over CEE) Requires no modification for IB applications. Provides very good latency and throughput. Available from IBM Solarflare SFN5122F 10GbE with OpenOnload (VLH from IBM US only) Support Solarflare OpenOnload (Kernel bypass socket library) Requires no application modification Provides very good latency and throughput. Available from IBM (VLH program). 14

15 STAC-M2 Benchmarks Securities Technology Analysis Center (STAC ) is a vendor-neutral, benchmarking organization comprised of leading end-user firms and vendors in financial markets The STAC-M2 Benchmark specifications Reflect the input of seven leading trading firms and six vendors of high-performing messaging. Tests the ability of a solution such as messaging middleware to handle real-time market data. Provides key performance metrics such as throughput, latency, and CPU/memory consumption under several scenarios. Requires tuning to achieve a balance between latency and throughput without any packet loss. Tests varied producer and consumer configurations using 16 servers. Benchmark Result Methodology Only Audited results can be published. Audit is a 5 9 day process where Benchmark tests are run in presence of Auditor. One restart of testing is allowed on failure. Unaudited results cannot be published and are confidential. 15

16 STAC-M2 Market Data Benchmark Comparison STAC-M2 Benchmark BASELINE Test Comparison (1 Producer, 5 Consumers) Mean (us) 99P (us) STDV (us) Highest Supply Rate (msg / sec) IBM LLM / IBM-BNT 8264 / Mellanox ConnectX-2 RoCE ,400,000 IBM LLM / IBM-BNT 8264 / Solarflare w/ OpenOnload ,500,000 IBM LLM / Juniper QFX 3500 / Solarflare w/ OpenOnload ,500,000 (Required additional EX4500 as IGMP Querier) IBM LLM / Voltaire IB / Mellanox ConnectX IB ,000,000 29W LBM / Cisco N5010 / Solarflare w/ OpenOnload ,300,000 29W LBM / Cisco 4900M / Solarflare w/ OpenOnload ,300,000 IBM BNT RackSwitch G8264 with LLM and ConnectX-2 RoCE delivers the best ever (STAC Published) latency performance System Configuration IBM WMQLLM 2.6 IBM 3550 M2: 2 x Intel Xeon (X5570) Quad Core 2.93 GHz. Linux RHEL 5.5 MLNX CX-2 with RoCE, OFED IBM-BNT Rackswitch G / 40 GbE

17 Reflector Test Latency: IBM BNT G8264 and WMQ LLM LLM Latency using IBM BNT G /40 GbE and Solarflare SFN5122F OpenOnload LLM Latency using IBM BNT G /40 GbE and Mellanox CX-2 RoCEE Msg Rate [msgs/sec] Average [µsec] Single Hop 99P [µsec] RTT Std Dev [µsec] Msg Rate [msgs/sec] Average [µsec] Single Hop 99P [µsec] RTT Std Dev [µsec] 10, , ,000, , , ,000, Additional IBM Reflector Tests show a 10GbE solution from IBM and its partners delivers extremely low latency performance that scales to very high message rates.

18 Sources of Latency by Network Layer More Latency Application Presentation Session Data 4 TCP windowing Flow control Packet re-send Segments 3 Address lookup Packet forwarding Routing Packets 2 Store-Forward Line coding Switching Frames 1 Framing FEC Bits Less Latency

19 M. Loro and J. Gerrity, Optical networks for low latency applications, Ciena webinar and white paper, March 29, 2011

20

21 T. Bundy et.al., Virtualized converged data centers in the cloud and their effect on optical n Internet2 spring workshop, Arlington, VA (April 2011)

22 Optical transport equipment XPDR AMP ReGen AMP XPDR Potential sources of latency Color conversion Inline amplification Dispersion compensation Electrical regeneration T. Bundy et.al., Virtualized converged data centers in the cloud and their effect on optical networks Internet2 spring workshop, Arlington, VA (April 2011)

23 Low-latency color conversion XPDR AMP ReGen AMP XPDR Function converts grey signal to color channel aggregates lower speed data into single signal Sources of latency Electrical TDM multiplexing Forward-Error-Correction algorithms Performance monitoring and protocol conversion Clock recovery

24 Low-latency inline amplification XPDR AMP ReGen AMP XPDR Function amplifies signal optically, but adds noise many different types to choose from Raman amps preferred to EDFAs Sources of latency Integrated doped fiber spools will add latency Bad noise figures require higher numbers of amplifiers Low gain figures require higher numbers of amplifiers More stages add more doped fibers

25 Low-latency dispersion compensation XPDR AMP ReGen AMP XPDR Function corrects for chromatic dispersion in fiber Uses fiber with special dispersion characteristics (DCF) Sources of latency Dispersion compensating fiber adds more fiber distance If placed in the wrong location DCFs require higher number of amplifiers

26 M. Loro and J. Gerrity, Optical networks for low latency applications, Ciena webinar and white paper, March 29, 2011

27 M. Loro and J. Gerrity, Optical networks for low latency applications, Ciena webinar and white paper, March 29, 2011

28 G.709 wrapper adds latency Should be disabled for ultra low latency networks (like FEC) Turn off OTN to facilitate vendor proprietary ISLs (Cisco, Brocade) and for jumbo frame support Constant bit rate mode is fully transparent to ISLs A unified, multi-purpose architecture

29 Low-latency regeneration XPDR AMP ReGen AMP XPDR Function electrically regenerates signal may add jitter different types to choose from (2R, 2.5R, 3R) Sources of latency Termination / new setup of FEC Non-optimized signal routing / two cards for regen Patch cables from/to filters and regenerator cards

30 Filter Filter Building a smarter planet: Financial Markets Trading Network Example Standard setup Channel Card Transmission Fiber Dispersion Compensating Fibers In F E C D C F D C F D C F Filter F E E D C Out 100ms Amplifier 100ms 10ns 100ns 100ns 1000ms due to equipment Low-latency setup Channel Card Transmission Fiber Dispersion Compensating Grating In FEC off D C G FEC off FEC off D C G Filter E FEC D off C Out Regenerator 10ns 10ns 100ns 10ns 1ms due to equipment

31 M. Loro and J. Gerrity, Optical networks for low latency applications, Ciena webinar and white paper, March 29, 2011

32 Traditional vs. Trading Networks Traditional Services Low-Latency Trading Network Indirect path with multiple hops Spools of DCF EDFA amplifiers at every node Minimal regeneration Frequent transitions up and down the OSI protocol stack Direct path with minimal or no hops FBGs instead of DCF Raman amplifiers Frequent regeneration Stay at Layer 1 avoiding frequent stack transitions

33 Additional Information IBM-BNT Rachswitch G8264: ibm.com/systems/x/options/networking/bnt8264 WebSphere MQ Low Latency Messaging: ibm.com/software/integration/wmq/llm/ STAC Reports stacresearch.com/reports Low Latency Solution Stack for HFT bladenetwork.net/userfiles/file/wp_ibm_solarflare_g8264.pdf * Links clickable only in screen show mode 33