Precision timing in the finance industry

Transcription

1 Precision timing in the finance industry Use cases, problems and suggestions NYSE Euronext. All Rights Reserved.

2 Overview Overview of NYSE Euronext IT infrastructure Use cases in a typical market environment Issues and observations Suggestions to the industry 2

3 400+ years of business and technology firsts We are the inventors of financial exchanges Amsterdam, founded 1602, is the world s oldest exchange Our markets in Brussels, Lisbon, NY and Paris and trace their roots to the 18 th and 19 th centuries We have over 16 centuries of combined experience running financial exchanges And we have a rich history of innovation in markets NYSE was the first exchange to use a stock ticker to disseminate data (1867) NYSE was the first to have telephones on the trading floor (1878) We had automated quotation services 18 years before Nasdaq was founded (1953) First electronic ticker display board (1966) First options exchange in Europe launched in Amsterdam (1978) First real time multicast electronic trading system launched in London (1989) Amex pioneered the Exchange Traded Fund (1993) Wireless handheld devices on NYSE s floor 15 years before ipad invented (1995) Euronext becomes first pan-european equities market (2000) 3

4 Scale of the Business The total market cap of all our listed companies is over $24 Trillion. Compared to country GDPs, that would make us the largest economy in the world. Largest Economies by GDP (in Trillions of $) US - $15 tr China - $10 tr Japan - $4 tr India - $4 tr Germany - $3 tr $ 24 tr NYX Issuer Mkt Cap 4

5 Scale of Transactions In the U.S., NYSE Euronext exchanges execute $70.8 billion in financial transactions every day ebay handled $62 billion all of last year. NYSE Euronext $70.8 billion daily in 2010 Ebay $62 billion in whole of

6 Message Activity In the U.S. alone, NYSE Euronext process more than 22.4 billion messages each day That s 7.5 times more than the number of internet searches Google handles daily. Number of Messages/Transactions per day NYX daily messages U.S. only 6.5 hour/5 trading days per week 22.4 billion messages daily Google daily searches 24/7 6

7 Total number of Trades in Equity Shares +700% Total number of trades in equity shares (in millions) Americas Asia - Pacific EAME Total Data Source: World Federation of Exchanges ( 7

8 120 Average value of trade - USD thousands (weighted by share value trading) Americas Asia - Pacific EAME WFE Average Data Source: World Federation of Exchanges ( 8

9 Approximate North American Options Data (Messages per Second)

10 Bandwidth Distributed to the Financial Community (Terabits/Sec)

11 Technology Foot Print Network Network devices (Routers, Switches, etc) 6,000 1G Ports or below 200,000 10G ports 10,000 Route table 500,000 Entitlements 40,000 Telco Circuits 20,000 Processing Servers 10,000 Cores 60,000 Storage 10 PB All devices need to be synchronised with various degrees of precision 11

12 Typical real time trading network characteristics High availability no single point of failure Verifiable resilience/redundancy down to cable duct level where necessary Technical and commercial redundancy typically dual vendors Small change control windows Very low packet loss every packet counted Bandwidth availability measured on a sub- second basis not a 15 minute average. We measure 1ms microbursts currently. Bandwidth sized to cope with busiest market days All production traffic requires highest priority Low latency Deterministic routing and latency of primary and repair paths Extensible/Expandable Global Low time to market Cost effective 12

13 Typical use cases 13

14 Use case 1 Precision timing to co-lo clients Co-location clients are typically users who place their algorithmic trading servers in our data centre in close proximity to our trading engines. Users need precision timing for measuring transaction round trip and other KPIs, server syntonisation and anything else they wish. Currently we offer the following services GPS we have 3 aerials on each data centre with a very large L1 distribution system that has been calibrated at each distribution point so users can add offset to the aerial to their GPS devices. IRIG-B a legacy protocol but still useful, not many users NTP still useful for general purpose use and can provide very good precision with the right client SW. PTP is under consideration. Although users can provide their own PTP GM from the GPS system some users would rather just have a service. 14

15 Use case 2 monitoring network latency The exchange sell network services both over the wide area network and to co-location clients. Both need to have network KPIs such as latency and jitter monitored, jitter is not appreciated by algorithmic traders. Locations along the path are tapped and data fed into data analysis devices such as those from Corvil The analysis devices use NTP for ToD and a GPS sourced PPS feed to provide precision Relative time is then reported between any monitored locations on the network path Measuring device precision is sub 10µs Most vendors have PTP synchronisation on their immediate roadmap 15

16 Use case 3 trading system time stamping Transactions are time stamped each time they are processed through stages of the trading system The timestamps are preserved in the final response to clients Clients can ascertain overall round trip time and system dwell time from these timestamps As message rates increase timing precision must improve Syntonisation between servers very important otherwise negative time deltas can result Sub 100µs precision is now required, preferably sub 10µs. Synchronisation to UTC to a similar precision also required, this assists clients to monitor transaction round- trip times 16

17 FIX inter-party latency - latency measurement points (LMP) FIX Financial Information exchange is a standard messaging specification used by many banks, brokers, exchanges etc. FIX inter-party latency group have proposed the above reference architecture More details here 17

18 Use case 4 distributed systems Some data centre locations do not have timing services so we have to use our network to deliver precision timing SLA sub 100us to UTC although 10us would be better. Traditionally trading systems are usually in one data centre with disaster recovery sites not too close by so loosely coupled do not need syntonisation of servers between sites. New commercial services associated with Cloud type delivery means that syntonisation between physically disparate systems is becoming more important. No SLAs from the business yet but a guess at 10us for locations with timing services and sub 100us for sites without. All SLAs will be under pressure as message rates increase. 18

19 Use case 5 Research and development 1. Provide timing to servers for syntonisation to allow application latency testing where TSC (time stamp counter) not sufficient i.e. one way not round trip measurement required 2. Testing internal application latency as a message traverses the software stack 3. Software throughput and performance testing requiring high granularity of timestamps especially when establishing exact order of published messages 4. Developing archival data solutions requiring precise timestamping of recorded data 5. Developing monitoring systems relying on timestamped data when passive recording / timestamping not easily available 19

20 GPS distribution

21 Issues and observations 21

22 PTP deliveries to date SW client based synchronisation in data centres OC circa 10us precision comparing SW client to a HW client in the servers that is directly connected to a GM No specific network assistance, no boundary clock or transparent clock or dedicated network. Client synchronisation across the WAN Some of our customer access centres are located in Vendor Neutral Carrier Hotels (commonly called PoPs) that do not provide any timing source. Have achieved 40us precision over our WAN between a GM in our central DC and a hardware PTP appliance in the remote site. Target was sub 100us so pleased with the result however 10us would be appreciated Boundary clock in remote sites desirable. 22

23 Suggestions to the industry TCO hindering wide scale roll out of precision timing in data centres Adding a dedicated timing card to the server not a good option. Cost of cabling, ports, server slot space etc. etc. Willing to pay the extra to the replace the onboard server oscillator with a TCXO/OCXO as required so software solutions more relevant GPS jamming/spoofing needs to be addressed Debatable whether IEEE 1588v2 transparent clocks will give appreciable cost/performance benefit until sub 1us precision required Boundary clocks definitely needed for scalability unless GM performance improves, also has additional precision benefits. Authentication from standard not implemented co-lo clients may require this. May need to use 2-step mode for this. No standardised status feedback from clients, needs to be integrated into PTP and reported back via SNMP (sync status and offset from master) 23

24 More detailed 1588 issues Client software available targeted towards telco or instrumentation / control systems, not enterprise: GM vendors, don't fully understand software-only client use cases. Existing client software tries to implement the same mechanisms that work with HW clients. Software-only clocks make it hard to reflect clock states present in HW clocks: freerun, holdover, acquiring, frequency locked, phase aligned. Software at least needs to implement indication of freerun and holdover states. Ideally also be able to indicate lock acquired. Little support for PTP failover in software clients 24

25 More detailed 1588 issues PTP hardware solutions for servers (NICs) are largely a myth: They do not integrate with the operating system well. Some H/W PTP cards do not do anything with the OS clock at all. gettimeofday() and clock_gettime() must still work and with precision HW-timestamping capable hardware is appearing, not all solutions can timestamp unicast packets No widespread use of hybrid mode (multicast-mix mode): typical multicast distribution in enterprise networks is one-way, hence the need for sync and announce to be multicast, but delay request/response unicast (hybrid mode), 25

26 Final points Data centre enterprise vertical is a growing market PTP will become the new NTP in the very near future, some issues but nothing that cannot be solved Financial exchanges do not tend to be led by single vendors. NYSE Euronext perform extensive testing on vendor kit, and invest in own R&D to verify equipment and drive the market Potatoes were first imported widely into Europe in 1588, and you can use a potato to make a clock, is this purely coincidental?

27