BSE Exchange s New Trading Architecture. BSE Market Data Interfaces. Manual

Transcription

1 BSE Exchange s New Trading Architecture BSE Market Data Interfaces Manual Version Date: July 2, 2014

2 Strictly for private circulation only. This document must not be circulated to other users without prior permission of BSE. 2

3 Contents 1 List of abbreviations Introduction Purpose of this document Main audience Data feeds Reference data interface Market data interfaces Interface version number Further reading matter for this topic How to read this document Differences between the interfaces Distribution sequence for BSE EMDI Distribution sequence for MDI / RDI Choosing between the EMDI and the MDI Choosing between the BSE RDI and BSE RDF Overview of the BSE Public Interfaces Infrastructure requirements Trading states Product State Changes Instrument State Changes Overview of the various message types RDI EMDI/MDI What is not included in these interfaces FIX over FAST Freedom of choice Testing Hours of operation/availability of messages FIX/FAST-Implementation Structure of Messages

4 5.2 FAST terminology FAST reset message Presence Map (PMAP) Template ID (TID) Dictionaries Stop bit encoding FAST operators Decoding the FAST-message Transfer decoding Composing the Actual FIX-Message New features in FAST version Data types FAST version 1.1 compatible templates of a typical trading day Start of day operation Receiving reference data via RDI at start of day Receiving reference data file (RDF) at start of day Build the initial order book Build the initial order book with the EMDI Build the initial order book with the MDI Update the order book Update the order book with the EMDI Update the order book with the MDI Recovery Detecting duplicates and gaps by means of the packet header How to recover data via the respective other service (A or B) Delayed packets Missing packets Recovery (EMDI) Recovery (MDI) Various time stamps in BSE and how to use them Time stamps (EMDI)

5 8.2 Time stamps (MDI) Important topics with use cases and examples Reference data messages General reference data rules General structure of the snapshot cycle Counters as part of the market data report message Use case 1: Reference data at the start of the reference data service Use case 2: Reference data after intraday addition of complex instruments Use case 3: Reference data after intraday deletion of a complex instrument Use case 4: Reference data on the next business day Use case 5: Failover or restart of RDI Use case 6: Chronological order of messages for complex instrument creation General order book rules and mechanics Determination of the price sources New price level Change of a price level Overlay Deletion of a price level Deletion of multiple price levels from a given price level onwards Deletion of multiple price levels up to a given price level Trade Volume Reporting (EMDI) Use case 1: Direct match of simple instruments Use case 2: Direct match of complex instruments Use case 4: Complex versus simple/complex match Use case 5: Opening auction Trade Volume Reporting (MDI) Failure of the market data feed/ matching engine Normal processing Market data feed fail-over (EMDI) Market data feed fail-over (MDI) Market data feed restart (EMDI) Market data feed restart (MDI)

6 9.6.6 Failure of the matching engine Trading states for a sample business day Start-Of-Day Pre-Trading Trading Continuous Trading Intraday Expiry Closing Post Closing End-Of-Day Fine tuning client applications Buffer size Packet and message processing Application level Discarding duplicate packets within the live-live environment Order book processing Optimal processing of desired products (EMDI) Detailed data feed description and layout Service messages FAST reset message Packet header (EMDI) Packet header (MDI / RDI) Functional beacon message Technical heartbeat message Market data report message Reference data messages Product snapshot message Instrument snapshot message Instrument incremental message Market data messages Depth snapshot message Depth incremental message

7 Product state change message Mass instrument state change message Instrument state change message Complex instrument update message Index change message Data files Reference data from file (BSE RDF) File name format of the reference data files Reference data file on the next business day Reference data file after a failover or restart of BSE RDI What receiving applications need to do Multicast addresses Reference data snapshot feed Reference data incremental feed FAST templates Appendix Example for a XML FAST template Example for determination of the price source Fully implied (example for 9.3.1, Determination of the price sources) Fully outright on level 1 (example for 9.3.1, Determination of the price sources) Partially implied (example for 9.3.1, Determination of the price sources) Several fully implied orders at Best Market (example for 9.3, General order book rules and mechanics) FIXML mapping table Index Code and Mapping Table Change log Changes compared to version Changes compared to version Changes compared to version

8 Part I 1 List of abbreviations The table below shows all the abbreviations and definitions used in this document. EMDI MDI RDI RDF ETI FAST Enhanced Market Data Interface Market Data Interface Reference Data Interface Reference Data File Enhanced Transaction Interface FIX Adapted for STreaming (FAST Protocol) (FAST ProtocolSM ). FIX Adapted for STreaming is a standard which has been developed by the Data Representation and Transport Subgroup of FPLs Market Data Optimization Working Group. FAST uses proven data redundancy reductions that leverage knowledge about data content and data formats. FIX Financial Information exchange. The Financial Information exchange ( FIX ) Protocol is a series of messaging specifications for the electronic communication of trade-related messages. In-band Match event Out-of-band Simple instruments Complex instruments Live-live concept OTC PMAP ToB Incremental and snapshots are delivered in the same channel. Part of the matching event having a unique match price. Incremental and snapshots are delivered on different channels. Single leg outright contracts Any combination of single leg outright contracts, e.g. Future Time Spreads. The concept whereby data is disseminated simultaneously via two separate channels called Service A and Service B. Over the Counter Presence Map Top of Book 8

9 2 Introduction BSE offers public market and reference data via three interfaces as part of the BSE Exchange s new trading architecture. All three interfaces distribute information via UDP multicast; following FIX 5.0 SP2 semantics and are FAST encoded. If any messages are lost, complete recovery is possible because every message is published on two identical services (A and B) with different multicast addresses (livelive concept). In the unlikely case that a message is lost on both services, participants can take advantage of the respective snapshot message and rebuild the order book. There are two types of Market Data Interface: The Enhanced Market Data Interface (EMDI): This interface provides un-netted market data. The updates of the order book are delivered for all order book changes up to a given level; all on-exchange trades are reported individually. The Market Data Interface (MDI): This interface provides netted market data. The updates of the order book are sent at regular intervals; they are not provided for every order book change and are sent significantly less frequently than the EMDI. Onexchange trades are not reported individually but statistical information (daily high/low price, last trade price and quantity) is provided instead. The EMDI and MDI provide the following information to the participant: Price level aggregated order book depth and on-exchange trade statistics. Product and instrument states. Information on newly created/deleted complex instruments. Reference data is sent with: The Reference Data Interface (RDI): This interface provides reference data for instruments that are available for trading on the BSE Exchange s new trading architecture. The reference data is delivered on a product and instrument level. Every tradable object is referenced by a unique identifier, for this reason the reference data information is absolutely essential for any trading application. The interface is currently unavailable and would be made available soon. The Reference Data File (RDF): Reference data is delivered as a start-of-day file. The file is available on the BSE website. The interface is currently unavailable and would be made available soon. 1 Fast Templates are provided as well. 9

10 2.1 Purpose of this document The purpose of this document is to provide guidance for programmers during development of applications that read the BSE Market & Reference Data Interfaces. It covers a complete reference for the three multicast based public interfaces, describes the general business behavior and provides concepts for the implementation. The most recent version is available at: Main audience The target audience of this interface specification is experienced software developers support staff that may be involved in development/support activities for the BSE Market & Reference Data Interfaces. Prior knowledge of developing for a capital market is beneficial but not a prerequisite. Knowledge in a programming language is expected. Programmers who have no experience in a market data interface environment can gain a basic understanding of the feed behavior by reading Part II (How to guide). This manual does not attempt to cover basic knowledge of programming techniques and software development. 2.3 Data feeds All interfaces deliver public reference and market data in the form of snapshots and incremental as can be seen in Figure 1. The two public market data interfaces, the EMDI for a high bandwidth network and the MDI for a low bandwidth network, disseminate information across the BSE network to the receiving application. The RDI is considered for participants with a high bandwidth network while the RDF should be used if only a low bandwidth network is available. 10

11 Figure 1: Reference data and Market data interfaces Reference data interface Public reference data delivered by RDI contains the technical configuration, e.g. Multicast address and port combinations for both market data interfaces for all products and instruments. Multicast addresses and port information do not change during trading hours. The reference data snapshot feed contains two message types: Constant number of snapshots and a variable number of incremental. The reference data incremental feed delivers reference information about intraday created and deleted complex instruments. 11

12 2.3.2 Market data interfaces The EMDI and the MDI disseminate public market data information in the form of incremental (event driven) and snapshots (time driven). The market data snapshot feed can be used to recover lost market data or build up the current order book. Receiving applications are not expected to be permanently subscribed to this feed. The market data incremental feed should be subscribed throughout the trading day for receiving order book updates. All incoming messages should be applied to the copy of the order book maintained by the member applications in order to have the latest information. 2.4 Interface version number Each of the interfaces described in this manual has a version number. The version numbers are also listed within the FAST XML templates. This manual relates to the following interface version numbers: EMDI: MDI: RDI: The version numbers for the interfaces are available at the beginning of the FAST XML files. 2.5 Further reading matter for this topic This document is designed as an independent learning and reference manual. However, for background information related to network connectivity, FAST/FIX messages or trading related information (functional), further documents are recommended. The documents listed below provide useful information. FAST- and FIX-related documents: FAST specification documents: Explains all FAST rules in detail. FAST 1.2 is the summary of the FAST 1.1 specifications plus the extension Proposal. > fastspec FIX specification documents: FIX-messages and FIX-tags > Specifications 12

13 FIX-Tags: Specifies all FIX-Tags > FIXimate How to read this document This manual covers the EMDI and MDI as well as the RDI. Differences in functionality between the EMDI and the MDI are described in separate sub sections, while being represented by different text colors: (EMDI) and (MDI). For example, section 7.4.2, Recovery (MDI), refers to the netted MDI only. Participants who are interested in the un-netted EMDI can ignore this sub chapter. This document consists of three parts: Part I (General Overview) introduces the interface for beginners. Part II (How to guide) provides methods and hands-on guidance. Part III (Reference) is a comprehensive reference with details on various message layouts in table format. A typical table would be the following: Delivered on: reference data snapshot feed Tag Field Name Req d Data Type 35 MsgType Y string User Defined Message 0 Beacon <Group Name> (Optional) group starts <Sequence Name> Sequence start <Group Name> (Optional) group starts <Sequence Name> Sequence start Table 2: Typical FIX message description Interpreting the fields above: Delivered on: Specifies the feed which delivers the specific message. A message can be delivered on more than one feed. 13

14 Tag: Describes the FIX Tags Field Name: Describes the FIX-name. Req d: Describes whether or not the field is included within the message after FAST-decoding, purely from the FIX-point of view. This does not refer to a FAST-rule, e.g. Operators or Presence Map (PMAP) in FAST. Data Type: FAST data type. This information is also provided in the XML FAST templates. : This column contains an explanation of the FIX-field and its valid values in table format for this particular message. Group Name, Sequence Name: The names correspond with the groups and sequences defined in the FAST XML templates. Cross references to other chapters within this document and the glossary are provided in blue color. Example: More information is provided in section 9.1, Reference data messages. In this document, the terms incremental and snapshots are used in various contexts. Within this document incremental and snapshots refer either to messages of the market data feed or to messages of the reference data feed. The actual meaning can be inferred from the context. Note: Important statements made in this manual are highlighted with a shadow box. 14

15 3 Differences between the interfaces A feed is a message flow of logically grouped messages, e.g. The depth incremental and product state change messages for a particular product are grouped together within the incremental feed of EMDI. The following diagram illustrates the available feeds for the three multicast based public interfaces: Figure 2: Overview of the three interfaces The RDI is published on exactly one snapshot channel, indicated by (A 1 S) and one incremental channel (A 1 I ). The EMDI has multiple channels that have either snapshots (A 1 S ) to (A n S ) and multiple incremental channels (A 1 I ) to (A n I ). The MDI has the snapshots and incremental combined over multiple channels (A 1 S,I ) to (A n S,I ). The snapshot and incremental messages for the EMDI are delivered via separate feeds (out-of band) and need to be synchronized. Each feed consists of several channels, each of which delivers the information for a group of products. Several partitions, each with a unique SenderCompID (49), may contribute to the same multicast address as shown in Figure 20. The SenderCompID (49) is unique across all partitions. However, it should not be relied upon as under unlikely but possible conditions on the exchange this is not true. 15

16 In contrast to the EMDI, the snapshot and incremental messages for the MDI are sent on one feed only (in-band), therefore there is no need to synchronize both messages. The feed is also divided into several channels grouped on product basis. The snapshot and incremental feed for the RDI are delivered via separate channels (out-of band) and need to be synchronized. In contrast to the order book information, the snapshot and incremental feeds are not divided into further channels. All feeds are sent on two different multicast addresses via different physical connections (Service A and B). Service A and Service B are identical in terms of the information provided, i.e. The packet contents, sequence numbers and sequence in which packets are sent are the same. This is called live-live concept. Product groups are distributed across several partitions on the BSE backend side. Service A and Service B cannot be published at exactly the same time. 3.1 Distribution sequence for BSE EMDI The rule for the distribution sequence across partitions is as follows: Even partitions: Publish on Service A first, then on Service B. Odd partitions: Publish on Service B first, then on Service A. The above rule is applied by using the field PartitionID (5948). It is available in the product snapshot message, in reference data file and in the packet header and contains the number of the partition for the product of interest. The PartitionID (5948) never changes intraday. Example: A PartitionID = 8 indicates an even partition and therefore Service A is published before Service B. The time difference between publication of Service A and B is currently not known; lab testing indicates an average time difference of about µs; the cable length for both Service A and Service B within the co-location is the same, i.e. both services have the same propagation delay. The multicast addresses for both of these services are disseminated in the product reference information. Due to the inherently unreliable nature of the UDP protocol, data packets may be lost in the transmission network. Therefore members are advised to join both services to reduce the probability of data loss. 3.2 Distribution sequence for MDI / RDI The rule for the distribution sequence across partitions is as follows: Even and odd partitions: Publish on Service A first, then on Service B 16

17 Example: The PartitionID (5948) for MDI and RDI is not available in the packet header but in the product snapshot message. However, the PartitionID (5948) doesn t need to be considered because Service A is always published first regardless of the partition. 3.3 Choosing between the EMDI and the MDI Both types of interface, un-netted and netted, provide market information via multicast using a price level aggregated order book (as opposed to, for example, order-by-order feeds) but they have different bandwidth requirements and service levels. The Enhanced Market Data Interface (un-netted) disseminates every order book change up to the configured depth and all on-exchange trades without netting. This interface is designed for participants that rely on low-latency order book updates and data completeness. The un-netted market data is partitioned over several channels; each channel provides information about a group of similar products. As the market becomes busier, the number of messages (and therefore bandwidth usage) increases. The Market Data Interface (netted) has a lower bandwidth requirement compared to the un-netted version. This interface is designed for participants who do not need to see every order book update, this has the advantage of keeping the infrastructure costs low. Snapshot and incremental updates are sent via the same IP multicast address and port combination. This interface aggregates the order book changes over a specified time interval. Currently, BSE plans to provide market data for benchmark products with a netting interval of 0.47 sec and depth of 10. For all other products, a netting interval of 0.67 sec is envisaged. This interface has less price levels than the EMDI. Furthermore, only statistical information is provided for on-exchange trades as well as the price and quantity of the last on-exchange trade in the netting interval. The following table shows the main differences between the EMDI and the MDI: Area EMDI MDI In-band/Out-of- band delivery Incremental and snapshots are delivered via different channels, i.e. out-of-band delivery LastMsgSeqNumProcessed In the snapshot feed provides a link between incremental and snapshot feed, as it carries the sequence number of the last message sent on the incremental feed. Snapshots are 17 Incremental and snapshots are delivered on the same channel, i.e. in-band delivery. Snapshots might contain new information. A flag (RefreshIndicator) within the snapshot indicates whether it has to be applied or not. LastMsgSeqNumProcessed is not used.

18 Sequence numbers on message level needed only for startup/recovery. Messages on the market data incremental feed have their own sequence number range per product; MsgSeqNum s exist on the depth incremental feed only as shown in Table 13. Messages on the combined market data incremental + snapshot feed have one sequence number range per product as shown in Table 14 Trade Volume Reporting Trade Volume Reporting is provided. Each on-exchange trade is reported individually. Packet header A Performance Indicator 4 is provided for incremental within the Packet Header as shown in figure 21 Functional beacon message A functional beacon message on a product level including the last valid MsgSeqNum is sent if no other message has been sent for a configured time period. Only statistical information (daily high/low price and total traded quantity) and last trade information is provided. A Performance Indicator does not exist as shown in figure 22 Snapshots act as functional beacon message, hence no separate functional beacon messages are provided. Table 3: Main differences between the EMDI and the MDI 3.4 Choosing between the BSE RDI and BSE RDF The Reference data is provided via the RDI and in file form as compressed Reference Data Files (RDF) in FIXML-layout, updated approximately every 5 minutes via the Common Report Engine 5 (CRE). The initial reference data file generated at start-of-day contains the reference data snapshots available from the previous day. During the actual trading multiple incremental files are created as complex instruments are added and deleted. New complex instruments predefined by the exchange are also sent in incremental files before the start of the actual trading. Please note that the intraday changes to reference data are also published in form of the complex instrument update messages via the market data incremental feed of the EMDI and the market data feed of the MDI. During normal operations participants do not need to listen to the incremental feed of the RDI, because the complex instrument update 6 can be received on the market data feed. Furthermore, market data for new complex instruments is never provided ahead of their reference data on EMDI or MDI but may come ahead of its publication via RDI. 18

19 Participants have the choice between the two different reference data sources. However, it is assumed that bandwidth conscious users will use BSE RDF for start-of-day processing and intraday re-starts. The reference data file is provided once the system is available (product state Start-Of-Day ). The following table shows the main difference between the BSE reference data messages and the reference data from File: Area RDI: Message based RDF: File based Reference Data High bandwidth users can use the multicast based Reference Data Interface. Low bandwidth users can use Start-Of-Day Reference Data Files and apply each Intraday Reference Data File as they become available. A late starting application can always retrieve the latest picture of the reference data by this method. Table 4: Differences between the RDI and the RDF 4 Time between arrival of an incoming order/quote transaction on the BSE matching engine and send time of the corresponding outgoing market data 5 For more information please see the "Common Report Engine User Guide" 6 The layout is the same as on the instrument incremental message except for the fields SecurityStatus (965) and SenderCompID (49) 19

20 4 Overview of the BSE Public Interfaces This chapter describes the public market data provided by the market- and reference data interfaces. 4.1 Infrastructure requirements The BSE market and reference data interfaces disseminate market and reference data over the BSE multicast network. A router which is capable of handling IP multicast is required for accessing this interface. The multicast management protocol is IGMPv2. When utilizing IGMPv3, the IGMPv2 compatibility mode must be enabled. 4.2 Trading states State changes are disseminated over both the EMDI and the MDI market data feeds. Trading state information is not communicated over the Enhanced Transaction Interface (ETI) or FIX interface. The EMDI and the MDI market data feeds follow the FIX protocol for the publication of trading state information. The BSE product and instrument states are displayed by these interfaces as shown in the following tables. Section 9.8, Trading states for a sample business day illustrates state messages for a typical business day. The hours of operations for the BSE system are provided in Section 4.8, Hours of operation/availability of messages Product State Changes The product state is published with a product state change message (FIX TradingSessionStatus, MsgType = h). In this message, the product state can normally be found in the field TradingSessionSubID (625). Only for quiescent product states, the field TradingSessionID (336) must be evaluated additionally to determine the actual product state. Product State Change Message Product State FIX TradingSessionID (336) FIX TradingSessionSubID FIX TradeSesStatus (625) (340) Start of Day 3 = Morning 9 = Quiescent 3 = Closed Pre-Trading 3 = Morning 1 = Pre-Trading 2 = Open Trading 1 = Day 3 = Trading 2 = Open Closing 1 = Day 4 = Closing 2 = Open Post-Trading 5 = Evening 5 = Post-Trading 2 = Open Post- Closing 5 = Evening 5 = Post-Closing 2 = Open End of Day 5 = Evening 9 = Quiescent 3 = Closed 20

21 Halt 1 = Day 9 = Quiescent 1 = Halted Holiday 7 = Holiday 9 = Quiescent 3 = Closed Table 5: Product states A Halt state is additionally indicated by the FIX field TradSesStatus (340) containing the value 1 = Halted Instrument State Changes The instrument state is published with an instrument state change message (FIX SecurityStatus, MsgType= f) in case of a single instrument, or with a (FIX SecurityMassStatus, MsgType = CO) message in case that all or most of the instruments of a product and of a specific instrument type 7 change their state. In the instrument state change message (FIX SecurityStatus, MsgType= f), the instrument state can be found directly in the field SecurityTradingStatus (326). In the mass instrument state change message (FIX SecurityMassStatus, MsgType = CO), the instrument state can be found in the field SecurityMassTradingStatus (1679). This message may contain an exception list of instruments that have a different instrument state. The exception list contains the instrument state in the field SecurityTradingStatus (326) for each of these instruments. Instrument State instrument state change message / mass instrument state change message FIX SecurityTradingStatus (326) / FIX SecurityMassTradingStatus (1679) Closed 200 = Closed Restricted 201 = Restricted Book 202 = Book Continuous 203 = Continuous Opening Auction 204 = Opening Auction Opening Auction Freeze 205 = Opening Auction Freeze Intraday Auction 206 = Intraday Auction Intraday Auction Freeze 207 = Intraday Auction Freeze Table 6: Instrument states The field FastMarketIndicator (28828) is also contained in the mass instrument state change message; each instrument state message also contains the information about whether the product that the instrument belongs to is in a Fast Market state. This implies that a mass instrument state change 21

22 message is sent when a product is set to Fast Market (or back) without a change in the instrument states. The status of the instrument (as opposed to the instrument state) distinguishes active and published instruments and is contained in the field SecurityStatus (965). 7 Instrument types distinguish simple instruments (option series, futures contracts) and various types of complex instruments. 22

23 4.3 Overview of the various message types The various message types can be divided into "Service Messages" and "Data Messages" RDI Service messages: Technical heartbeat message is sent out periodically by the BSE system on every multicast address and on a specific port assigned for the technical heartbeat; it consists of a FAST reset message only. The purpose of the heartbeat message is network related only 8. Data messages: Market data report message flags the start and end of reference data. Each message is flagged by a start/stop event identifier. Product snapshot message contains product specific reference data. Instrument snapshot message contains a snapshot of instrument specific reference data and also contains the reference data related to complex instruments that existed at start-of-day. Instrument incremental message used with intraday added or deleted complex instruments. Identical messages are also sent on the market data incremental feed of the BSE EMDI as well as on the market data feed of the BSE MDI EMDI/MDI Service messages: Technical heartbeat message is sent out on all multicast addresses of the EMDI/MDI. The description is the same as for RDI. Functional beacon message (EMDI) contains the last valid MsgSeqNum of each product and is only sent on the market data incremental feed when there is no activity in a product for a certain amount of time. No functional beacons are sent for the MDI because the snapshots act as a functional beacon. 8 It is used to keep spanning tree alive 23

24 Data messages: Depth snapshot message is used to send a snapshot of all price levels of the order book and statistical information about on-exchange trades. This message can be used whenever the order book needs to be rebuilt. Depth incremental message is used to receive updates on the initial order book. Product state change message is used to publish the state of the BSE products. Mass instrument state change message provides the state information for all instruments of a product. This message can publish different states for instruments of the same product, e.g. In case of volatility interruption the front month could be in a different state than the back month. Instrument state change message provides state information for a single instrument. Complex instrument update message is used to publish new or deleted complex instruments. This message is sent via the market data incremental feed of the BSE EMDI and the market data feed of the MDI. A message is sent for each newly created complex instrument. Index Change Message is used to publish the indices current values and also day s high, low open and close values. This message is sent via the market data incremental feed of the BSE EMDI and the market data feed of the MDI. The message is sent periodically at a defined interval. A detailed description of the message types listed above is given in section 11, detailed data feed description and layout. 4.4 What is not included in these interfaces The following information is not provided via the new interfaces: For auctions, the best bid/ask prices are disseminated at price level 1 without a quantity. If a potential auction price is calculated, it is also sent without the quantity. Order book depths are not delivered during auctions, only top of book information is disseminated. 24

25 Market Supervision News is not provided. This information is available via the BSE ETI in recoverable form. In case of derivatives contracts, the prices for underlying are not provided. These prices will be available in multicast form in the multicast stream of the respective segment. Retransmission functionality is not provided, but recovery is possible from the respective other service (A or B). In case a message is lost a snapshot can be used to rebuild the order book. Implied prices are only sent for Best Market, they are not sent for the order book depth except for top of book. 4.5 FIX over FAST FIX messages are sent out in FAST 1.2 encoded format. The receiving software decodes the FAST messages according to the FAST 1.2 rules. Note: FAST 1.2 templates and FAST 1.1 compatible templates are provided. After the decoding process, the actual FIX message can be built by applying the FIX structure to the decoded message. The detailed process is shown in Part II, FIX/FAST-Implementation. Participants need a standard FAST template based decoder in order to be able to use the EMDI, MDI and RDI. Alternatively participants can use their own FAST decoder implementation. 4.6 Freedom of choice BSE does not need to provide any software for accessing the services offered. The BSE market and reference data interfaces can be accessed using any platform capable of receiving multicast data feeds. Participants can use any operating system, compiler version or programming language in order to develop or use specific third party applications that are tailored to their requirements. 4.7 Testing It is recommended to test the functionality application logic sufficiently in a simulation environment. Receiving applications must be able to cope appropriately with a variety of BSE service fail-over scenarios. 25

26 4.8 Hours of operation/availability of messages The product state Pre-Trading (TradingSessionSubID (625) = 1) for many of the BSE products begins at 6:30 IST. Post-Trading (TradingSessionSubID (625) = 5) for some BSE products lasts until 22:30 CET. BSE is available from approximately 6:30 IST. It is recommended to start applications between 6:30 IST and 7:00 IST. Market data messages are sent from the time a product changes to the state Start-Of- Day and stops when it changes to the state End-Of-Day. During that period depth snapshots are sent. The reference data is independent to any one product state so it has its own schedule. Receiving applications are expected to stay connected from product state Start-Of- Day until pro- duct state End-Of-Day. The following table provides further details about the availability of messages per instrument state: State Market Data Orderbook Market Data State Info Continuous Yes Yes Auction Yes Yes Freeze Yes Yes Book No Yes Restricted No Yes Closed No Yes Table 7: Availability of messages per instrument state 26

27 Part II How to guide 5 FIX/FAST-Implementation This chapter describes the message structure for the three interfaces. It also provides the basic FAST rules used by the interface and describes the basic steps from receiving a FAST datagram, decoding it and building FIX-messages out of it. The FAST 1.2 specification is provided as an extension to the FAST 1.1 specification. The documents can be found under the following links: FAST Specification (Version 1.1) > Technical Specifications > FAST Protocol > FAST Protocol Specifications > FAST Specification Version 1.1 FAST version 1.2 Extension Proposal > Technical Specifications > FAST Protocol > FAST Protocol Specifications > FAST Extension Version Structure of Messages The three public interfaces disseminate data in UDP datagram in network byte order also known as big endian byte order. This includes vector encoded numbers. A UDP datagram has the following structure: Figure 3: Structure of a UDP datagram The UDP datagram starts with the packet header message as shown in section Followed by a FAST reset message. Followed by the actual message (Message1). Possibly followed by one or more messages (Message 2 -Message n ). 27

28 Each message shown in the picture above has the following sub structure: PMAP (Presence Map). TID (Template ID). Data Part. This is shown in the following diagram: Figure 4: Structure of consecutive messages within one datagram One UDP datagram contains one or more FAST encoded FIX 5.0 SP2 messages. The UDP protocol adds a 28 byte header to every packet (20 byte IP header plus 8 byte UDP protocol header). Due to the unreliable nature of UDP, every UDP datagram is self contained; there is no dependency across datagram. 5.2 FAST terminology FAST reset message The BSE Market Data Interfaces use global dictionary scope for FAST operators 9. All operators share the same dictionary regardless of the template and application type. The FAST reset message is inserted at the start of every datagram to explicitly reset all the dictionaries Presence Map (PMAP) The presence map is a bit combination indicating the presence or absence of a field in the message body, one bit in the PMAP for each field that uses a PMAP bit according to the FAST type. The allocation of a bit for a field in the presence map is governed by the FAST field encoding rules. 9 The dictionary scope should always be derived from the template definition. 28

29 5.2.3 Template ID (TID) The template identifier is represented by a number (integer) and points to a specific FAST template which describes the layout and characteristics of the message to be decoded. The FAST XML files are provided in section 13, FAST templates. FAST uses templates to reduce redundancies within a message by using the following methods: The order of fields within the FAST message is fixed, so the field meaning is defined by its position in the message and there is no need to transfer the field tag to describe the field value. The templates specify the order and occurrence of message fields like type, presence and operators. The following list contains the message types and their corresponding template identifiers used with the three BSE interfaces: Message TID RDI TID EMDI TID MDI Functional Beacon 109 Packet header for RDI / EMDI / MDI FAST Reset Message MarketDataReport 125 ProductSnapshot 122 InstrumentSnapshot 123 InstrumentIncremental 121 ComplexInstrumentUpdate DepthSnapshot DepthIncremental ProductStateChange MassInstrumentStateChange InstrumentStateChange Table 8: Template identifiers for RDI/EMDI/MDI Note: The template id for the packet header will increase in future releases and can be used to identify the software release. Example: The TID=116 indicates the packet header for MDI in the current release. In the next release the TID for the packet header would increase to the next available integer, i.e. TID=

30 5.2.4 Dictionaries A dictionary is a cache in which previous values are stored. FAST operators (-> 5.2.6) make use of the previous values Stop bit encoding Most FAST fields are stop bit encoded; each byte consists of seven data bits for data transfer and a stop bit to indicate the end of a field value. An exception from this rule is Byte Vectors as they are used in the packet header of EMDI/MDI/RDI FAST operators Field operators are used to remove redundancies in the data values. Message templates are the metadata for the message and are provided earlier. When the messages arrive, the receiving application has complete knowledge of the message layout via the template definition; it is able to determine the field values of the incoming message. The following FAST operators are used in BSE EMDI/MDI/RDI: delta. copy. constant. default. increment. For more information on the new FAST 1.2 features please refer to: > Technical Specifications > FAST Protocol > FAST Protocol Specifications > FAST Extension Version Decoding the FAST-message The FAST messages need to be decoded by means of the FAST templates. The FAST templates provide all necessary information to decode a message such as data types (e.g. uint32), field names (e.g. Mistype), FIX tags (e.g. 35) And FAST operators (e.g. Increment). The FAST templates also contain information about repeating groups (sequences). A typical example for a XML FAST template with a repeating group is shown in figure 23 of section 14.1, Example for a XML FAST template. 30

31 5.4 Transfer decoding Transfer decoding describes the process of how the fields are decoded from the FAST format. For further information, please refer to section 10 of the FAST Specification Version 1.1. Transfer encoding describes the opposite process. 5.5 Composing the Actual FIX-Message A typical FAST decoder would not deliver FIX messages after the decoding process. In order to compose FIX messages, applications need to apply additional rules. The sequence of FIX-fields after composing the FIX-message on participant s side is not governed by the FIX-layout of the messages, i.e. the fields names of the FIX-message do not need to be in the same sequence. The FIX message, however, needs to fulfill the minimum requirement: BeginString(8) in the Standard Header must be the first tag in the message. BodyLength(9) in the Standard Header must be the second tag in the message. MsgType(35) in the Standard Header must be the third tag in the message. CheckSum(10) Standard Trailer must be the last tag in the message. 5.6 New features in FAST version 1.2 The following new features from the FAST 1.2 protocol are used: New Type Definition Syntax: This allows the separation of the type definitions from the type usage within template definitions. Enumeration: This feature can be used when there is a fixed set of valid values for a single field. Set (multi-value field): This feature can be used when there is a fixed set of valid values which could be sent together as a bit combination instead of using a repeating group. An example for a set would be the field TradeCondition (277) in the Depth incremental message. Sets are used to define the valid values for fields. 31

32 Timestamp Data Type: The use of this feature allows native support of time stamp fields which becomes increasingly important for the BSE market data interface. A time stamp is an integer that represents a number of time units since an epoch. 5.7 Data types The BSE implementation of FAST utilizes the following FAST data types: Decimal Length String uint32/uint64 Byte vector Set Enum Timestamp 5.8 FAST version 1.1 compatible templates Participants who choose not to upgrade their FAST 1.2 decoders can use FAST 1.1 compatible files offered by BSE. The following needs to be considered: Enumerations: As described in the previous chapter enumerations have a list of codes. Participants receive an integer but not the description (meaning) of the integer. Since FAST 1.1 does not support enumerations this description of codes needs to be taken from the valid values provided in the FIX tables, chapter 11, Detailed data feed description and layout. Sets: Similar to enumerations, however, participants receive a bitmap and multiple items from the list. The items need to be taken from the valid values provided in the FIX tables, chapter 11; Detailed data feed description and layout. The FAST version 1.2 Extension Proposal available at > fast spec describes how the encoded field (wire format) value looks. 32

33 Example for enumeration: TradingSessionID (336) can have one of the following values as defined in the FAST 1.2 XML files: <define name="tradingsessionid"> <enum> <element name="1" id="day"/> <element name="3" id="morning"/> <element name="5" id="evening"/> <element name="7" id="holiday"/> <copy/> </enum> </define> The wire format of the values 1, 3, 5, 7 is 0, 1, 2, 3, i.e. each value is represented by an index. Enumerations are not defined in the FAST 1.1 XML files. When the decoder receives a 3 he needs to know that it means Holiday. Example for set: TradeCondition (277) can have one or more values as defined in the FAST 1.2 XML files: <define name="tradeconditionset"> <set> <element name="u" id="exchangelast"/> <element name="r" id="openingprice"/> <element name="ax" id="highprice"/> <element name="ay" id="lowprice"/> <element name="aj" id="officialclosingprice"/> <element name="aw" id="lastauctionprice"/> <element name="k" id="outofsequenceeth"/> </set> The wire format of the values U, R, AX, AY, AJ, AW, k is 1, 2, 4, 8, 16, 32 and 64 i.e. each value is represented by a different bit. The values can be added together to form combinations of the values. If U, AX is sent then = 5 are the encoded field values. Sets are not defined in the FAST 1.1 XML files. When the decoder receives a 5 he needs to know that it is a combination of 1 and 4 which is ExchangeLast and HighPrice. 33

34 6 of a typical trading day This chapter describes a typical trading day, from the start until the end of trading; the following steps need to be taken to prepare for and to receive market data: Figure 5: Typical trading day 6.1 Start of day operation Before processing any market data, receiving applications need to retrieve technical and functional information via the BSE RDI. Alternatively, reference data can be received in file format (Reference Data from File). A detailed description of the reference data feeds and messages is provided in section 9.1, Reference data messages. At start-up, reference data must be processed to create the initial order book baseline. 6.2 Receiving reference data via RDI at start of day At the start of the business day, receiving applications need to join the static multicast address/port of the reference data interface in order to receive the following messages: Product snapshot to receive the functional and technical parameters. Instrument snapshot to receive instrument details. Instrument incremental to receive intraday updates. 34

35 Port information and multicast addresses for the reference data feeds as well as the address ranges for market data are in section 12, Multicast addresses. Port information and multicast addresses for market data feeds are delivered as part of the reference data feeds. Further detailed information about reference data is provided in section 9.2, General reference data rules. However, the basic steps in order to receive reference data are the following: 1. Listen to the reference data incremental feed and start buffering instrument incremental messages. If an application starts listening to the reference data messages early enough, there are no instrument incremental messages available. 2. Listen to the reference data snapshot feed. Ignore all the messages until you reach the market data report message denoting the beginning of a snapshot. Take note here of two values: MDCount, containing the number of reference data snapshot messages in the initial snapshot cycle. LastMsgSeqNumProcessed, containing the sequence number of the last message at the end of the snapshot cycle; this could be a snapshot or an incremental message. Process all messages of the snapshot until you encounter the market data report message denoting the end of the snapshot cycle. 3. At this point you need to complete the list with the messages received on the incremental feed since you started listening. But only after you have discarded all messages having 10 : MsgSeqNum <= LastMsgSeqNumProcessed - MDCount. 4. Store the reference data information for future use. 5. Join the market data incremental feed of EMDI or the market data feed of MDI in order to receive additional reference data changes. 6. Leave the reference data snapshot and incremental feeds. Note: Applications starting early do not require steps 1 and 3 since no incremental message exists at this time. New complex instruments predefined by the exchange are also sent in instrument incremental before the start of the actual trading. 10 The snapshot and incremental feeds have a different sequence number range. 35

36 Note: Participants interested in complex instruments should use the complex instrument update messages. These are published on the market data incremental feed of the EMDI as well as on the market data feed of the MDI. They are published faster than the instrument incremental message of the reference data incremental feed. 6.3 Receiving reference data file (RDF) at start of day Participants with low bandwidth connections may retrieve the start-of-day reference data in a file based format. The initial reference data file generated at start-of-day contains the reference data snapshots available from the previous day. During the actual trading multiple incremental files are created as complex instruments are added and deleted. New complex instruments predefined by the exchange are also sent in incremental files before the start of the actual trading. In case a receiving application starts late, each of the intraday Reference Data Files in addition to the Start-Of-Day Reference Data File must be applied. Start-Of-Day and Intraday Reference Data Files are available via the Common Report Engine. Note: In case a late starting application uses the Start-Of-Day Reference Data File without the intraday files, the intraday created complex instruments remain unknown and hence order book data may be received for unknown instruments. 6.4 Build the initial order book Participants first have to build the initial order book. instrument. The order book has to be maintained per Note: Sequence numbers contained in the market data messages are incremented per product Build the initial order book with the EMDI For each instrument within the desired products do the following: 36

37 Figure 6: EMDI initial order book Build the initial order book with the MDI The following sequence is recommended for the MDI: Figure 7: MDI initial order book 37

38 The field LastMsgSeqNumProcessed (369) in the MDI snapshots can be ignored because snapshots and incremental are sent in-band and don t need to be synchronized with each other. Note: MDI applications must process depth snapshots beside the depth incremental because the snapshots might contain new information. If the RefreshIndicator (1187) is set the depth snapshot contains order book information that has not been sent in a depth incremental. 6.5 Update the order book Every update in the form of a depth incremental or depth snapshot message contains the price level and the actual price to which the instruction needs to be applied. The receiver application can update information at a particular level with the new information. Once participants have built the current order book it needs to be continuously updated: Update the order book with the EMDI As long as the MsgSegNum values for the depth incremental message are contiguous per product do the following 11: Keep applying all depth incremental messages to the current order book. Note: Depth snapshot messages are sent on a different channel as the depth incremental messages. Changes to the order book are also sent using the depth snapshot messages but the information is also provided with the incremental messages. Snapshot messages don t need to be processed unless the order book needs to be recreated Update the order book with the MDI As longs as the MsgSegNum values for the depth incremental message are contiguous per product do the following 11 : Keep applying all depth incremental as well as depth snapshot 12 messages to the current order book. 11 The reason is that the unreliable nature of UDP multicast can cause packets to arrive delayed, in incorrect sequence or may be missing. 12 only if the Refresh Indicator (1187) = Y 38

39 Each incremental message can carry different update instructions with the update action (New, Change, Delete, Delete From, Delete Thru, Overlay). Note: The depth snapshot messages for the MDI are sent on the same channel as the depth incremental messages. If the RefreshIndicator (1187) is set, changes to the order book are processed into the depth snapshot messages and not provided as separate depth incremental messages. 39

40 7 Recovery Due to the unreliable nature of UDP multicast it is possible that some packets may either be delayed, arrive in the incorrect order or may be missing. Furthermore the UDP packets may be duplicated at the network level. Receiving applications need to be capable of handling these issues. This chapter describes the scenarios which might occur and provides a guideline on how a receiving application needs to react to those scenarios. Recovery actions are possible on a packet level by using the respective other service (A or B). In case a packet is lost on both services (A and B) clients can create a new current order book by using snapshot information. 7.1 Detecting duplicates and gaps by means of the packet header The packet header allows receiving applications to identify identical packets between Service A and Service B. This is achieved by a simple memory comparison on the first 9 bytes for EMDI or 8 Bytes for MDI of a datagram containing SenderCompId and PacketSeqNum as shown in figure 21, Structure of the packet header for EMDI and figure 22, Structure of the packet header for MDI and RDI. Another important function of the packet header is to identify gaps by means of the PacketSeqNum which can be retrieved just by decoding the packet header. Note: Packets with the same SenderCompID (field length: 1 Byte) have contiguous sequence numbers per multicast address / port combination. This means that field PacketSeqNum can be used not only to detect duplicates but also to detect missing packets. PacketSeqNum is a Byte vector and therefore not stop bit encoded as per the FAST specification. The packet header itself does not contain any product information. In order to find out which product is missing, the product level sequence number must be used in addition to the packet level sequence number; the packet needs to be decoded further down to the message level. This leaves participants with two recovery options when a gap in the PacketSeqNum s of the packet header is detected. Example: A single multicast address carries products FDAX and FGBL, but the participant is only interested in FGBL. 40

41 I. Pessimistic approach: The receiving application assumes that FGBL is part of the missing packet. It immediately starts recovery actions 13 just by decoding the packet header. Advantage: Recovery is triggered immediately when observing a missing PacketSeqNum without decoding the entire message. Disadvantage: The recovery might not be necessary, if FGBL is not part of the message which is inside the lost packet. II. Optimistic approach: The receiving application assumes that FGBL is not part of the missing packet. It waits for the next message on the same service and decodes the packet up to the message level to find out if a packet for FGBL has been lost before triggering recovery actions. Advantage: This approach allows the participant to recover only products of interest. Disadvantage: The receiving application needs to wait for the next message. However, the next packet may not contain a message for the product in question. 13 by means of the other service (live-live concept) or by listening to the depth snapshot 7.2 How to recover data via the respective other service (A or B) Feeds are replicated onto two services, Service A or Service B, and carried on different multicast addresses. This feature provides the possibility to recover missed packets, and participants are advised to join both services. In each of the following tables, the Time column is entirely arbitrary and is intended to show only the sequence of events and in some cases the relative delay between dependent events. The following table explains the design concept for Service A and B. The table contains the field MsgSeqNum from the message itself. However, it could also contain the field PacketSeqNum from the Packet Header. Time Service A: Message Time Service B: Message MsgSeqNum MsgSeqNum 10:30: New 151@4 10:30: New 151@4 10:30: Delete 151@5 10:30: Delete 151@5 Lost 10:30: New 151@5 10:30: New 152@4 10:30: New 152@4 Table 9: Recovery via Service B (live-live concept) 41

42 As the above example shows, the same information is delivered on Service A and B. While MsgSeqNum= 208 is missing on Service A, it is provided on Service B. Ideally a receiving application processes packets from both Service A and B simultaneously and would take into account the message that arrives first and discards the second (identical) message. In the unlikely event that the message has neither been received via Service A nor Service B, the receiver is required to initiate a loss of data scenario: The order book needs to be recreated by using the depth snapshot messages in conjunction with the depth incremental messages. This procedure is similar to the Start Up procedure. Please see section 6.4, Build the initial order book. 7.3 Delayed packets The following example indicates a simple case: Time MsgSeqNum Message 10:30: New 151@4 10:30: Delete 151@5 10:30: New 152@4 Table 10: Packets arriving in correct sequence In this example, messages arrive in the correct order. The message was not delayed between BSE and the receiving application. There is no special requirement on the application; the message can be processed in the same order as they arrive. Multicast does not guarantee that the order in which packets are received is the same as the order in which they are sent. For instance, BSE Market Data Interface sends incremental messages in ascending MsgSeqNum order, but they might arrive in an incorrect order at the receiving application. Consider the following example: Time MsgSeqNum Message 10:30: New 151@4 10:30: Delete 151@5 10:30: New 152@4 Table 11: Delayed Packet

43 In this example, message 207 is delayed within the network, allowing message 208 to arrive first. A correct communications layer responds as follows: 1. Release message 206 to the application immediately on arrival. 2. On arrival of 208, recognizes that 207 is missing. 3. Start an appropriate timed operation to trigger the recovery actions if the out-of-sequence message 207 fail to arrive in a reasonable time. 4. Assuming that 207 arrives within that reasonable time, release 207 and then 208 to the application in that order and cancel the timed recovery action. 7.4 Missing packets All lost packets start life as delayed packets, as illustrated in the preceding case. The communications layer of the receiving application is responsible for deciding when to declare a network packet as lost. In the following example it is assumed that MsgSeqNum = 207 from the example above does not arrive within the allowed time. Therefore it is considered as lost: Time MsgSeqNum Message 10:30: New 151@4 lost 10:30: Delete 151@5 10:30: New 152@4 Table 12: Missing seqnum 207 The correct behavior in this instance is: 1. Release message 206 immediately on arrival. 2. Hold on to 208 because it is out-of-sequence, and initiate timer-based recovery actions. 3. Hold on to 209 for the same reason. Timer-based recovery actions are already pending for this product, so do not reset the timer. 43

44 (a) Even though message 209 is a New operation, it may be unsafe to apply 208 and 209 because we do not know what 207 contains. 4. If the missing message (207) fails to arrive within the allowed time: (a) Initiate recovery from the respective other service (A or B) for message (207). If this works then release (207) and then all messages with higher MsgSeqNum s. (b) In case the recovery from the respective other service (A or B) fails: initiate recovery via snapshots Recovery (EMDI) Depth snapshot and depth incremental messages are distributed via separate channels for the EMDI. For instance, depth incremental messages could be sent on multicast address A 2 I, port x and the snapshot message on multicast address A 2 S with port y (see Figure 2, Overview of the three interfaces). Incremental are sent whenever there is a change of the order book (event-driven); snapshots are sent periodically in intervals regardless of whether the order book has changed since the last snapshot (timedriven). Each message sequence number (field: MsgSeqNum) on the market data incremental feed is unique and contiguous by product across messages. Therefore the sequence number can be used to detect losses. If any gap of the arriving sequence numbers is detected and this gap cannot be filled by using the respective other service (A or B) the receiving application should initiate a snapshot recovery. The following example shows missing depth incremental messages (MsgSeqNum s ) and depth snapshots (with LastMsgSeqNumProcessed ) which relate to the missing message. MsgSeqNum s for the depth snapshot do not exist, which is indicated with N/A in the table. MsgSeqNum Product LastMsgSeq Message Type Channel NumProcessed 205 A depth incremental I A A depth incremental I A A depth incremental I A 1 Lost A depth incremental I A 1 Lost A depth incremental I A A depth incremental I A B depth incremental I A 2 N/A A 209 depth snapshot s A A depth incremental I A 1 N/A B 1000 depth snapshot s A 2 44

45 1001 B depth incremental A 2 I Table 13: Snapshots and incremental within the EMDI The appropriate recovery action for missing depth incremental is the same as the logic described in section 6.4.1; Build the initial order book with the EMDI. There are some additional points to be aware of when performing recovery: During recovery, applications should be prepared to receive depth incremental messages for instruments they didn t know existed. This can occur if a strategy creation event (via a complex instrument update on the market data feed) is missed due to packet loss. In this case, applications must consult the reference data snapshot feed to obtain the strategy description. The corollary to the above is that an application may have missed a complex instrument update indicating a strategy deletion message. Deleted strategies are immediately removed from the snapshot feed, so having seen a complete cycle for a product and noting that no order book information was received for a strategy, the application should consult the reference data snapshot feed to confirm that it has been deleted. Depth snapshot messages are not sequenced, but they are still theoretically subject to out-of-order packet delivery. Applications must consider this in determining that their snapshot cycle is complete. The packet sequence number in the packet header can be used to detect out-of-order delivery. The LastMsgSeqNumProcessed (369) is not necessarily the same for all instruments belonging to a product on the market data snapshot feed. Note: The market data snapshot feed does not contain any start or end messages to delineate the cycle. There are two ways to determine when to leave the snapshot feed during recovery: Method 1: Process specific products For each SenderCompID (49) contributing to the market data snapshot feed, depth snapshot messages are grouped by product as illustrated below: P 1 I 1 P 1 I 2 P 1 I 3 P 1 I n P 2 I 1 P 2 I 2 P 2 I 3 P 2 I n P 3 I 1 P 3 I 2 P 3 I 3 P 3 I q [...] 45

46 with: P n : Product n I q : Simple or complex instrument q for product n Depth snapshots for instruments in the same product will often all appear in the same packet, but this should not be relied upon as it is not true when the amount of data is simply too great to fit into a single packet, and under certain other technical conditions on the exchange. A change of product MarketSegmentID (1300) for a given SenderCompID (49) indicates the end of the depth snapshot messages for the respective product. This allows applications to easily determine when they ve received a snapshot for every instrument in the products they re interested in and leave the snap- shot feed. Method 2: Process an entire depth snapshot cycle It s also easy for an application to listen to an entire snapshot cycle. Applications can determine when they ve seen an entire snapshot cycle simply by remembering the SecurityID (48) of the first depth snapshot message they saw from each SenderCompID (49). When they see the same SecurityID (48) again for each SenderCompID (49), they know that a complete depth cycle has been seen and can leave the snapshot feed. Note: Receiving applications also need to consider depth snapshot messages for newly created complex instruments. Note: If a failover occurs during snapshot processing the SenderCompID (49) for the affected partition changes and the snapshot cycle for that partition starts again Recovery (MDI) Snapshot and incremental messages are sent on the same channel and carry a contiguous sequence number (field: MsgSeqNum) per product. The snapshot always carries the latest information and might carry new information, not already sent with an incremental message. The following table shows an example for the distribution of incremental and snapshot messages for two products: MsgSeqNum Product Message Type Channel 5 A depth incremental S,i A 1 6 A depth incremental S,i A 1 46

47 Lost A depth incremental S,i A 1 25 B depth incremental S,i A 2 8 A depth incremental S A 1 9 A depth snapshot S A 1 10 B depth snapshot S,i A 2 11 A depth incremental S,i A 1 26 B depth snapshot S,i A 2 27 B depth snapshot S,i A 2 Table 14: Snapshots and incremental within the MDI If the depth incremental message for product A with MsgSeqNum = 7 is lost, a consistent order book can be rebuilt from the next snapshot message for product A, in this case arriving with MsgSeqNum=9. All depth incremental messages for product A with a lower sequence number than the next market data snapshot message for product A must be discarded, e.g. MsgSeqNum = 8 (incremental) must be discarded as its effect is included in MsgSeqNum = 9 (snapshot). Since multicast doesn t guarantee the correct sequence of the incoming message, it is recommended to buffer all incoming incremental while waiting for the next snapshot message. The buffered incrementals for product A with MsgSeqNum 11 can be applied to the latest snapshot with MsgSeqNum = 10. Note: LastMsgSeqNumProcessed is not necessary for recovery purposes in the MDI. 47

48 8 Various time stamps in BSE and how to use them This section provides a list of each time stamp field in the two BSE Market Data Interfaces; it describes the measurement point and explains what is being measured. All time stamps are provided in UTC (nanoseconds since Unix Epoch ( )). While the format is provided in nanoseconds the actual precision of time stamps can be in microseconds. In that case the last three digits of the time stamp field is Time stamps (EMDI) The following picture shows all time stamps on the BSE Matching Partition for requests/responses for orders and execution messages: Figure 8: Time stamps and their measurement points 48

49 Message Field Name Time stamp of time stamp Packet header PerformanceIndicator t 3 -t 0 Time between the arrival of an incoming order transaction on the BSE matching engine and send time of the corresponding outgoing market data. The PerformanceIndicator is sent for Incremental only. Sending Time t 3 Time, the BSE Market Data is written onto the socket for the fastest Service (A or B). For more information please see distribution sequence. The time stamp is the same for Service A and Service B even though packets on both services are not sent out at the same time. Depth incremental MDEntryTime 14 t 2 Two possibilities: In case of an order: Time of the last order book update. In case of a trade: Match time. Depth incremental AggressorTimeStamp 15 t 0 Entry time of the incoming order that triggered the trade. This time stamp is only available in case of a trade (MDEntryType=2). The AggressorTimeStamp is empty if The trade resulted from an auction. Depth snapshot LastUpdateTime t 2 Time of the last order book update. Instrument incremental Complex instrument update TransactTime t 2 Creation time of complex instrument. This field is empty for deletions of complex instruments. TransactTime t 2 Time when request was 49

50 Mass instrument state change Product state change Instrument state change processed by the matcher. Table 15: Meaning of the time stamps Further information about time stamps are available in the ETI manual, section Timestamps 8.2 Time stamps (MDI) The field PerformanceIndicator in the packet header message is not available for the BSE MDI. Also the field AggressorTimeStamp (28820) will not have any value. All other fields are the same as for the BSE EMDI. The field LastUpdateTime in the depth snapshot message contains the time at which the last update was applied to the order book. In the special situation that several orders entered in the middle of a netting interval cancel each other out, this field shows the time stamp of the last order entry even though the net result is that there is no actual change. An example of when this would happen is the creation and subsequent deletion of an order within the netting interval. 14 This field maps to ExecID (17) in BSE ETI order / quote responses and notifications as well as to TransactTime (60) in the BSE ETI Trade Notification. 15 This field maps to TrdRegTSTimeIn (21002) in BSE ETI. 50

51 9 Important topics with use cases and examples The following section Use Cases describes situations which require special attention. Various examples are provided. 9.1 Reference data messages Reference data provides technical and functional information about all products and instruments available in BSE. Reference data messages are sent within different feeds: Snapshot feed of BSE RDI provides a snapshot of all products and instruments (simple and complex) and is sent out on a regular basis throughout the day. Additions or deletions of complex instruments are incorporated into the next snapshot cycle. Incremental feed of BSE RDI is event triggered and provides real-time information about complex instruments 16 that are added or deleted intraday. Any change is incorporated within the next snapshot cycle. The incremental feed never contains simple instruments. Market data incremental feed of EMDI is event triggered and provides real-time information about complex instruments that are added or deleted intraday on the same channel as market data. Market data feed of MDI is event triggered and provides real time information about complex instruments that are added or deleted intraday. The following messages are sent via different feeds: a) Snapshot feed of RDI: Product snapshot for products available at start of day. Instrument snapshot for simple and complex instruments available at start of day. Instrument incremental for complex instruments added or deleted intraday. Market data report indicates the start of reference data (MDReportEvent=1). 51

52 Market data report indicates the end of reference data (MDReportEvent=2). b) Incremental feed of RDI: Instrument incremental for complex instruments added or deleted intraday. c) Market Data incremental feed of EMDI: Complex instrument update 17 for complex instruments added or deleted intraday. d) Market data feed of MDI: Complex instrument update 17 for complex instruments added or deleted intraday. 16 No product information is delivered 17 The layout is the same as on the instrument incremental message except for the field SecurityExchange (207), SecurityStatus (965) and SenderCompID (49) 52

53 9.2 General reference data rules General structure of the snapshot cycle A snapshot cycle consists of (see figure 9): A market data report message (MDReportEvent = 1 = StartOfReferenceData ). A sequence of a product snapshot followed by the associated instrument snapshots (simple and complex), repeating for all products and instruments. A dynamically growing sequence of instrument incremental messages (complex instruments). Finally market data report message (MDReportEvent = 2 = EndOfReferenceData ). Figure 9: Entire snapshot cycle on the RDI snapshot feed with: P n : Product n I nq : Simple instrument q for product n C nm : Complex instrument m for product n Product and instrument snapshot messages are sent for the initial set of products and instruments. While the snapshots do not change intraday, the number of incremental messages increases if complex instruments are added or deleted. Figure 10 illustrates how more instruments incremental are added over the course of n cycles: 53

54 Figure 10: Reference data with constant snapshots and extending incremental Note: Overnight changes to the products and instruments are reflected in the reference data snapshot messages after the technical start on the next business day Counters as part of the market data report message The message sequence numbers of the market data report messages preceding each snapshot cycle represent counters for the number of snapshots, incremental and overall number of messages within the current cycle. The market data report message, type: StartOfReferenceData, contains the following sequence number fields: MDCount (5488): Number of reference data messages in the snapshot cycle, which is available at the start of day and remains constant throughout the operational hours of reference data service for the current business day. The value represents the number of products + the number of instruments (simple + complex) at start-of-day. If a failure of BSE RDI occurs the number of messages in the reference data snapshot and herewith MDCount (5488) changes. LastMsgSeqNumProcessed (369): This is the MsgSeqNum value of the last reference data message (snapshot or incremental) in the snapshot cycle (products and instruments share a single sequence number.). Note: The number of incremental updates in a snapshot cycle can be calculated as: Number of incremental updates = LastSeqNumProcessed - MDCount. TotNoMarketSegments (8825): Contains the number of product messages sent in the snapshot feed. This value remains constant intraday as products are not created or deleted intraday. 54

55 TotNoInstruments (8826): Contains the number of instrument messages sent in the snapshot feed. This value changes as more complex instruments are created or deleted intraday. TotNoMarketSegments (8825) and TotNoInstruments (8826) can be used as sanity check and to preallocate the product and instrument containers. The market data report message, type: EndOfReferenceData marks the end of reference data messages and doesn t contain any counters. The following examples highlight a few scenarios which require special attention. The focus lies on the reference data snapshot feed which provides constant snapshot messages and a variable part with incremental for complex instruments Use case 1: Reference data at the start of the reference data service At the start of the reference data service the reference data snapshot is sent. Figure 11 shows how snapshots for simple and complex instruments which are already in the system are sent at start of day: Figure 11: Reference data snapshot message on the reference data snapshot feed at the start of the reference data service The first snapshot cycle can already contain some complex instruments in the snapshot messages. A reference data incremental message does not exist at this time Use case 2: Reference data after intraday addition of complex instruments The next example shows an intraday addition of three complex instruments C 11, C 41 and C 33. See figure12). The reference data incremental messages for complex instruments C 11, C 41 and C33 are appended to the reference data snapshot messages: 18 Participants interested in complex instruments can also use the complex instrument update message via the faster depth incremental feed of EMDI or the market data feed of MDI. 55

56 Figure 12: Reference data snapshot after intraday addition of complex instruments C 11, C 41 and C33 LastMsgSeqNumProcessed (369) in the market data report, type: Start of Reference Data increases to 103. The no. of incrementals (products + instruments) can be calculated as LastMsgSeqNumProcessed - MDCount = = 3. New complex instruments predefined by the exchange are sent in instrument incremental and not in Snapshot messages on the day of creation; the messages are sent before trading starts Use case 3: Reference data after intraday deletion of a complex instrument The next example shows an intraday deletion of a complex instrument C 41 * (figure 13). The deleted complex instrument C 41 * is sent as reference data incremental message and appended again to previously generated complex instruments C 11, C 41 and C 33 : Figure 13: Reference data snapshot after intraday deletion of complex instrument C 41 * 56

57 Since one complex instrument is deleted, LastMsgSeqNumProcessed (369) increases to 104 snapshot and incremental messages Use case 4: Reference data on the next business day The complex instruments which still exist on the next business day and which have been sent as reference data instrument incremental on the previous business day, are sent as instrument snapshot messages on the next business day, if orders still exist in the respective order books as shown in figure 14: Figure 14: Reference data snapshot on the next business At the beginning of the next business day 19 LastMsgSeqNumProcessed (369) is 102 as this reflects the number of remaining complex instruments C 11 and C33. C 41 has already been deleted on the previous business day (see use case 3). At this point in time MDCount (5468) has increased to 102 as well. The same scenario would be true if the reference data server would have been restarted intraday (due to a technical failure) Use case 5: Failover or restart of RDI In the event that RDI fails, another instance takes over. Receiving applications can detect this by a change of the SenderCompID and the receipt of a market data report message. Applications should respond to this situation as described in section 6.2, Receiving reference data via RDI at start of day. The same recovery actions apply in case of a complete restart of RDI. If RDI fails over the instrument snapshot contains any complex instruments already created and deleted during the day, i.e. the entire history. If RDI needs to be restarted by the exchange a new snapshot cycle is generated. This cycle contains the currently existing complex instruments but not the entire history of creations and deletions. 19 also in case of a feed restart on the exchange side 57

58 9.2.8 Use case 6: Chronological order of messages for complex instrument creation The intraday creation and deletion of complex instruments results in the following sequence of messages: 1. On the market data incremental feed of the EMDI and market data feed of the MDI: An instrument incremental message is sent to inform the participant as fast as possible. In case a new complex instrument has been created the corresponding message is sent prior to the publication of any order book data for the new complex instrument. 2. On the reference data incremental feed of the RDI: An instrument incremental message is also sent with additional fields populated. There is no product incremental message. 3. On the reference data snapshot feed of the RDI: An instrument incremental message is appended to the end of the current snapshot cycle without removing or changing any of the existing snapshot or incremental messages in the cycle 20. Therefore the cycle is only extended intraday and never reduced, even if complex instruments are added and deleted intraday. 9.3 General order book rules and mechanics The BSE Market Data Interfaces, EMDI and MDI, provide order book updates from level 1 to the maximum level. The maximum level is provided for each product in the product snapshot records in the reference data field MarketDepth (264). The order book can be constructed by the depth incremental messages or by the depth snapshot message. All on-exchange trades and order book updates are reported via the same depth incremental messages. However, trades are always sent out prior to order book updates. The following design principles apply to order book updates: Orders are aggregated per price level and are not distributed individually. Changes to the book that result from one atomic action in the matching engine are disseminated in one depth incremental message for EMDI. Each BSE EMDI packet relates only to a single product. In other words, although each BSE EMDI packet may contain multiple messages, those messages will always relate to the same product. This does not apply to BSE MDI where a single packet may relate to multiple products. 20 A complete snapshot cycle is a combination of start, refdata snapshots, refdata incremental and end message. 58

59 Price levels are provided explicitly (field: MDPriceLevel (1023)) and do not need to be derived through the price itself. During the product states Start-Of-Day, Pre-Trading, Post-Trading and End-Of- Day or when no price levels exist, an empty book (MDEntryType=J) is disseminated for the depth snapshot message (not for incremental). In addition to an empty book, statistical information is sent in Pre-Trading, Post-Trading and in End-Of-Day. An implied price is the only element of the group without a price level (for MDEntryType = 0 = Bid or 1 = Offer). For price levels from 1 to max price levels, outright prices are distributed. An implied price can either be fully implied or partially implied (for more information please refer to section 9.3.1, Determination of the price sources). If two (or more) synthetic prices (with the same price) are created for the Best Market via a different path, the sums of the quantities are reported for the particular price. An example is provided in section , table 49. There can be multiple updates in one message. The bid side is updated first followed by the ask side. If update instructions new or delete is sent for an implied price, the order book levels 1-n don t need to be shifted down or up. Order book update instructions are sent for each order book side without a specific order of update actions but ordered by price level instead. from best outright price (price level 1) down to the worst price (max. price level configured per product). if the resulting book depth is larger than the specified maximum product depth only the specified maximum product depth must be saved. For auctions, the best bid/ask prices are disseminated at price level 1 without a quantity. Receiving applications need to delete a pre existing quantity when an absent value is received during a transition into an auction. During an auction, there can either be a crossed or an uncrossed book situation. A crossed book is identified to the user by means of a auction clearing price (MDEntryType=Q) (aka indicative or potential auction price). An uncrossed book is identified by means of ToB prices (MDEntryType 0=Bid and/or 1=Offer). 59

60 The change from a crossed to an uncrossed book situation and vice versa is implicitly identified by sending ToB information instead of a auction clearing price and vice versa. The previous ToB information or auction clearing price is only explicitly removed (MDUpdateAction 2=Delete) whenever an uncrossed or crossed book changes to an empty book during an auction. A instrument state change message identifying an instrument leaving an auction also implicitly deletes the Auction Clearing Price. The visibility of the order book is limited during an auction. Depth information will be explicitly provided again when transitioning from an auction to continuous trading as the user cannot know how much of an order book situation prior to the auction is still valid. Depth information will also be explicitly removed when transitioning from continuous trading to an auction and the auction starts with an uncrossed order book. Figure 15 illustrates a particular scenario for an opening auction. The three situations order book Crossed, uncrossed and empty can appear in any sequence within the auction Figure 15: Example for an Opening Auction with implicit and explicit Deletes Note: In general, BSE sends explicit deletes during an auction. However, during the transitions described below, receiving applications need to perform the following actions: - From ToB to Auction Clearing Price: Receiving applications need to delete ToB. - From Auction Clearing Price to ToB: Receiving applications need to delete the Auction Clearing Price. - Whenever the auction ends and another instrument state starts: Receiving applications need to delete the Auction Clearing Price. 60

61 A state transition to Freeze is sent as an instrument state change message and does not require any implicit action. If the book is crossed, an indicative auction price is calculated and disseminated. The new indicative auction prices are always sent with update action New. Intraday expired instrument information is provided by a depth incremental and instrument state change message. Only the snapshot and incremental messages of the MDI carry a common and contiguous sequence number per product. The incremental message of EMDI contains a contiguous sequence number per product across all messages, while the snapshot message provides the last sequence number (LastMsgSeqNumProcessed ) sent in the incremental message. Only the best implied price is published. The best implied price will be included in market data only in case it is equal to or better than the best direct price in the respective instrument. Whenever the quantity or price of the Best Market changes it is disseminated with update action New on the incremental feed. Similarly, the Best Market is removed with update action Delete. Note: The order book is only valid after the entire incremental message has been fully processed. 61

62 Figure 16 illustrates a typical order book and terminology used in the following chapters. Figure 16: Typical order book An implied price can be either better (fully implied) or the same (partially implied) as price level Determination of the price sources The new trading platform supports synthetic matching, where the implied prices from complex instruments can create prices equal or better than the best outright price in the instrument. The implied prices are disseminated in the market data in addition to the prices from outright orders. These prices are shown without a price level. The reported quantities for implied prices and level 1 are not aggregated, i.e. quantities on level 1 are fully outright and do not contain any implied components. The BSE system publishes implied prices in market data only in case it is equal to or better than the best outright price in the respective instrument. In order to find out which situation applies, a price comparison between the implied price (with empty price level) and level 1 (see figure 16) needs to be done: 1. Implied price is better than the outright price at level one -> Fully Implied. 2. Implied price disseminated is equal to the outright price at level 1 -> Partially Implied. 3. Implied price is deleted or absent -> the Best Market price is fully outright and is the same as on level 62

63 Examples for all three cases are provided in section 14.2, Example for determination of the price source New price level When a new price level is created in the order book, a depth incremental message is sent with field MDUpdateAction (279) = 0 ("New ). This indicates that: The new price level is to be inserted at the specified price level. 21. All existing rows in the order book at the specified and higher levels are to be incremented accordingly. 22. Price levels exceeding the maximum specified depth must not be kept in memory. Note: The field MDPriceLevel (1023) is used to identify which level is being inserted. Example: Buy Limit Order, 10@58.22, enters an empty order book: Tag number Tag name 35 MsgType X MarketDataIncrementalRefresh 34 MsgSeqNum SenderCompID 75 Unique id of a sender 1300 MarketSegmentID 89 Product 268 NoMDEntries > MDUpdateAction 0 New 269 > MDEntryType 0 Bid 48 > SecurityID 8852 Instrument 22 > SecurityIDSource M Marketplace-assigned identifier 270 > MDEntryPx Price 271 > MDEntrySize 10 Quantity 346 > NumberOfOrders 1 Number of order on this level 1023 > MDPriceLevel 1 Book level 273 > MDEntryTime t 0 official time of book entry Table 16: MDUpdateAction New 21 A MDUpdateAction (279) = 0 ( New ) is also disseminated whenever the quantity changes for the implied price (empty price level). 22 This is not the case if the MDUpdateAction (279) = 0 ( New ) is sent for the implied price (with empty price level). 63

64 9.3.3 Change of a price level A depth incremental message with MDUpdateAction = 1 ("Change ) indicates A change at a given price level. All fields but the price on the specified side at the price level should be updated. Note: MDUpdateAction= Change is sent only for depth 1 when the price does not change. A MDUpdateAction (279) "Change contains a price which can be used as a consistency check. However, it never contains a price that is different from the existing one on the current price level. Example: Quantity changed to 8 for limit order above: Tag number Tag name 35 MsgType X MarketDataIncrementalRefresh 34 MsgSeqNum SenderCompID 75 Unique id of a sender 1300 MarketSegmentID 89 Product 268 NoMDEntries > MDUpdateAction 1 Change 269 > MDEntryType 0 Bid 48 > SecurityID 8852 Instrument 22 > SecurityIDSource M Marketplace-assigned identifier 270 > MDEntryPx Price 271 > MDEntrySize 10 Quantity 346 > NumberOfOrders 1 Number of order on this level 1023 > MDPriceLevel 1 Book level 273 > MDEntryTime t 1 official time of book entry Table 17: MDUpdateAction Change Overlay A depth incremental message with MDUpdateAction (279) = 5 ("Overlay ) is used to Change the price of a given price level. Other parameters, e.g. quantity might also change. Note: MDUpdateAction= Overlay is sent only for depth 1, i.e. the field MDPriceLevel (1023) must be present. In contrast to the MDUpdateAction= Change this instruction contains a price change. The overlay function is normally used when there is just one price level disseminated. 64

65 Example: Buy limit order replaces the best buy limit order during instrument state Auction : Tag number Tag name 35 MsgType X MarketDataIncrementalRefresh 34 MsgSeqNum SenderCompID 75 Unique id of a sender 1300 MarketSegmentID 89 Product 268 NoMDEntries > MDUpdateAction > MDEntryType 0 Bid 48 > SecurityID 8852 Instrument 22 > SecurityIDSource M Marketplace-assigned identifier 270 > MDEntryPx 2.48 Price 271 > MDEntrySize N/A Quantity remain same in this example 1023 > MDPriceLevel 1 Book level 273 > MDEntryTime t 5 official time of book entry Table 18: MDUpdateAction Overlay Deletion of a price level A depth incremental message with MDUpdateAction (279)= 2 ("Delete ) is used to delete a specified price level. Note: All price levels greater than the deleted one should be decremented. The price of the price level to be deleted is also sent within the message and can be used as a consistency check. Example: Deletion of limit order modified above: Tag number Tag name 35 MsgType X MarketDataIncrementalRefresh 34 MsgSeqNum SenderCompID 75 Unique id of a sender 1300 MarketSegmentID 89 Product 268 NoMDEntries > MDUpdateAction 2 Delete 269 > MDEntryType 0 Bid 48 > SecurityID 8852 Instrument 22 > SecurityIDSource M Marketplace-assigned identifier 65

66 270 > MDEntryPx Price 271 > MDEntrySize N/A Quantity not populated for Delete 1023 > MDPriceLevel 1 Book level 273 > MDEntryTime t 1 official time of book entry Table 19: MDUpdateAction Delete Deletion of multiple price levels from a given price level onwards A depth incremental message with MDUpdateAction (279) = 4 ("Delete From ) is used to Delete all price levels specified price level. Note: All price levels from the specified one and up to the maximum need to be deleted. Example: Deletion of all orders for SecurityID = 8852, MarketSegmentID = 89 from level 3 and above: Tag number Tag name 35 MsgType X MarketDataIncrementalRefresh 34 MsgSeqNum SenderCompID 75 Unique id of a sender 1300 MarketSegmentID 89 Product 268 NoMDEntries > MDUpdateAction 4 Delete From 269 > MDEntryType 0 Bid 48 > SecurityID 8852 Instrument 22 > SecurityIDSource M Marketplace-assigned identifier 270 > MDEntryPx Price 271 > MDEntrySize N/A Quantity not populated 1023 > MDPriceLevel 3 Book level 273 > MDEntryTime t 3 official time of book entry Table 20: MDUpdateAction Delete From Deletion of multiple price levels up to a given price level A depth incremental message with MDUpdateAction (279) = 3 ("Delete Thru ) is used to Delete all price levels from 1 to the specified price level. 66

67 Note: All higher than the specified price levels are shifted down to fill the gap of the deleted price levels. Example: Deletion of all price levels from 1 to price level 3. Tag number Tag name 35 MsgType X MarketDataIncrementalRefresh 34 MsgSeqNum SenderCompID 75 Unique id of a sender 1300 MarketSegmentID 89 Product 268 NoMDEntries > MDUpdateAction 3 Delete Thru 269 > MDEntryType 0 Bid 48 > SecurityID 8852 Instrument 22 > SecurityIDSource M Marketplace-assigned identifier 270 > MDEntryPx Price on level > MDEntrySize 10 Quantity 346 > NumberOfOrders 1 Number of order on this level 1023 > MDPriceLevel 3 Book level 273 > MDEntryTime t 4 official time of book entry Table 21: MDUpdateAction Delete Thru 67

68 9.4 Trade Volume Reporting (EMDI) All on-exchange trades executed on the new BSE trading platform are reported via depth incremental messages. The depth snapshot messages contain statistical information about trades only. Trades can be identified in the incremental messages when MDEntryType is set to 2 (Trade). The EMDI only disseminates information about on-exchange trades. OTC trade information is not disseminated via the market data incremental and market data snapshot feed of the BSE EMDI. When an order executes against the book at multiple price levels, this is reflected by a matching event with multiple match steps. Each match step has the trades at one price level and is represented by a unique MDEntryID (278) and published in the market data. The field MDEntryID (278) is a unique id on product level for each business day. A synthetic match can result in more than one trade volume record with the same MDEntryID (278) as shown in use case 3, section and use case 4, section Every match step occurring in the exchange has an identifier in BSE ETI that is provided in the field FillMatchID (28708) in the Execution Report (8), QuoteEventMatchID (8714) in the Quote Execution Report (U8) and TrdMatchID (880) in the Trade Capture Report (AE). This identifier allows participants to link trade capture reports and the corresponding execution report of the BSE ETI with the market data incremental feed of the BSE EMDI. The following 4 use cases illustrate the MDEntryID (278) and how Trade Volume Reporting works: Use case 1: Direct match of simple instruments An incoming simple order is matched against two orders of the opposite side of the order book on different price levels. Incoming buy order, 150@2885, FESX Sep Existing Order book: BID ASK 50@2884 FESX Sep 100@2885 FESX Sep 68

69 Trade Volume Reporting: Two trades are reported because two different price levels are involved in the matching process: First gets reported due to a higher matching priority of this price level; afterwards 100@2885. Instr. MDEntryID MDUpdateAction size@prc TradeCond. AggrSide #Buy #Sell Total FESX 1 NEW 50@2884 U,R,AX,AY BUY Sep FESX Sep 2 NEW 100@2885 U,AX BUY with: U = Exchange last R = Opening price AX = High price AY = Low price AW = Last auction price Use case 2: Direct match of complex instruments An incoming complex order 23 involving 2 legs are matching against the opposite side of the complex instrument order book. Incoming buy order, 100@8, FESX Sep/Dec Existing Order book: BID ASK 50@7 Sep/Dec 100@8 Sep/Dec Trade Volume Reporting: Only the entire strategy trade is reported and no trade information is published for the individual legs. Instr. MDEntryID MDUpdateAction size@prc TradeCond. AggrSide #Buy #Sell Total Sep/Dec 3 NEW 50@7 U,R,AX,AY BUY Sep/Dec 4 NEW 100@7 U,AX BUY In general, a complex order is any combination of simple instruments. This example is based on a Future Time Spread. However, 69

70 Use case 3: Complex versus simple order match A buy spread order as an incoming complex order (Time Spread) matches (synthetically) against several simple instrument leg orders (outright orders). Incoming buy order, 200@8.0 FESX Sep/Dec Existing Order book: BID 120@2878 Dec 30@2878 Dec BID ASK ASK 60@ Sep 50@ Sep 40@ Sep This results in the following implied price: BID ASK Sep12/Dec12 150@8.0 Trade Volume Reporting: The incoming spread order matches against the implied-in order of the order book which is a composition of all 5 outright orders in the order book. Again, the trades are aggregated per price level. The fields NumberOfBuyOrders (28821) and NumberOfSellOrders (28822) show how many orders are involved. In case of a synthetically matched complex order either the buy or sell side contains an empty value. In case of a direct matched complex instrument, both sides are filled. Instr. MDEntryID MDUpdateAction size@prc TradeCond. AggrSide #Buy #Sell Total Sep/Dec 5 NEW 150@8 U BUY Sep 5 NEW 150@2886 U,AX Dec 5 New 150@2878 U,R,AX,AY In case the incoming buy spread order is matching against sell spread orders in the same instrument (direct matching), only one trade volume reporting record is published. As a general rule, only one trade 70

71 volume record is published per match step in case of direct matching (as an example, see section 9.5.2) while there are more than one trade volume reporting record per match step in case of synthetic matching as described in the example of this section Use case 4: Complex versus simple/complex match Incoming buy order, FESX Sep/Dec Existing Order book: Bid Ask Dec Sep Dec Sep/Dec Trade Volume Reporting: Incoming complex order is matching directly against the opposite side of a complex order; another part is matching against an implied-in order which was created by two existing outright orders for the Sep and Dec contracts. The direct match of the complex orders can be identified by existing entries for NumberOfBuyOrders (28821), NumberOfSellOrders (28822). The synthetic match can be identified by the missing entry for NumberOfSellOrders (28822). Instr. MDEntryID MDUpdateAction size@prc TradeCond. AggrSide #Buy #Sell Total Sep/Dec 6 NEW 100@8 U BUY Sep/Dec 6 NEW 150@8 U Buy Sep 6 New 150@2886 U Dec 6 New 150@2878 U,AY Use case 5: Opening auction After the uncrossing of the order book in a simple instrument at the end of an auction call phase, five orders on the buy side and 3 orders on the sell side of the order book have been matched. The Trade- Condition (277) is set to AW for Auctions. The field TrdType (828) specifies the type of the auction. For trades outside the auction, TrdType (828) is not set. Existing Order Book during Auction: Bid 30@24.39 Sep 25@24.39 Sep 20@24.39 Sep 55@24.39 Sep 5@24.39 Sep Ask 60@24.39 Sep 57@24.39 Sep 18@24.39 Sep 71

72 Trade Volume Reporting: All orders are matching on the same price level. Therefore they are reported only once but with different NumberOfBuyOrders (28821) /NumberOfSellOrders (28822). The AggressorSide (28731) is left empty because during an auction, orders are not considered to be aggressive. The following depth incremental message is sent: Instr. MDEntryID MDUpdateAction TradeCond. AggrSide #Buy #Sell Total Sep 1 New 135@24.39 U,R,AX,AY,AW OPENING The following depth snapshots belong to the depth incremental above: Instr. MDUpdateAction size@prc TradeCond. TrdType Total Sep New 135@24.39 U,R,AX,AY,AW +135 Sep New 135@24.39 AW OPENING N/A In the snapshot, the last auction prices are published in dedicated entries for each auction type separately. Each additional trade from another auction type, adds an entry in the snapshot up to a maximal number of four entries, one for each type of auction. If an auction trade gets reversed the respective snapshot entry for the auction trade does not get deleted. The TradeVolume (1020) is only set if TradeCondition (277) is U. 9.5 Trade Volume Reporting (MDI) The BSE MDI only provides statistical data (daily high/low price as well as total trade volume) for trades as well as the last traded price and quantity. Other information such as NumberOfBuyOrders (28821), NumberOfSellOrders (28822) are not provided. MDI reports the total volume traded through the respective order book only. This means, for example, that a direct match of two futures spread orders does not increase the total traded volume of the involved single leg contracts. 72

73 9.6 Failure of the market data feed/ matching engine The following chapters explain fail-over scenarios and how receiving applications need to process them Normal processing At start-up, the system assigns a unique sender identifier, the SenderCompID (49) to each market data feed. Afterwards the SenderCompID (49) remains constant for a given product during the entire business day. The SenderCompID (49) as shown in section 7.1 is available in the packet header and in the data message 24, e.g. Depth incremental or depth snapshot itself. For each incremental and snapshot message sent by market and reference data feeds: The field content for SenderCompID (49) in the packet header and in each data message is always the same. For each incremental and snapshot message sent by the market data feeds: The PacketSeqNum s in the packet header are contiguous per SenderCompID, multicast address and port combination. The MsgSeqNum s in the data message are contiguous per product on the incremental feed of the EMDI. The MsgSeqNum s in the data message are contiguous per product on the market data feed of the MDI 25. Figure 17 provides an example for constant SenderCompID s and increasing sequence numbers: 24 the content is the same. 25 because the BSE MDI delivers incremental and snapshots on the same channel. 73

74 Figure 17: Normal processing of sequence numbers for the BSE EMDI Market data feed fail-over (EMDI) A new SenderCompID, available in the packet header and in each data message for incremental and snapshots, indicates a fail-over of the market data feed. Incremental: the PacketSeqNum s in the packet header are reset to 1 and are contiguous per SenderCompID (80), multicast address and port combination. The MsgSeqNum s in the data message remain contiguous per product. Snapshots: the PacketSeqNum s in the packet header are reset to 1 and are contiguous per SenderCompID (80), multicast address and port combination. 74

75 Figure 18 illustrates the different behavior for incremental and snapshot messages: Figure 18: Data feed fail-over for EMDI Market data feed fail-over (MDI) Receiving applications are able to identify a failure as follows: the PacketSeqNum s in the packet header are reset to 1 and are contiguous per SenderCompID(80), multicast address and port combination. by a change of the SenderCompID (80) in the packet header and in all subsequent messages. by a reset of the MsgSegNum s for all products to 1. The snapshots are sent for all instruments before the incremental are generated. 75

76 Figure 19 illustrates the different behavior for incremental and snapshot messages: Figure 19: Data feed fail-over for MDI Participants can identify this failover scenario by decoding the packet header of UDP datagram and comparing the SenderCompID value with the previous value Market data feed restart (EMDI) A new SenderCompID, available in the packet header and in each data message for incremental and snapshots, indicates a failure. Incrementals: the PacketSeqNum s in the packet header are reset to 1 and are contiguous per SenderCompID, multicast address and port combination. the MsgSeqNum s in the data message is reset to 1 and are contiguous per product for incrementals. 76

77 Snapshots: the PacketSeqNum s in the packet header are reset to 1 and are contiguous per SenderCompID, multicast address and port combination. Once this condition is observed it is safe to assume that a fail-over scenario took place and the only correct action is to rebuild the order book. The receiving application needs to invalidate its view of the order book until an explicit message has been received containing new information. This can either be as a result of a recovery from depth snapshots or from depth incremental messages, as described in section 6.4.1, Build the initial order book with the EMDI Market data feed restart (MDI) Receiving applications are able to identify a failure as follows: by a change of the SenderCompID (80) in the packet header and in all subsequent messages. by a reset of the MsgSegNum s for all products to 1. The snapshots are sent for all instruments before the incremental are generated. Once this condition is observed it is safe to assume that a fail-over scenario took place and the only correct action is to rebuild the order book. The receiving application needs to invalidate its view of the order book until an explicit message has been received containing new information. This can either be as a result of a recovery from depth snapshots or from depth incremental messages, as described in section 6.4.2, Build the initial order book with the MDI Failure of the matching engine All non-persistent orders and quotes are deleted. Participants can see a product state change as a result of the market reset. No special processing is necessary for market data applications. In addition, participants receive a market reset event from their ETI-interface. The service availability Message indicates the unavailability of the matcher by the ETI-field MatchingEngineStatus (25005). 77

78 9.7 Trading states for a sample business day Section 4.2, Trading states introduced the trading state information. This section describes a typical trading day with the new BSE Exchange Trading Architecture. The examples refer to an equity product A10O5A and a derivatives product FDAX future on a typical expiry day. The times for each trading phase are valid for FDAX and A10O5A. Participants should not rely on any specific order or sequence of messages as described in the following chapters. For instance, the system could send an instrument state change message instead of a mass instrument state change message resulting in the same trading state at the participants side Start-Of-Day The system startup occurs in the morning. Note that with the BSE New Trading Architecture, business days are not technically linked to the local calendar. Under normal circumstances a business date is equal to the local calendar date. Nevertheless it is possible that the system startup and with it the new business day starts before midnight on the previous calendar day. At startup, the FDAX, A10O5A product goes into the product state Start-of-Day, while all its instruments are in the state Closed. Traders have no access to the order book. The system sends a product state change message (FIX TradingSessionStatus (MsgType = h)) with the field TradingSessionID (336) set to 3 = Morning and the field TradingSessionSubID (625) set to 7 = Quiescent. This indicates the product state Start-of-Day. The system furthermore sends mass instrument state change message (FIX SecurityMassStatus (MsgType = CO)) with the field SecurityMassTradingStatus (1679) containing 200 = Closed, which indicates that all instruments are in the state Closed. This message is sent once for the futures contracts and equity securities (specified in the field InstrumentScopeProductComplex (1544) containing 1 = Simple Instrument) and once for futures spreads (specified in the field InstrumentScopeProductComplex (1544) containing 5 = Futures Spread) which is the only complex instrument type supported for futures. The reference data feed begins with the system startup. Instruments that are scheduled to expire during the day are included in the reference data, but instruments that have already expired on a previous business day are no longer included in the reference data. 78

79 9.7.2 Pre-Trading At 7:50 IST, the FDAX, A10O5A product goes into the product state Pre-Trading while all its instruments change their instrument state to Restricted. Traders have no access to the order book. The system sends a product state change message with the field TradingSessionID (336) set to 3 = Morning and the field TradingSessionSubID (625) set to 1 = Pre-Trading. This indicates the product state Pre-Trading. The system furthermore sends mass instrument state change message with the field SecurityMassTradingStatus (1679) containing 201 = Restricted, which says that all instruments are in the state Restricted. This message is sent once for simple instruments and once for futures spreads Trading At 9:00 IST, the A10O5A product goes into the product state Trading. At the same time, all the simple instruments belonging to product A10O5A change their instrument state to Opening Auction. For the simple instruments, the system publishes the best bid and ask prices if the order book is not crossed, or an indicative auction price if the order book is crossed. Traders can do full order maintenance. At 9:15 IST, the FDAX product goes into the product state Trading. At the same time, all the simple instruments belonging to product A10O5A change their instrument state to Continuous. Traders can do full order maintenance. The system sends a product state change message with the field TradingSessionID (336) set to 1 = Day and the field TradingSessionSubID (625) set to 3 = Trading. This indicates the product state Trading. For A10O5A product the system furthermore sends one mass instrument state change message with the field SecurityMassTradingStatus (1679) containing 204 = Opening Auction, which says that all instruments are in the state opening auction. This message is sent only for simple instruments. For FDAX product the system furthermore sends one mass instrument state change message with the field SecurityMassTradingStatus (1679) containing 203 = Continuous, which says that all instruments are in the state opening auction. This message is sent only for simple instruments Continuous Trading 79

80 At 9:15 IST, the opening auction period of the A10O5A product ends and continuous trading starts. There is no product state change involved, but all the instruments transition to the instrument state Continuous. The change of the instrument state implies an auction trade if the order book was crossed. This applies also to the complex instruments (futures spreads), even though they had no formal auction call phase before. In the instrument state Continuous, traders can maintain their orders. Incoming orders are continuously matched. The system publishes order book and trade data. The system sends two mass instrument state change messages with the field SecurityMassTradingStatus (1679) containing 203 = Continuous, which means that all instruments are in the state Continuous. This message is sent once for simple instruments and once for futures spreads Intraday Expiry At 12:30 IST, the front month contract of the FDAX future expires on an expiration day. The affected simple instrument goes to the instrument state Restricted. The same happens to all complex instruments (futures spread) that have the affected simple instrument as a leg. For these instruments, all quotes are deleted automatically. Traders may delete their orders but not modify them or add new orders. For the expired simple instrument, the system sends a instrument state change message (FIX Security- Status (MsgType = f)) with the field SecurityTradingStatus (326) containing 201 = Restricted, which says that this particular instrument is in the state Restricted. Furthermore, the field SecurityStatus (965) contains the value 4 = Expired. For each impacted complex instrument, the system sends a instrument state change message with the field SecurityTradingStatus (326) containing 201 = Restricted. However, the field SecurityStatus (965) still contains the value 1 = Active Closing At 15:30 ISTT, the A10O5A, FDAX product is set into the product state Closing. At the same time, all its simple instruments (futures contracts) and complex instruments (futures spreads) change their instrument state to Restricted. Traders can do only order deletion. The system sends a product state change message with the field TradingSessionID (336) set to 1 = Day and the field TradingSessionSubID (625) set to 4 = Closing. This indicates the product state closing. For simple and complex instruments, the system sends one mass instrument state change message with the field Se- curitymasstradingstatus (1679) containing 201 = Restricted. The message carries an 80

81 exception list which contains the expired instrument as the only list item. For this instrument, the list item field SecurityTradingStatus (326) contains 201 = Restricted Post Closing At 15:40 IST, the product A10O5A goes into the product state Post Closing. The simple instruments that have been in the instrument state restricted now change to the state continuous. The expired front month contract and the related futures spread instruments are not affected. They remain in the state Restricted. The system sends a product state change message with the field TradingSessionID (336) set to 5 =Evening and the field TradingSessionSubID (625) set to 6 = Post Closing. This indicates the product state: Post Closing. For simple instruments, the system sends one mass instrument state change message with the field SecurityMassTradingStatus (1679) containing 203 = Continuous. The message carries an exception list which contains the expired instrument as the only list item. For this instrument, the list item field SecurityTradingStatus (326) contains 201 = Restricted End-Of-Day After 16:00 IST, the FDAX product goes into the product state End-Of-Day. All its instruments change into the instrument state Closed. Traders can no longer access the order book. The exchange system will start the end-of-day processing. The system sends a product state change message with the field TradingSessionID (336) set to 5 = Evening and the field TradingSessionSubID (625) set to 7 = Quiescent. This indicates the product state End of Day The system also sends two mass instrument state change messages with the field SecurityMassTradingStatus (1679) containing 200 = Closed, which means that all instruments are in the state Closed. This message is sent once for simple instruments and once for future spreads. 81

82 10 Fine tuning client applications This chapter covers some aspects of application tuning which should be considered during the design process of receiving applications Buffer size Messages need to be buffered and sorted in order to deal with datagram arriving in reversed order. A bigger buffer size usually slows down the processing of messages and should therefore be avoided. Conversely, receiving applications might falsely declare a message as lost if the buffer size is too small. As you can see from this example, a bigger buffer size works contrary to the speed of an application but reduces the chances of lost messages. Another factor which affects the ideal buffer size is the ratio of joined multicast streams to available band- width of a Market Data connection. A connection which operates at high network utilization levels might more often cause multicast drops or packets arriving in an incorrect sequence. Last but not least, the location of the receiving application also matters. For instance, an application running in co-location has very few out-of-order multicast packets (none in most cases) while an application which is located at a far distance from the BSE host receives a few packets out-of-order. Therefore a general recommendation concerning the buffer size cannot be made. Developers need to determine the ideal buffer size during internal testing. Please take into account that the message rate for the public broadcast is usually much lower in the simulation environment than it is in production Packet and message processing It is important that messages are removed from the network in a timely fashion to prevent them from being dropped by the client machine due to "receive buffer" overflow in the IP stack. In addition to the removal of messages from the network stack (as might be performed in response to a select() operation, for example), this design requires a time-based component to determine when a missing packet is declared lost (as opposed to simply delayed). The mechanism behind this is an implementation detail, and is platform-specific, but in its simplest form a timed select () and polling of an internal list of overdue packets would suffice. The actual time out value applied is very implementation-specific, and may be either determined dynamically (with a knowledge of when the first overdue packet is declared lost) or a simple static value. Note: Depth incremental messages must not be applied to the order book unless they are in sequence. For each network packet received, decode it into the constituent FIX message and then for each message: 82

83 The market data feeds may contain information about multiple products. If it is not for a product that the client s application is interested in: Throw it away. If the message is already in the cache: The client s application already received this message from the mirror channel, or it has been duplicated in the network. Throw it away. Otherwise: Add it to the cache Application level Various approaches can lead to faster processing on application level. The approaches depend primarily on the purpose and algorithm of the application Discarding duplicate packets within the live-live environment It is expected that receiving applications process packets from Service A and B simultaneously. The concept of the packet header allows receiving applications to detect duplicates based on the PacketSeqNum. It is recommended to discard a packet after decoding the packet header once it has been identified as duplicate. The actual message following the packet header no longer has to be decoded, allowing a faster processing speed Order book processing Depth incremental messages deliver changes of the order book from ToB to worse price levels. Trading algorithms which are based on fast matching without the knowledge of the order book could process ToB only before making a decision and process the order book afterwards. Conversely, trading algorithms with a matching logic based on the knowledge of the order book need to process the order book before sending orders. 83

84 Optimal processing of desired products (EMDI) Receiving applications interested in certain products need to join a multicast address which contains the desired products according to the mapping table provide in the reference data. Packets may arrive from different partitions on the same multicast address as shown in figure 20. The PartitionID in the packet header for the BSE EMDI can be used to identify packets arriving from partitions which carry the desired products. All other packets can be easily discarded without decoding the entire message. Figure 20: Discarding packets with unwanted products based on the PartitionID of the BSE EMDI packet header The example provided in figure 20 shows two products arriving on multicast address The participant is only interested in product A. Packets containing product A or product X can be identified by the field PartitionID in the packet header. As product X is not one of the desired products it can be discarded after decoding the message. Based on the reference data, the receiving application knows that packets coming from PartitionID 2 contain only undesired products. It discards all packets with PartitionID = 2 in the packet header without decoding message 1. 84

85 Part III Reference 11 Detailed data feed description and layout This chapter provides message layouts and field information. It is structured by service messages, data messages and data files Service messages Service messages do not carry any market information. synchronization or to indicate the status of the service. These messages are sent for the purpose of FAST reset message The template with ID = 120 is not included in the FAST Message Templates file. This TID is reserved in the main FAST specification and allocated by the FAST Session Control Protocol specification (SCP ) Note: A conforming decoder must be able to deal with the FAST reset message even though it is not mentioned in the template file. Once the FAST reset message is sent out, the dictionary needs Packet header (EMDI) Delivered in: Every UDP-datagram The packet header is a technical header used for identification of datagrams and is sent on a channel basis. Every partition stamps outgoing datagram with a sequence number (field: PacketSeqNum). One method to identify duplicates between Service A and B is by the use of the field PacketSeqNum which is unique per sendercompid; a faster way is to perform a memory comparison on the first 9 bytes of the packet header. This method eliminates the need to even decode the header in order to determine, if it has already been processed. This is especially useful to applications using both Service A and Service B feeds, allowing 26 > FAST Session Control Protocol 85

86 them to determine that a packet has already been processed without incurring any decoding overhead at all. As the packet header message is not defined in the FIX standard, the FIX Tags for PacketSeqNumber, SendingTime and PerformanceIndicator are not shown in the table below. The following layout is available after FAST decoding of the packet header: Field Name Data Type PartitionID uint32 Sending partition. SenderCompID uint32 Unique id for a sender. PacketSeqNumber byte vector Datagram sequence number. SendingTime byte vector Time when market data feed handler writes packet on the wire. PerformanceIndicator byte vector Current load of system. Time difference between the incoming ETI- order/quote and the time the market data is written to the socket. This information is provided for the incremental feed of EMDI only and is not provided for the MDI. The following picture shows the structure of the packet header before FAST-decoding: Figure 21: Structure of the packet header for BSE EMDI The last three fields are byte vectors with fixed length. Byte vectors are not stop bit encoded according to the FAST standard. Each of them is preceded by a FAST encoded 1 Byte length field as per the FAST specification for byte vector fields. Note: The field PerformanceIndicator including the length field is only available in messages on the BSE EMDI incremental feed. The PartitionID is available in messages on both incremental and snapshot feed of the EMDI. 86

87 Packet header (MDI / RDI) Delivered in: every UDP-datagram The packet header of BSE MDI and BSE RDI doesn t contain the PerformanceIndicator with length field and the PartitionID. The rest of the packet header is identical to EMDI. Duplicates between Service A and Service B can be detected by a memory comparison on the first 8 bytes of the packet header. Field Name Data Type SenderCompID uint32 Unique id for a sender. PacketSeqNumber byte vector Datagram sequence number. SendingTime byte vector Time when market data feed handler writes packet on the wire. Wire representation: Figure 22: Structure of the packet header for MDI and RDI Functional beacon message Delivered on: EMDI incremental only The functional beacon message is sent as a line active indicator whenever there are no messages generated on the EMDI incremental feed for the respective product within the last 10 seconds in production. Functional beacons are sent once the market data service becomes available. If no messages have been sent on the incremental feed for a product then LastMsgSeqNumProcessed (369) is set to zero. Tag Field Name Req d Data Type 35 MsgType Y String 0 Beacon 49 SenderCompID Y uint32 Unique id of a sender. 87