Data Factory TM In Depth

Transcription

1 Data Factory TM In Depth CQG Data Factory offers decades of historical data online. Order and download accurate, top-quality data from over 60 exchanges worldwide. Immediately access 20 years of historical market data online, including Time & Sales (tick data), End of Day, and Intraday data from September Table Of Contents Format of Data Factory Data File Names File Format of Time & Sales, Bar, and Daily Data Daily Bar Data Volume & Open Interest Data Factory Addendum for End of Day File Format Field Definitions CQG Data Factory Time Stamps Cash Futures Special Case Saturday Time Stamps for Certain Contracts Data Delivery by FTP Data File Compression Additional Support Contact Us 2008 CQG, Inc. All rights reserved worldwide.

2 Data Factory In Depth CQG s Data Factory offers 20 years of historical market data, and intraday numbers since Some of our most frequently asked questions concern: Format of Data Factory Data File Format of Time & Sales, Bar, and Daily Data Data Delivery by FTP Data File Compression Format of Data Factory Data The CQG Data Factory can produce files in a variety of time intervals: Time & Sales Data (Tick Data) Intraday Bar Data (1-60 Minute Bars) Daily Bar Data Daily Volume / Open Interest Data is delivered for a CQG trading day that typically runs from 16:00 one day to 15:59 the next day (all time stamps are US Central Time). This is not the same as a calendar day. The evening session will be included in the next trading day s data. Sessions are set up in accordance with exchange specifications. The exception is that Data Factory data requires an additional session for sessions spanning 00:00 CT. Sessions may not be consistent across contracts, but tend to be consistent for a specific contract over time. The data is delivered with CQG PC symbols. These symbols may or may not correspond to Exchange symbols. 2 Data Factory In Depth

3 Data File Names The contract name is in the format F.US.PPSSMYY, where F is the type (F for future), US is the country code (US for all futures), PS is the CQG PC Symbol for a commodity (one or more characters), M is the contract month and YY are the last two digits of the year. For cash contracts the Contract name is in the format X.CC.PPSS, where X is the type (X for cash or index), CC is the country code, and PPSS is the CQG PC Symbol (4 or more characters).the type of data in each file is identified by a file name extension. The CQG contract name appears before the file name extension. Data Type Extension (T)ime & (S)ales.ts (B)ar Data - (03) Minute Bars.b03 (B)ar Data - (15) Minute Bars.b15 (B)ar Data - (45) Minute Bars.b45 (B)ar Data - (60) Minute Bars.b60 (B)ar Data - (D)aily Bars.bd (V)olume - (D)aily Volume.bd Examples: F.US.US92M.B03 (F=Future, US=United States, US=Treasury Bonds (Pit), 92=1992, M=June, B=Bar, 03=3 minute) X.US.SPC5.BD (SPC5=S&P 500 Cash, B=Bar, D=Daily) File Format of Time & Sales, Bar, and Daily Data Time & Sales (Tick Data) Each record contains eight data fields separated by one or more spaces. The fields are: CQG Contract Name 6-14 characters Trade Date 8 digits - yyyymmdd (S)ession Indicator 1 character - 0=First Session, 1=Second Session, 2=Third Session Transaction Time 4 digits - hhmm (U.S. Central Time) Transaction Price 1-7 digits Transaction (T)ype 1 character - B=Bid A=Ask T=Trade S=Settle (not trade vol) F.US.DDZ F.US.DDZ F.US.DDZ F.US.DDZ Data Factory Addendum for End of Day file format End of Day File Format is similar to Time & Sales format, and also includes columns for Contributor Identification, Tick Flags, and Trade Volume. Data Factory In Depth

4 Data Factory Addendum for End of Day file format (Cont) Forex Sample * CID TickF Vol X.US.USDZAR B N T BNN 0 X.US.USDZAR A N T BNN 0 X.US.USDZAR B N T BNN 0 X.US.USDZAR A N T BNN 0 Daily Bar Data Each daily records contains six data fields separated by one or more spaces. The fields are Contract Name, Date, Open, High, Low, and Last. Example: Contract Date Open High tlow Last F.US.DDM F.US.DDM Volume & Open Interest (Daily) Each Volume & Open Interest record contains six data fields separated by one or more spaces. The fields are Contract Name, Date, Contract Volume, Contract Open Interest, Commodity Volume, and Commodity Open Interest. Note: Commodity Volume and Commodity Open Interest are the sums of all Volume and Open Interest for all maturities of the commodity. Example: Contract Date Volume Open Int. Volume Open Interest F.US.QEAF F.US.QEAF Field Definitions Note: The Transaction (C)orrection Indicator (defined in Readme.doc) has a value Normal = N. In End of day format, the value Normal is designated by T. CID: Contributor Identification -- which contributor sent the quote. This is not used for futures, as the exchange is the only contributor. It is used for Comprehensive FX (such as USDJPY). This could potentially be used for any consolidated feed in the future, for example US equities on CTS. TickF: Tick flags. 5 characters XxyzN where: Xx - 2 character sales condition, specified in configuration file and subject to change (see below). Sales Condition Indicator -- no sales condition n- notional settle ba indicative bid/ask Z- out of sequence 4 Data Factory In Depth

5 C- cash sale G- bunch sold N- next day O- open out of sequence R- seller S- Spread Trade T- formt out hours W- Average price b- indicative bid a- indicative ask y - Intraday/historical processing flag, set if tick makes intraday or historical (or both) bars. B - both intra/hist H - historical bars only I - intraday bars only N - neither intra/hist z - flat filtered flag. F - tick is flat filtered N - tick not flat filtered N - unused, reserved for future flag. Vol: Trade Volume and Bid Ask Size. This is generally available on electronic markets, but not on open outcry markets. Full trade volume is available when sent explicitly by the exchange, that is, not as cumulative volume. CQG Data Factory Time Stamps This section describes CQG Data Factory time stamps. It is recommended that the (README.doc) is understood prior to reading this. There are two distinct sets: time stamps for Time and Sales Data, and for bar data (1 minute to 60 minute). The time stamps for Time and Sales data are by trade date, and time stamps for all other intraday data (1 minute bars, 5 minute bars, etc.) are chronological. All time stamps are US Central Time, or Chicago. These are adjusted for US Daylight Savings. Cash The foreign exchange currencies have three sessions: Asian European American By convention, these three sessions, in sequence, define the trade date, which runs from 1600 the first calendar day to 1559 the next calendar day. For futures and cash, the trade date will be the last calendar day, regardless of the calendar day in which the trading began. Here are some extracts from the comprehensive foreign exchange symbol USDJPY. The open begins at 16:00 daily. The following is a one-minute bar representation. (The legend and further data description is available in the document README.doc.) 5 Data Factory In Depth

6 Cash (Cont) Contract Date Time Open High Low Last #Ticks_in_Bar (not trade volume) USDJPY USDJPY USDJPY USDJPY USDJPY The previous example was built from the Time and Sales representation, which follows. Contract Date (S)Time Price (T)(M)(C) USDJPY B N N USDJPY A N N USDJPY B N N USDJPY A N N USDJPY B N N USDJPY A N N USDJPY B N N USDJPY A N N USDJPY B N N USDJPY A N N USDJPY B N N USDJPY A N N USDJPY B N N USDJPY A N N USDJPY B N N USDJPY A N N USDJPY B N N USDJPY A N N USDJPY B N N USDJPY A N N USDJPY B N N USDJPY A N N USDJPY B N N USDJPY A N N USDJPY B N N USDJPY A N N USDJPY B N N USDJPY A N N USDJPY B N N USDJPY A N N USDJPY B N N USDJPY A N N USDJPY B N N 6 Data Factory In Depth

7 Cash (Cont) USDJPY A N N USDJPY B N N USDJPY A N N USDJPY B N N USDJPY A N N USDJPY B N N USDJPY A N N USDJPY B N N USDJPY A N N Immediately we can see that the prices are correct, but the first sample shows a data beginning and the second a date beginning The reason for this is that data from 1600 to 2359 daily belongs to the trade date. To repeat, the time stamps for Time and Sales data are by trade date, and time stamps for all other intraday data (1 minute bars, 5 minute bars, etc.) are chronological. The.ts file, when opened in a text file, is in the correct sequence as to be chronological. However, some scripts will assume the portion of the time stamp means the tick occurred at , which is not the case. In a transition from one day to the next, the step from 23:59 to 00:00 shows a change from the date portion of the time stamp from to At the same time, the session indicator incremented from 0 to 1. It is therefore possible to use the session indicator to order time and sales time stamps into the correct sequence chronologically. To sequence the prices, when comparing two with the same date stamp such as... USDJPY A N N USDJPY B N N... the larger time (2359) occurs with the smaller sequence number (0), then this price appeared first chronologically. The above algorithm may be standardized as follows. To standardize time stamps to US Central time, compare ticks with a given date and more than one session. In the case that a larger time (e.g. 2359) occurs with the smaller sequence number (e.g. 0), then this price appeared chronologically before ticks with a larger session indicator. The date portion of all ticks with the smaller session indicator must be decremented by one. In the case there is only one session indicator or in which all trading halts previous to 00:00 US Central Time, the date portion of ticks for all sessions must be decremented by one. Futures A similar pattern will be observed with futures contracts such as EP and SPA, which trade overnight. However, there is one additional consideration for the fact that a session spans midnight. This is not evident in the previous example, because the first session ended at 23:59. 7 Data Factory In Depth

8 Futures (Cont) The sessions for EP (E-mini S&P 500 futures) are: Globex Night Globex Day (plus settle) The first session spans midnight. Since the CQG database archives data by chronological date, data for this session is drawn from two different files. Therefore the DF will mark session 0 as and session 1 as The second session (Globex Day) will be marked as session 2. Special Case Question I am now having problems with the JGB dates. Sometimes the day changes when the time passes midnight, sometimes it doesn t as in the following two examples. JGB DEC00 Contract Example 1: Date does not change after midnight Tick# Oct-00 23: Oct-00 23: Oct-00 23: Oct-00 23: Oct-00 23: Oct-00 00: Oct-00 00: Oct-00 00: Oct-00 00: Oct-00 00: Oct-00 00:01 Example 2: Date changes after midnight Tick # Dec-00 23: Dec-00 23: Dec-00 23: Dec-00 23: Dec-00 23: Dec-00 23: Dec-00 00: Dec-00 00: Dec-00 00: Dec-00 00: Dec-00 00: Dec-00 00: Dec-00 00:48 8 Data Factory In Depth

9 Special Case (Cont) These differences are causing me major problems trying to read the data (I can t read the times). Are you familiar with this problem? Answer Because of the time difference between Tokyo and the US, the Tokyo trade date crosses midnight in Chicago. US Daylight Savings hours (which are not observed in Japan) add to the confusion. In regards to the first example (trade date ) the date portion of the time stamp is representing the trade date, and not the chronological date. This is the case for all Time and Sales files. Therefore in your example, the two ticks Oct-00 23: Oct-00 00:00...follow each other immediately. It appears that these dates in 2000 were the last of the US Daylight Savings hours. The Data Factory extract indicates a session indicator that increased. This means that the tick from the new session belongs to the same trade date as the tick from the previous session. F.US.JGBZ F.US.JGBZ A N N B N N In the second example, you indicate a date that increases Dec-00 23: Dec-00 00:30 The Data Factory extract shows the following: F.US.JGBZ F.US.JGBZ A N N T N N Both the date and the session indicator are indicative of a new trade date. The date increased, and the session reset to 0. This pattern is observed in US winter hours. The observation you made indicates that, due to US Daylight Savings, the sessions from a single trade date will appear differently. The current sessions for JGB (summer hours), in CDT, are: The winter session hours for JGB, in CST, are Data Factory In Depth

10 Additional Support Use our Symbols and Exchanges search to view updated symbols lists and session information. Two important notes: 1. For the electronic platforms of CME and CBOT, please refer to Globex and e-cbot respectively. 2. The search lists sessions beginning with session 1, 2, etc., whereas the Data Factory begins with 0, 1, etc. Recall that the DF inserts an additional session at midnight for sessions that span midnight. Saturday Time Stamps for Certain Contracts Contracts such as SPA (CME) and DFA (CBOT) have evening sessions Sunday - Thursday that open the trade dates Monday - Friday like this (as defined by the exchange): Day of week Evening Session Day session (inc. settle) S M T W R F S X X X X X X X X X X Sessions of the same color belong together in one trade date. It can be seen that the evening session is immediately prior to the day session in each case. Contracts such as JGB (TSE) and AP (SFE) have evening sessions Monday -- Friday: Day of week Evening Session Day Session (inc. settle) S M T W R F S X X X X X X X X X X While Monday s evening session opens Tuesday s trade date, Friday s evening session opens the Monday trade date. This means that the Friday evening session is exceptional in that it does not occur immediately prior to the day with its settlement; that is, there are two calendar dates (Saturday and Sunday) in between. Sydney is 15 hours ahead of Chicago and our systems run on US Central Time. Given that Sydney contracts trade on a more or less 24 hour basis, the beginning of a session will be up to 37 hours ahead; or in the case of the Friday session, up to 85 hours ahead. This comes up against system limitations. In order to capture the evening sessions for contracts such as AP and JGB in their correct sequence (that is, evening session first, day session second, not the reverse) we have to trick our system into accepting the session by pushing the session forward one trade date. This should be transparent to the user except in the case of a non-sequential session progression, that is, the trade date made up of Friday-Monday sessions. This means that the Data Factory must necessarily capture Friday evening sessions with a Saturday time stamp. Data delivery by FTP CQG offers data delivery by FTP. This allows for quicker delivery and eliminates shipping costs. Before data can be posted to ftp, five conditions must be satisfied. 1) A valid order including specific symbols, dates, and formats must be received. 2) For new customers, a billing account must be created. 3) Billing (preferably to credit card if non-cqg customer) must be completed satisfactorily. 4) CQG s 10 Data Factory In Depth

11 Data delivery by FTP (Cont) IT department needs to create an ftp login. 5) The data extraction from the database needs to be completed, and the file posted to ftp. FTP downloads may be completed as follows. The customer will be provided with a unique login and password. 1) Using an Internet browser, the customer may type ftp://ftpb.cqg.com into the address line. 2) Then, pull down the file menu and click on Login as.... Alternately, it is possible to type the address line as follows: ftp://login:password@ftp.cqg.com. Naturally, the password is case sensitive. 3) Watch for characters such as & and (,). This document and other important Data Factory documents are located at: ftp://sample:cqg@ftp.cqg.com. Extracted data files will be in the format login.01.zip and will be located in folders named , etc. These folders will be removed after eight days. Downloads from ftp must be done in binary (BIN), not ASCII format. If you are having problems logging in, check that you are in binary mode. In addition, it is generally not possible to login directly to a sub-directory such as login/ Finally, if you do not see your folder update, refresh the browser. Data File Compression The data files compressed by CQG to approximately 1/5th to 1/20th of their normal decompressed size. The files must be decompressed before they can be used. Compressed files are delivered in *.zip format. To decompress the data files, use pkunzip or WinZip which are available free on the web at and The ZIP format is a standard several decades old. Microsoft compression on XP uses a different standard. If Microsoft uncompress on XP does not recognize the format, please install a standard format compression such as Winzip. When the data files are decompressed they become ASCII text files on the hard drive. The ASCII text files can be loaded into standalone systems, spreadsheets, or other types of software packages for analysis. Contact Us 2008 CQG, Inc. All rights reserved worldwide.